Data processing method and apparatus

Abstract

The present disclosure provides a data processing method and device. The data processing method comprises: obtaining target data from a message queue middleware; obtaining a verification value of the target data, and detecting whether the verification value of the target data exists in a preset double-linked list in a memory; if the verification value does not exist in the preset double-linked list, detecting whether the verification value exists in a preset database in a cache; and processing the target data when the verification value does not exist in the preset database. The data processing method provided by the present disclosure can quickly determine whether there is a data duplication consumption situation and does not affect the data processing speed.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a data processing method and apparatus. Background Technology

[0002] Before performing distributed real-time computing, data streams are typically consumed from subscribed message queue middleware topics, and then further processing and computation are performed based on their content. However, mainstream message queue middleware such as Kafka, RabbitMQ, and RocketMQ can all experience message duplication issues during data consumption. If the service consuming the data is not idempotent, it can lead to deviations in the computation results.

[0003] Currently, the handling of data duplication issues is mainly achieved by verifying consumption feedback messages. That is, after processing the retrieved data, the service thread executing the consumption process returns a feedback message to the message queue middleware, informing it that the data has been processed and prompting the middleware to begin distributing the next data item. If the message queue middleware does not receive a feedback message within a preset time, it will redistribute the current data to the next service thread for execution. If the preset time is too short, it can easily lead to data being judged as failing before it is fully processed, resulting in duplicate data distribution and consumption, causing inaccurate calculation results. If the preset time is too long, it cannot detect client anomalies in a timely manner, leading to excessively long waiting times when client anomalies occur, reducing the system's data processing efficiency. Therefore, the current preset time setting often fails to meet both requirements, necessitating a method to avoid data duplication that balances efficiency and accuracy.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a data processing method and a data processing apparatus to overcome, at least to some extent, the problem of duplicate data consumption caused by limitations and defects in related technologies.

[0006] According to a first aspect of the present disclosure, a data processing method is provided, comprising: obtaining target data from a message queue middleware; obtaining a verification value of the target data and detecting whether the verification value of the target data exists in a preset doubly linked list in memory; if the verification value does not exist in the preset doubly linked list, detecting whether the verification value exists in a cached preset database; and processing the target data when the verification value does not exist in the preset database.

[0007] In one exemplary embodiment of this disclosure, it further includes:

[0008] After processing the target data, obtain the verification value of the target data;

[0009] The verification value of the target data is stored in the preset doubly linked list and the preset database.

[0010] In one exemplary embodiment of this disclosure, the preset doubly linked list is a LinkedHashMap, and the preset database is a Redis database.

[0011] In one exemplary embodiment of this disclosure, storing the verification value into the preset doubly linked list and the preset database includes:

[0012] Obtain the first identifier of the target data;

[0013] The first identifier of the target data is processed using a preset algorithm to obtain the second identifier of the target data;

[0014] Store the tag values ​​corresponding to the first identifier and the target data into the preset doubly linked list;

[0015] The second identifier and the tag value corresponding to the target data are stored in the preset database.

[0016] In one exemplary embodiment of this disclosure, obtaining the verification value of the target data includes:

[0017] The identifier of the target data is obtained according to the preset attributes of the target data;

[0018] Determine the tag value corresponding to the target data;

[0019] Set the identifier and its corresponding value as the verification value of the target data.

[0020] In one exemplary embodiment of this disclosure, the elements in the preset doubly linked list are arranged in access order, and both the read pointer and the write pointer of the preset doubly linked list point to the last element of the preset doubly linked list.

[0021] In one exemplary embodiment of this disclosure, the preset doubly linked list is configured to automatically delete the head element of the preset doubly linked list when the number of elements exceeds a preset value.

[0022] According to a second aspect of the present disclosure, a data processing apparatus is provided, comprising:

[0023] The data acquisition module is used to obtain target data from the message queue middleware;

[0024] The first verification module is used to obtain the verification value of the target data and detect whether the verification value of the target data exists in a preset doubly linked list in memory;

[0025] The second verification module is used to detect whether the verification value exists in the cached preset database if the verification value does not exist in the preset doubly linked list.

[0026] The data processing module is used to process the target data when the verification value does not exist in the preset database.

[0027] According to a third aspect of this disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to perform the method as described in any of the preceding methods based on instructions stored in the memory.

[0028] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a program stored thereon that, when executed by a processor, implements the data processing method as described in any of the preceding claims.

[0029] This embodiment of the disclosure determines whether data has been processed by verifying the existence of verification values ​​in memory and cache, which can effectively avoid the problem of duplicate data consumption. In addition, since the verification target is in memory and cache, the verification speed can be greatly improved, and the data processing efficiency can be accurately identified without reducing the data processing efficiency.

[0030] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0031] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0032] Figure 1 This is a flowchart of a data processing method in an exemplary embodiment of this disclosure.

[0033] Figure 2 This is a flowchart of step S104 in one embodiment of this disclosure.

[0034] Figure 3 This is a flowchart of the verification value writing process in one embodiment of this disclosure.

[0035] Figure 4 This is an overall flowchart of the data consumption process in one embodiment of this disclosure.

[0036] Figure 5 This is a block diagram of a data processing apparatus according to an exemplary embodiment of the present disclosure.

[0037] Figure 6 This is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0038] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0039] Furthermore, the accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0040] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0041] Figure 1 A flowchart illustrating a data processing method in an exemplary embodiment of this disclosure is shown schematically.

[0042] refer to Figure 1 The data processing method 100 may include:

[0043] Step S102: Obtain the target data from the message queue middleware;

[0044] Step S104: Obtain the verification value of the target data, and check whether the verification value of the target data exists in the preset doubly linked list in memory;

[0045] Step S106: If the verification value does not exist in the preset doubly linked list, check if the verification value exists in the cached preset database;

[0046] Step S108: Process the target data if the verification value does not exist in the preset database.

[0047] This embodiment of the disclosure determines whether data has been processed by verifying the existence of verification values ​​in memory and cache, which can effectively avoid the problem of duplicate data consumption. In addition, since the verification target is in memory and cache, the verification speed can be greatly improved, and the data processing efficiency can be accurately identified without reducing the data processing efficiency.

[0048] The following is a detailed explanation of each step of the data processing method 100.

[0049] In step S102, the target data is obtained from the message queue middleware.

[0050] In this embodiment of the disclosure, the types of message queue middleware include, but are not limited to, Kafka, RabbitMQ, RocketMQ, etc., and this disclosure does not impose any special restrictions on them.

[0051] The service process retrieves target data from the message queue middleware. The number of target data items can be one, two, or more. In some embodiments, the target data is, for example, an e-commerce order. The service process processes the target data according to the process settings. The specific processing method is determined by the type of target data and the type of service process, and can be configured by those skilled in the art.

[0052] In step S104, the verification value of the target data is obtained, and it is detected whether the verification value of the target data exists in the preset doubly linked list in memory.

[0053] In this embodiment, the verification value is written by the service process after processing the target data, indicating that the target data has been processed. By checking whether the verification value exists in the preset doubly linked list and the preset database, it can be quickly determined whether the target data has been processed by other service processes. If it has been processed, the current service process abandons processing the target data to avoid duplicate consumption.

[0054] Since the pre-defined doubly linked list is stored in memory, the current service process can quickly determine whether the target data's verification value is stored therein, so that if the target data's verification value is stored therein, the processing of the target data can be quickly abandoned. In one embodiment of this disclosure, the pre-defined doubly linked list is, for example, a LinkedHashMap.

[0055] When setting up a pre-defined doubly linked list, the LRU (Least Recently Used) algorithm can first be implemented using LinkedHashMap. LinkedHashMap maintains an additional doubly linked list on top of the hash table to ensure iteration order, guaranteeing either insertion order or access order. To implement the LRU algorithm using LinkedHashMap, the accessOrder flag must first be set to true when calling the LinkedHashMap constructor, setting the elements in the pre-defined doubly linked list to be arranged in the order of access. At this time, both the read and write pointers of the pre-defined doubly linked list point to the last element of the list. After multiple operations, the head element of the pre-defined doubly linked list will be the least recently used data. Furthermore, in this embodiment, the removeEldestEntry method can be written to automatically delete the head node's data when the number of nodes reaches or exceeds a preset value, because this setting ensures that the head node's data is the least recently used.

[0056] LinkedHashMap, which implements the LRU algorithm, maintains its set capacity by removing the least recently used element, preventing unlimited memory growth. Since duplicate data consumption typically occurs around the same time, LinkedHashMap, with its fast readability and small capacity, is well-suited for handling duplicate value detection. Experiments show that LinkedHashMap can efficiently handle most duplicate data consumption scenarios in memory.

[0057] Figure 2 This is a flowchart of step S104 in one embodiment of this disclosure.

[0058] refer to Figure 2 In one embodiment, the process of obtaining the verification value of the target data may include:

[0059] Step S1041: Obtain the identifier of the target data according to the preset attributes of the target data;

[0060] Step S1042: Determine the tag value corresponding to the target data;

[0061] Step S1043: Set the identifier and the corresponding value of the identifier as the verification value of the target data.

[0062] For example, when the target data is an e-commerce order, the identifier of the target data is a string composed of the order number and the product number, and the tag value corresponding to the target data can be a fixed string, such as "Done" or "True". In other embodiments of this disclosure, other forms can also be used as the identifier and corresponding tag value of the target data, and this disclosure is not limited thereto.

[0063] When checking if a verification value for target data exists in a pre-defined doubly linked list, this verification value can be read in key-value format. Here, the key is the identifier of the target data, and the value is the corresponding tag value. By checking if the key-value pair of the target data exists, a preliminary judgment can be made as to whether the target data has already been processed. This allows for the swift abandonment of processing the current target data if it has already been processed, thus improving data processing efficiency.

[0064] In other embodiments of this disclosure, the verification value of the target data may also be only the identifier of the target data. That is, an identifier of the target data can be obtained according to a preset formula, and the existence of the identifier in a preset bidirectional list can be used to identify whether the target data has been processed. This disclosure does not impose any special restrictions on this.

[0065] In step S106, if the verification value does not exist in the preset doubly linked list, the system checks whether the verification value exists in the cached preset database.

[0066] If the verification value for the target data does not exist in the preset doubly linked list, it could be that the target data has not yet been processed, or it could be that the verification value has been cleared due to the length limit of the preset doubly linked list. In this case, the preset database can be read to determine whether the verification value exists, thus performing a secondary check on the existence of the verification value.

[0067] In one embodiment of this disclosure, the preset database can be, for example, a Redis database. Redis has a large capacity, but because it is stored in a cache, its read speed is slower than that of a LinkedHashMap stored in memory. Therefore, this embodiment uses it as a secondary verification tool for verification values, providing a more secure storage location for verification values ​​when the LinkedHashMap's linked list fails to record enough verification values. Experiments have shown that Redis can effectively handle duplicate consumption cases missed by LinkedHashMap. Because Redis is a high-speed cache database, it can ensure both accuracy and data processing efficiency during real-time computation.

[0068] In other embodiments of this disclosure, the preset database in the cache may also be of other types, and this disclosure does not impose any special restrictions on it.

[0069] In step S108, the target data is processed if the verification value does not exist in the preset database.

[0070] If the target data's verification value does not exist in either the pre-set doubly linked list or the pre-set database, it means the target data has not yet been processed and there is no duplicate consumption. In this case, the current service thread can process the target data normally. If the target data's verification value exists in the pre-set database, it means the target data has been consumed repeatedly and has been processed by another service thread. The current service thread can immediately abandon processing the target data.

[0071] Since the entire verification process is carried out in memory and cache, the verification speed is extremely fast, thus ensuring data processing efficiency; and since two verification steps are set up, the verification accuracy can be greatly improved, achieving compatibility between verification accuracy and verification speed.

[0072] After processing the target data, the service process can obtain the verification value of the target data and write it into a pre-defined doubly linked list in memory and a pre-defined database in the cache. The writing format can be, for example, a key-value pair. Here, the key is the identifier of the target data, and the value is the verification value of the target data.

[0073] Figure 3 This is a flowchart of the verification value writing process in one embodiment of this disclosure.

[0074] refer to Figure 3 In one embodiment of this disclosure, the process of writing the verification value of the target data 300 may include:

[0075] Step S31: Obtain the first identifier of the target data;

[0076] Step S32: Process the first identifier of the target data using a preset algorithm to obtain the second identifier of the target data;

[0077] Step S33: Store the tag value corresponding to the first identifier and the target data into the preset doubly linked list;

[0078] Step S34: Store the second identifier and the tag value corresponding to the target data into the preset database.

[0079] exist Figure 3 In the illustrated embodiment, the preset algorithm may be, for example, a consistent hashing algorithm.

[0080] In one embodiment, if the target data is order data and the default database is a Redis database, considering the performance issues when the business volume is large, in order to ensure that the verification value of the target data is evenly distributed across the servers in the Redis cluster during storage, and to maintain load balance even when the number of servers in the Redis cluster changes dynamically, a consistent hashing algorithm can be used to process the identifier of the target data. After processing, the processed identifier and the verification value of the target data are stored in Redis, and a reasonable expiration time is set for the verification value.

[0081] Figure 4 This is an overall flowchart of the data consumption process in one embodiment of this disclosure.

[0082] refer to Figure 4 In this embodiment of the disclosure, the process of consuming data may include:

[0083] Step S41: Obtain the target data from the message queue middleware;

[0084] Step S42: Determine the first identifier and verification value of the target data;

[0085] Step S43: Determine whether the first identifier and the corresponding verification value exist in the LinkedHashMap in memory. If not, proceed to step S44; if yes, end data processing.

[0086] Step S44: Perform consistent hashing on the first identifier to obtain the second identifier;

[0087] Step S45: Determine whether the second identifier and its corresponding verification value exist in the Redis database in the cache. If not, proceed to step S46; if yes, end the data processing.

[0088] Step S46: Process the target data;

[0089] Step S47: Store the first identifier and verification value of the target data into a LinkedHashMap;

[0090] Step S48: Store the second identifier and verification value of the target data into Redis.

[0091] By performing two verifications on the target data, one in memory and one in the cache, it is possible to quickly determine whether the target data has been processed, thereby improving the accuracy of duplicate consumption while ensuring data processing efficiency.

[0092] Corresponding to the above method embodiments, this disclosure also provides a data processing apparatus that can be used to execute the above method embodiments.

[0093] Figure 5 The diagram schematically illustrates a block diagram of a data processing apparatus according to an exemplary embodiment of the present disclosure.

[0094] refer to Figure 5 The data processing device 500 may include:

[0095] Data acquisition module 502 is used to acquire target data from message queue middleware;

[0096] The first verification module 504 is used to obtain the verification value of the target data and detect whether the verification value of the target data exists in a preset doubly linked list in memory;

[0097] The second verification module 506 is used to detect whether the verification value exists in the cached preset database if the verification value does not exist in the preset doubly linked list.

[0098] The data processing module 508 is used to process the target data when the verification value does not exist in the preset database.

[0099] In one exemplary embodiment of this disclosure, the first verification module 504 is used to:

[0100] The identifier of the target data is obtained according to the preset attributes of the target data;

[0101] Determine the tag value corresponding to the target data;

[0102] Set the identifier and its corresponding value as the verification value of the target data.

[0103] In one exemplary embodiment of this disclosure, the data processing module 508 is used to:

[0104] After processing the target data, obtain the verification value of the target data;

[0105] The verification value of the target data is stored in the preset doubly linked list and the preset database.

[0106] In one exemplary embodiment of this disclosure, the data processing module 508 is further configured to:

[0107] Obtain the first identifier of the target data;

[0108] The first identifier of the target data is processed using a preset algorithm to obtain the second identifier of the target data;

[0109] Store the tag values ​​corresponding to the first identifier and the target data into the preset doubly linked list;

[0110] The second identifier and the tag value corresponding to the target data are stored in the preset database.

[0111] In one exemplary embodiment of this disclosure, the preset doubly linked list is a LinkedHashMap, and the preset database is a Redis database.

[0112] In one exemplary embodiment of this disclosure, the elements in the preset doubly linked list are arranged in access order, and both the read pointer and the write pointer of the preset doubly linked list point to the last element of the preset doubly linked list.

[0113] In one exemplary embodiment of this disclosure, the preset doubly linked list is configured to automatically delete the head element of the preset doubly linked list when the number of elements exceeds a preset value.

[0114] Since the functions of the device 500 have been described in detail in their respective method embodiments, they will not be repeated here.

[0115] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0116] In an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0117] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuits,” “modules,” or “systems.”

[0118] The following reference Figure 6 To describe an electronic device 600 according to this embodiment of the present invention. Figure 6 The electronic device 600 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0119] like Figure 6As shown, the electronic device 600 is manifested in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, and a bus 630 connecting different system components (including storage unit 620 and processing unit 610).

[0120] The storage unit stores program code that can be executed by the processing unit 610, causing the processing unit 610 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform actions such as... Figure 1 The steps shown are as follows: Step S102: Obtain target data from message queue middleware; Step S104: Obtain the verification value of the target data and check whether the verification value of the target data exists in the preset doubly linked list in memory; Step S106: If the verification value does not exist in the preset doubly linked list, check whether the verification value exists in the cached preset database; Step S108: Process the target data when the verification value does not exist in the preset database.

[0121] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0122] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0123] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0124] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. As shown, network adapter 660 communicates with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0125] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0126] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section of this specification.

[0127] The program product for implementing the above-described method according to embodiments of the present invention may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0128] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0129] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0130] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0131] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0132] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0133] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and concept of this disclosure are indicated by the claims.

Claims

1. A data processing method, characterized in that, include: Retrieve target data from message queue middleware; Obtain the verification value of the target data, and check whether the verification value of the target data exists in a preset doubly linked list in memory; If the verification value does not exist in the preset doubly linked list, check if the verification value exists in the cached preset database; The target data is processed when the verification value does not exist in the preset database; After processing the target data, obtain the verification value of the target data; The verification value of the target data is stored in the preset doubly linked list and the preset database; The verification values ​​for obtaining the target data include: The identifier of the target data is obtained according to the preset attributes of the target data; Determine the tag value corresponding to the target data; Set the identifier and the corresponding tag value of the identifier as the verification value of the target data.

2. The data processing method as described in claim 1, characterized in that, Storing the verification value into the preset doubly linked list and the preset database includes: Obtain the first identifier of the target data; The first identifier of the target data is processed using a preset algorithm to obtain the second identifier of the target data; Store the tag values ​​corresponding to the first identifier and the target data into the preset doubly linked list; The second identifier and the tag value corresponding to the target data are stored in the preset database.

3. The data processing method as described in claim 1, characterized in that, The elements in the preset doubly linked list are arranged in access order, and both the read pointer and the write pointer of the preset doubly linked list point to the last element of the preset doubly linked list.

4. The data processing method as described in claim 3, characterized in that, The preset doubly linked list is configured to automatically delete the head element of the preset doubly linked list when the number of elements exceeds a preset value.

5. The data processing method according to any one of claims 1 to 4, characterized in that, The preset doubly linked list is a LinkedHashMap, and the preset database is a Redis database.

6. A data processing apparatus, characterized in that, include: The data acquisition module is used to obtain target data from the message queue middleware; The first verification module is used to obtain the verification value of the target data and detect whether the verification value of the target data exists in a preset doubly linked list in memory; The second verification module is used to detect whether the verification value exists in the cached preset database if the verification value does not exist in the preset doubly linked list. The data processing module is used to process the target data when the verification value does not exist in the preset database, and after processing the target data, obtain the verification value of the target data. The verification value of the target data is stored in the preset doubly linked list and the preset database; The verification values ​​for obtaining the target data include: The identifier of the target data is obtained according to the preset attributes of the target data; Determine the tag value corresponding to the target data; Set the identifier and its corresponding value as the verification value of the target data.

7. An electronic device, characterized in that, include: Memory; as well as A processor coupled to the memory, the processor being configured to execute the data processing method as described in any one of claims 1-5 based on instructions stored in the memory.

8. A computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the data processing method as described in any one of claims 1-5.

Images