Data processing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a data processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring bitmap information corresponding to a first data set; wherein each piece of data in the first data set carries a first identification code; the bitmap information is determined based on each first identification code; acquiring bitmap difference information from the bitmap information according to a second identification code of each datum in the second data set; wherein the second data set is determined by performing link processing calculation on each piece of data in the first data set; for each piece of data in the second data set, the second identification code of the data is the same as the first identification code before link processing; and determining target lost data according to the bitmap difference information. By adopting the method, the lost data can be timely and accurately positioned, and the correctness of the subsequent data processing process is ensured.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the popularization of the Internet, the flow of data in the operation of large-scale commercial systems usually needs to be processed by multiple computing engines or multiple processes. For example, when a customer makes a purchase and generates consumption account information, it is necessary to generate data, receive data, process data, forward data, and finally push data to the customer's corresponding WeChat / SMS. The overall processing link is extremely complex. If there is a problem in one link, data loss may occur.

[0003] However, current solutions cannot timely detect data that is lost after being processed by the processing link. Summary of the invention

[0004] Based on this, it is necessary to provide a data processing method, device, computer equipment, computer-readable storage medium and computer program product that can timely locate lost data in response to the above technical problems.

[0005] In a first aspect, the present application provides a data processing method, the method comprising:

[0006] Obtaining bitmap information corresponding to a first data set; wherein each data in the first data set carries a first identification code; and the bitmap information is determined based on each of the first identification codes;

[0007] Obtaining bitmap difference information from the bitmap information according to the second identification code of each data in the second data set; wherein the second data set is determined by link processing calculation for each data in the first data set; for each data in the second data set, the second identification code of the data is the same as the first identification code before link processing;

[0008] The target lost data is determined according to the bitmap difference information.

[0009] In one embodiment, obtaining bitmap information corresponding to the first data set includes:

[0010] Acquire the first identification code of each data in the first data set; the first data set is stored in an input node;

[0011] Generate a corresponding first hash value according to each of the first identification codes in the first data set;

[0012] Inserting the first hash value into the bitmap of the input node to generate the bitmap information.

[0013] In one embodiment, the bitmap information of the input node is serialized and transmitted to the output node; the second data set is stored in the output node; and the bitmap difference information is obtained from the bitmap information according to the second identification code of each data in the second data set, including:

[0014] Obtain a second hash value of each of the second identification codes in the second data set;

[0015] Each second hash value is queried from the bitmap information of the output node. For each second hash value, if the first hash value identical to the second hash value exists in the bitmap information, the identical first hash value is deleted from the bitmap information to obtain the bitmap difference information.

[0016] In one embodiment, querying each second hash value from the bitmap information of the output node includes:

[0017] Based on the scheduled task, each second hash value is queried from the bitmap information of the output node.

[0018] In one embodiment, the method further comprises:

[0019] When the first hash value that is the same as the second hash value exists in the bitmap information, it indicates that the data corresponding to the second hash value has passed verification, and the verified data is sent.

[0020] In one of the embodiments, determining the target lost data according to the bitmap difference information includes:

[0021] Determine the target lost data in the first data set according to each of the first hash values ​​recorded in the bitmap difference information;

[0022] After determining the target lost data according to the bitmap difference information, the method further comprises:

[0023] Perform link processing calculation on the target lost data.

[0024] In a second aspect, the present application further provides a data processing device, the device comprising:

[0025] A first acquisition module, configured to acquire bitmap information corresponding to a first data set; wherein each data in the first data set carries a first identification code; and the bitmap information is determined based on each of the first identification codes;

[0026] A second acquisition module, configured to acquire bitmap difference information from the bitmap information according to a second identification code of each data in a second data set; wherein the second data set is determined by link processing calculation for each data in the first data set; and for each data in the second data set, the second identification code of the data is the same as the first identification code before link processing;

[0027] A determination module is used to determine target lost data according to the bitmap difference information.

[0028] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0029] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0030] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0031] The above-mentioned data processing method, device, computer equipment, computer-readable storage medium and computer program product, through each data in the first data set carries the bitmap information generated by the first identification code, because the second identification code of the data in the second data set is the same as the first identification code of the data in the first data set before link processing, the second identification code of each data in the second data set is queried from the bitmap information to obtain the bitmap difference information, the bitmap difference information records the data belonging to the first data set, but not the data belonging to the second data set, that is, the data lost after link processing, and the target lost data is determined according to the bitmap difference information. Based on the characteristic that the identification code of the data before and after link processing remains unchanged, the data in the first data set and the second data set are verified, and the target lost data can be located in a timely and accurate manner, ensuring the correctness of the subsequent data processing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 is a flow chart of a data processing method in one embodiment;

[0034] Figure 2 A schematic diagram of a process for obtaining bitmap information corresponding to a first data set in an embodiment;

[0035] Figure 3 A schematic diagram of a process of obtaining bitmap difference information from bitmap information in one embodiment;

[0036] Figure 4 A schematic diagram of a flow chart of determining target lost data and recalculating the target lost data in one embodiment;

[0037] Figure 5 is a flow chart of a data processing method in another embodiment;

[0038] Figure 6 is a structural block diagram of a data processing device in one embodiment;

[0039] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0041] In one embodiment, Figure 1 As shown, a data processing method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes steps S102 to S106. Among them:

[0042] Step S102, obtaining bitmap information corresponding to the first data set; wherein each data in the first data set carries a first identification code; and the bitmap information is determined based on each first identification code.

[0043] Among them, a bitmap is a data structure used to efficiently store and manipulate information about the existence (or other simple properties) of each element in a set of data. Its main features are space efficiency and access speed. By mapping data onto a bit array, the existence of each element is represented by a bit (0 or 1) in the array. A bitmap represents a data set through a bit array, which essentially compresses information and stores it at the bit level. Each bit only needs two states, 0 and 1, to express whether an element exists or not, which greatly reduces the need for storage space.

[0044] Bitmap information refers to information related to the data stored in the bitmap.

[0045] The first data set may be a data set corresponding to a data source, such as a data set corresponding to an upstream system, or may be a data set in the current system that has not been processed by a link.

[0046] The first identification code refers to an identifier that can uniquely represent or identify a specific object or instance. This identifier is unique, that is, it will not be repeated with the identifiers of other objects or instances in the entire system or data set.

[0047] Optionally, within the same system, the first data set B stored in the data source A needs to be processed by the link processing node C, the link processing node D, and the link processing node E, and finally flows to and is stored in the node F. The server obtains the bitmap information corresponding to the first data set B of the data source A. The first data set includes multiple data, each of which carries a first identification code, such as a serial number, a serial number, or other unique identification. The bitmap information of the first data set is determined according to one of the serial numbers, serial numbers, or other unique identifications of each data in the first data set. The server obtains the bitmap information corresponding to the first data set.

[0048] Optionally, the server receives bitmap information of the first data set from the upstream system as an object of link processing in this system.

[0049] Step S104: acquiring bitmap difference information from the bitmap information according to the second identification code of each data in the second data set.

[0050] Among them, the second data set is determined by link processing calculation for each data in the first data set; the second database includes multiple data, and for each data in the second data set, the second identification code of the data is the same as the first identification code before link processing. After the data of the first data set is processed by link, the second data set is generated, and the identification code of the data does not change in the process of link processing and calculation of the data. For example, the first data set includes ABCD, where the first identification code of A is 111111; the first identification code of B is 222222... After multiple link processing by multiple computing engines, the second data set abcd is obtained, where the second identification code of a is 111111, and the second identification code of b is 222222. It is considered that A in the first data set and a in the second data set are the same data. Similarly, B in the first data set and b in the second data set are the same data.

[0051] Optionally, the server obtains bitmap difference information from the bitmap information according to the second identification code of each data in the second data set. The bitmap information is determined according to the first identification code of each data in the first data set, and the server queries from the bitmap information according to the second identification code in the second data set, and the information not queried in the bitmap information is the bitmap difference information. For example, the bitmap information is determined to be {1234567} according to the first identification code of each data in the first data set; {1257} is queried from the bitmap information according to the second identification code of each data in the second data set, and {346} is not queried from the bitmap information according to the second identification code of each data in the second data set. Then {346} is used as the bitmap difference information. Among them, both the first identification code and the second identification code can use serial numbers or running numbers.

[0052] Step S106: determining target lost data according to the bitmap difference information.

[0053] Optionally, the server restores the target lost data according to the obtained bitmap difference information, for example, according to the bitmap difference information {346}, restores the data corresponding to 3, and restores the data corresponding to 4 and 6 in sequence; the corresponding sequence of 3, 4, and 6 can also be restored in parallel.

[0054] In the above data processing method, each data in the first data set carries the bitmap information generated by the first identification code. Since the second identification code of the data in the second data set is the same as the first identification code of the data in the first data set before link processing, the bitmap difference information is queried from the bitmap information according to the second identification code of each data in the second data set, and the bitmap difference information is obtained. The bitmap difference information records the data in the first data set, but not the data in the second data set, that is, the data lost after link processing. The target lost data is determined according to the bitmap difference information. Based on the characteristic that the identification code of the data before and after link processing remains unchanged, the data in the first data set and the second data set are verified, and the target lost data can be located in a timely and accurate manner, ensuring the correctness of the subsequent data processing process.

[0055] In an exemplary embodiment, Figure 2 As shown, obtaining the bitmap information corresponding to the first data set includes steps S202 to S206. Among them:

[0056] Step S202, obtaining a first identification code of each data in a first data set; the first data set is stored in an input node.

[0057] The input node refers to the node where the data source is located, or the starting point of the verification process.

[0058] Optionally, the server obtains a first identification code stored in the input node to obtain the first data set and obtains each data in the first data set. The first identification code is, for example, a serial number, a running number or other unique identification.

[0059] Step S204: Generate a corresponding first hash value according to each first identification code in the first data set.

[0060] The hash value is the output value calculated by the hash function. This value is usually a fixed-length string or number that uniquely identifies the original input data.

[0061] Optionally, the server performs dot matrix mapping according to each first identification code in the first data set to generate a first hash value corresponding to each first identification code.

[0062] Step S206: insert the first hash value into the bitmap of the input node to generate bitmap information.

[0063] Optionally, the server inserts each first hash value into the bitmap of the input node to generate bitmap information. At this time, each data in the first data set is recorded in the bitmap information. Since a large amount of data may be stored in the first data set, the first identification code of each data is mapped to the first hash value, and the first hash value is inserted into the bitmap to generate the bitmap information. This will reduce the space occupied by the bitmap.

[0064] In this embodiment, by storing the hash values ​​generated by each first identification code in the first database in the bitmap, the space occupied by the bitmap is reduced, the data size of the generated bitmap information is as small as possible, and more data can be recorded in the limited bitmap space.

[0065] In an exemplary embodiment, Figure 3 As shown, the bitmap information of the input node is serialized and transmitted to the output node; the second data set is stored in the output node; and the bitmap difference information is obtained from the bitmap information according to the second identification code of each data in the second data set, including steps S302 to S304. Wherein:

[0066] Step S302: Obtain a second hash value of each second identification code in the second data set.

[0067] The output node refers to a node storing each data in the first data set after being processed by all links. It can be understood as the end point of the verification process.

[0068] Optionally, the server stores the second data set generated by link processing of the first data set in the output node; obtains each second identification code in the second data set, and determines the second hash value of each second identification code through hash calculation.

[0069] Step S304, query each second hash value from the bitmap information of the output node, and for each second hash value, if there is a first hash value identical to the second hash value in the bitmap information, delete the identical first hash value from the bitmap information to obtain bitmap difference information.

[0070] Optionally, the server serializes the bitmap information of the input node and transmits it to the output node; the server queries each second hash value from the bitmap information of the output node one by one or in parallel, that is, matches the second hash value with the first hash value recorded in the bitmap information. The output node is recorded as (out) and the input node is recorded as (in). The server compares the second hash value of the output node (out) with the bitmap information BitMap transmitted from the input node (in) to the output node.

[0071] Optionally, for each second hash value, if there is a first hash value identical to the second hash value in the bitmap information, the identical first hash value is deleted from the bitmap information. When all first hash values ​​identical to the second hash value are deleted from the bitmap information, if (a∈in ^ a∉out) appears, a is the bitmap difference information.

[0072] In this embodiment, by comparing the second hash value of the output node (out) with the bitmap information BitMap transmitted from the input node (in) to the output node, and deleting the first hash value in the bitmap information that is the same as the second hash value, the bitmap difference information can be accurately obtained, so that the target lost data can be accurately obtained subsequently based on the bitmap difference information.

[0073] In an exemplary embodiment, querying each second hash value from the bitmap information of the output node includes: querying each second hash value from the bitmap information of the output node based on a scheduled task.

[0074] Optionally, the server is based on a scheduled task, such as setting the start time of the dump thread, such as starting it every five minutes. The server queries each second hash value from the bitmap information of the output node every five minutes through the dump thread. For each second hash value, if there is a first hash value that is the same as the second hash value in the bitmap information, the server deletes the same first hash value from the bitmap information through the dump thread to obtain the bitmap difference information (diff set). The setting of the scheduled task is related to the size of data that the bitmap can store and the amount of data generated per unit time.

[0075] It should be noted that the time interval of the scheduled task is set according to the actual situation and is not specifically limited in this embodiment.

[0076] In this embodiment, by setting a scheduled task, the bitmap storage space can be reasonably utilized and the bitmap utilization rate can be improved.

[0077] In an exemplary embodiment, the data processing method further includes: when there is a first hash value identical to the second hash value in the bitmap information, indicating that data corresponding to the second hash value has passed verification, sending the verified data.

[0078] Optionally, when there is a first hash value that is the same as the second hash value in the bitmap information, indicating that the data corresponding to the second hash value has passed verification, the server sends the verified data from the output node to the next data processing node.

[0079] Optionally, when there is no first hash value identical to the second hash value in the bitmap information, that is, the first hash value identical to the second hash value cannot be obtained in the bitmap information BitMap, it indicates that the data corresponding to the second hash value is abnormal.

[0080] In this embodiment, by sending verified data to subsequent processing nodes, the correctness of subsequent data processing can be guaranteed.

[0081] In an exemplary embodiment, Figure 4 As shown, determining the target missing data and recalculating the target missing data includes steps S402 to S404. Among them:

[0082] Determining target lost data according to the bitmap difference information includes step S402, determining target lost data in the first data set according to each first hash value recorded in the bitmap difference information.

[0083] Optionally, since the bitmap difference information records the first hash value corresponding to each data in the first database, the server determines the first identification code of each data in the first data set according to the first hash value (a∈in ^ a∉out) different from the second hash value recorded in the bitmap difference information, and determines the target lost data according to each first identification code.

[0084] After the target lost data is determined according to the bitmap difference information, the method further includes step S404: performing link processing calculation on the target lost data.

[0085] Optionally, the server re-processes and calculates the link of the target lost data.

[0086] In this embodiment, the data correctness is ensured by being able to discover and accurately locate the specific target lost data and reprocessing it.

[0087] In an exemplary embodiment, Figure 5As shown. The input node refers to the node where the data source is located, and can also be called the starting point of the verification process. The output node refers to the node that stores each data in the first data set after being processed by all links. The end point of the verification process can be understood. The server obtains the first identification code stored in the input node to obtain the first data set and obtains each data in the first data set. The first identification code is such as a serial number, a serial number or other unique identification. Dot matrix mapping is performed according to each first identification code in the first data set to generate a first hash value corresponding to each first identification code. The server inserts each first hash value into the bitmap of the input node to generate bitmap information. At this time, each data in the first data set is recorded in the bitmap information. Since a large amount of data may be stored in the first data set, the first identification code of each data is mapped to a first hash value, and the first hash value is inserted into the bitmap to generate bitmap information. Assume that the bitmap information is {1234567}.

[0088] The second data set is determined by link processing and calculation of each data in the first data set; the second database includes multiple data, and for each data in the second data set, the second identification code of the data is the same as the first identification code before link processing. After the data of the first data set is processed by link, the second data set is generated, and the identification code of the data does not change during the process of link processing and calculation of the data.

[0089] The server stores the second data set generated after the first data set is processed by the link in the output node; obtains each second identification code in the second data set, and determines the second hash value of each second identification code through hash calculation. The server serializes the bitmap information of the input node and transmits it to the output node; the server is based on a scheduled task, such as setting the start time of the dump thread, such as starting it every five minutes. The server queries each second hash value from the bitmap information of the output node every five minutes through the dump thread, that is, matches the second hash value with the first hash value recorded in the bitmap information. Among them, the setting of the scheduled task is related to the size of data that the bitmap can store and the amount of data generated per unit time. For each second hash value, if there is a first hash value that is the same as the second hash value in the bitmap information, the server deletes the same first hash value from the bitmap information through the dump thread. When all the first hash values ​​that are the same as the second hash value are deleted from the bitmap information, if (a∈in ^ a∉out) appears, a is the bitmap difference information (diff set). According to the second hash value of each data in the second data set, the first hash value identical to the second hash value is queried from the bitmap information, such as {1257}, and according to the second identification code of each data in the second data set, the first hash value not queried from the bitmap information is {346}. Then {346} is used as the bitmap difference information.

[0090] Since the bitmap difference information records the first hash value corresponding to each data in the first database, the server determines the first identification code of each data in the first data set according to the first hash value (a∈in ^ a∉out) different from the second hash value recorded in the bitmap difference information, and determines the target lost data according to each first identification code. The server re-processes the link calculation for the target lost data.

[0091] The method also includes: when there is a first hash value identical to the second hash value in the bitmap information, indicating that the data corresponding to the second hash value has passed verification, the server sends the verified data from the output node to the next data processing node. When there is no first hash value identical to the second hash value in the bitmap information, that is, the first hash value identical to the second hash value cannot be obtained in the bitmap information BitMap, indicating that the data corresponding to the second hash value is abnormal.

[0092] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0093] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.

[0094] In an exemplary embodiment, Figure 6 As shown, a data processing device is provided, including: a first acquisition module 601, a second acquisition module 602 and a determination module 603, wherein:

[0095] The first acquisition module 601 is used to acquire bitmap information corresponding to the first data set; wherein each data in the first data set carries a first identification code; and the bitmap information is determined based on each first identification code.

[0096] The second acquisition module 602 is used to obtain bitmap difference information from the bitmap information according to the second identification code of each data in the second data set; wherein the second data set is determined by link processing calculation for each data in the first data set; for each data in the second data set, the second identification code of the data is the same as the first identification code before link processing.

[0097] The determination module 603 is used to determine the target lost data according to the bitmap difference information.

[0098] In an exemplary embodiment, the first acquisition module 601 includes:

[0099] The first acquisition unit is used to acquire a first identification code of each data in a first data set; the first data set is stored in an input node.

[0100] The hash value generating unit is used to generate a corresponding first hash value according to each first identification code in the first data set.

[0101] The insertion generation unit is used to insert the first hash value into the bitmap of the input node to generate bitmap information.

[0102] In an exemplary embodiment, the bitmap information of the input node is serialized and transmitted to the output node; the second data set is stored in the output node; the second acquisition module 602 includes:

[0103] The second acquisition unit is used to acquire a second hash value of each second identification code in the second data set.

[0104] The query and delete unit is used to query each second hash value from the bitmap information of the output node. For each second hash value, if there is a first hash value identical to the second hash value in the bitmap information, the identical first hash value is deleted from the bitmap information to obtain bitmap difference information.

[0105] Each second hash value is queried from the bitmap information of the output node.

[0106] In an exemplary embodiment, the query deletion unit is used to query each second hash value from the bitmap information of the output node based on the scheduled task.

[0107] In an exemplary embodiment, the data processing apparatus further includes:

[0108] The verification and sending module is used to send the verified data when there is a first hash value that is the same as the second hash value in the bitmap information, indicating that the data corresponding to the second hash value has passed the verification.

[0109] In an exemplary embodiment, the determination module 603 is further configured to determine target lost data in the first data set according to each first hash value recorded in the bitmap difference information.

[0110] The data processing device also includes: a link processing module, which is used to perform link processing calculations on the target lost data.

[0111] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0112] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store bitmap information. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0113] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0114] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0115] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0116] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0117] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0118] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0119] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining bitmap information corresponding to a first data set; wherein each data in the first data set carries a first identification code; and the bitmap information is determined based on each of the first identification codes; Obtaining bitmap difference information from the bitmap information according to the second identification code of each data in the second data set; wherein the second data set is determined by link processing calculation for each data in the first data set; for each data in the second data set, the second identification code of the data is the same as the first identification code before link processing; The target lost data is determined according to the bitmap difference information.

2. The method according to claim 1, characterized in that: The obtaining bitmap information corresponding to the first data set includes: Acquire the first identification code of each data in the first data set; the first data set is stored in an input node; Generate a corresponding first hash value according to each of the first identification codes in the first data set; Inserting the first hash value into the bitmap of the input node to generate the bitmap information.

3. The method according to claim 2, characterized in that Serializing the bitmap information of the input node and transmitting it to the output node; The second data set is stored in the output node; The acquiring bitmap difference information from the bitmap information according to the second identification code of each data in the second data set includes: Obtain a second hash value of each of the second identification codes in the second data set; Each second hash value is queried from the bitmap information of the output node. For each second hash value, if the first hash value identical to the second hash value exists in the bitmap information, the identical first hash value is deleted from the bitmap information to obtain the bitmap difference information.

4. The method according to claim 3, characterized in that The querying each second hash value from the bitmap information of the output node includes: Based on the scheduled task, each second hash value is queried from the bitmap information of the output node.

5. The method according to claim 3, characterized in that: The method further comprises: When the first hash value that is the same as the second hash value exists in the bitmap information, it indicates that the data corresponding to the second hash value has passed verification, and the verified data is sent.

6. The method according to any one of claims 2 to 5, characterized in that The step of determining target lost data according to the bitmap difference information includes: Determine the target lost data in the first data set according to each of the first hash values ​​recorded in the bitmap difference information; After determining the target lost data according to the bitmap difference information, the method further comprises: Perform link processing calculation on the target lost data.

7. A data processing device, characterized in that: The device comprises: A first acquisition module, configured to acquire bitmap information corresponding to a first data set; wherein each data in the first data set carries a first identification code; and the bitmap information is determined based on each of the first identification codes; A second acquisition module, configured to acquire bitmap difference information from the bitmap information according to a second identification code of each data in a second data set; wherein the second data set is determined by link processing calculation for each data in the first data set; and for each data in the second data set, the second identification code of the data is the same as the first identification code before link processing; A determination module is used to determine target lost data according to the bitmap difference information.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images