Data Processing Method and Device

Abstract

The present application provides a data processing method and apparatus. The method is applied to a first server and includes: receiving a data collection request, where the data collection request contains a serial number, obtaining target mirror data corresponding to the business to be executed from a data source according to the data collection request, associating the serial number with the target mirror data, processing the target mirror data associated with the serial number to obtain first target mirror data, so as to implement the business to be executed according to the first target mirror data and second target mirror data, where the second target mirror data is obtained by the second server processing the target mirror data associated with the serial number. The present application improves the accuracy of deduplication processing, thereby improving the effectiveness of data backup and ensuring the normal implementation of the business.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of data processing, and in particular, to a data processing method and apparatus. Background Art

[0002] With the development of network technology, in the digital background, the process of factory digitization has been initiated.

[0003] In the prior art, the realization of many operations is involved in the process of factory digitization. To ensure the normal realization of operations, the on-site data collected can be backed up, that is, the data source is connected to the servers corresponding to at least two data collection systems. Each server corresponding to a data collection system can obtain the on-site data and send the obtained on-site data to the service provider for processing. When the service provider processes the received on-site data, since the received on-site data contains duplicate on-site data, it is necessary to first perform duplicate data removal processing on the received on-site data, and then perform subsequent processing on the on-site data after duplicate data removal processing, so as to realize relevant operations.

[0004] However, when performing duplicate data removal processing on the received on-site data, since there may be multiple received on-site data, it is impossible to effectively identify which data are duplicate data, which reduces the accuracy of duplicate data removal processing, and further reduces the effectiveness of data backup, affecting the normal realization of operations. Summary of the Invention

[0005] Embodiments of the present application provide a data processing method and apparatus to improve the accuracy of duplicate data removal processing and ensure the normal realization of operations.

[0006] In a first aspect, embodiments of the present application provide a data processing method applied to a first server, and the method includes:

[0007] Receiving a data collection request, where the data collection request contains a serial number;

[0008] Obtaining target mirror data corresponding to the operation to be executed from the data source according to the data collection request, and associating the serial number with the target mirror data;

[0009] Processing the target mirror data associated with the serial number to obtain first target mirror data, so as to realize the operation to be executed according to the first target mirror data and second target mirror data, where the second target mirror data is obtained by the second server processing the target mirror data associated with the serial number.

[0010] Optionally, the processing the target mirror data associated with the serial number to obtain first target mirror data, so as to realize the operation to be executed according to the first target mirror data and second target mirror data, includes:

[0011] Perform storage processing on the target mirror data associated with the serial number to obtain the first target mirror data;

[0012] Distribute the first target mirror data to a third-party service platform, so that the third-party service platform performs deduplication processing on the first target mirror data and the second target mirror data according to the serial numbers associated with the first target mirror data and the serial numbers associated with the second target mirror data, and implements the to-be-executed service based on the target mirror data after the deduplication processing.

[0013] Optionally, the processing the target mirror data associated with the serial number to obtain the first target mirror data to implement the to-be-executed service based on the first target mirror data and the second target mirror data includes:

[0014] After obtaining the resource lock permission, process the target mirror data associated with the serial number to obtain the first target mirror data;

[0015] Perform bit setting processing on the bit map corresponding to the serial number according to the first target mirror data and the target mirror data associated with the serial number;

[0016] Release the resource lock permission, so that the second server obtains the resource lock permission and processes the target mirror data associated with the serial number according to the bit map after the bit setting processing to obtain the second target mirror data;

[0017] Send the first target mirror data to a third-party service platform, so that the third-party service platform implements the to-be-executed service according to the first target mirror data and the second target mirror data, where the second target mirror data is sent by the second server to the third-party service platform.

[0018] Optionally, the processing the target mirror data associated with the serial number to obtain the first target mirror data includes:

[0019] Perform storage processing and / or data distribution processing on the target mirror data associated with the serial number to obtain the first target mirror data, where the storage processing and the distribution processing correspond to different bit maps.

[0020] Optionally, the performing bit setting processing on the bit map corresponding to the serial number according to the first target mirror data and the target mirror data associated with the serial number includes:

[0021] Determine the data with processing failures according to the first target mirror data and the target mirror data associated with the serial number;

[0022] Set the bit corresponding to the data with processing failure in the bitmap to zero, and set the bit corresponding to the data with successful processing in the bitmap to one.

[0023] Optionally, releasing the resource lock permission enables the second server to obtain the resource lock permission, and processes the target mirror data associated with the serial number according to the bitmap after the setting process to obtain the second target mirror data, including:

[0024] Release the resource lock permission to enable the second server to obtain the resource lock permission, and obtain the target sub-mirror data from the target mirror data associated with the serial number to obtain the second target mirror data, where the target mirror data associated with the serial number is the target mirror data corresponding to the business to be executed obtained by the second server from the data source according to the data collection request, and the serial number is associated with the target mirror data, and the target sub-mirror data is the data corresponding to the bit in the bitmap being zero obtained from the target mirror data associated with the serial number.

[0025] Optionally, the method further includes:

[0026] Determine the quantity of the target sub-mirror data;

[0027] If the quantity of the target sub-mirror data is zero, generate a no-abnormality-found prompt and release the resource lock permission;

[0028] If the quantity of the target sub-mirror data is not zero, set the target sub-mirror data as the second target mirror data and send it to the third-party service platform.

[0029] Optionally, releasing the resource lock permission includes:

[0030] Release the resource lock permission after the setting process is completed or after the processing duration exceeds the preset duration threshold.

[0031] Optionally, receiving the data collection request includes:

[0032] Control the beat module to generate a frame synchronization signal every preset duration, where the frame synchronization signal includes a serial number;

[0033] Receive the frame synchronization signal and generate a data collection request including the serial number according to the frame synchronization signal.

[0034] In a second aspect, an embodiment of the present application provides a data processing apparatus, including:

[0035] A receiving module, configured to receive a data collection request, where the data collection request includes a serial number;

[0036] A processing module, configured to obtain target mirror data corresponding to the to-be-executed service from a data source according to the data collection request, and associate the serial number with the target mirror data;

[0037] The processing module is further configured to process the target mirror data associated with the serial number to obtain first target mirror data, so as to implement the to-be-executed service according to the first target mirror data and second target mirror data, where the second target mirror data is obtained by a second server processing the target mirror data associated with the serial number.

[0038] An embodiment of the present application provides a data processing method and device. After adopting the above solution, the first server can first receive a data collection request including a serial number, then obtain target mirror data corresponding to the to-be-executed service from a data source according to the data collection request, and associate the serial number with the target mirror data. The first server can also process the target mirror data associated with the serial number to obtain first target mirror data. The second server can process the target mirror data associated with the serial number to obtain second target mirror data, and then implement the to-be-executed service according to the first target mirror data and second target mirror data. By associating the same serial number with the same data, after receiving duplicate data sent by multiple servers, it is possible to effectively identify which data are duplicate data according to the serial number, improve the accuracy of deduplication processing, and thus improve the effectiveness of data backup, ensuring the normal implementation of the service. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0040] Figure 1 It is an application schematic diagram of the data processing process in the prior art of the present application;

[0041] Figure 2 It is an architecture schematic diagram of an application system for the data processing method provided by the embodiment of the present application;

[0042] Figure 3 It is a flowchart of the data processing method provided by the embodiment of the present application;

[0043] Figure 4 It is an application schematic diagram of the data processing process in another prior art of the present application;

[0044] Figure 5 Schematic diagram of the application of the data processing method provided by the embodiment of the present application;

[0045] Figure 6 Schematic diagram of the structure of the data processing device provided by the embodiment of the present application;

[0046] Figure 7 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present application. Detailed implementation manners

[0047] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0048] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described here can also include other sequence examples in addition to those illustrated or described. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily need to be limited to those clearly listed steps or units, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0049] In the prior art, the digitalization process of factories involves the implementation of numerous operations. When implementing relevant operations, field data is generally collected by a processor and then transmitted to the server corresponding to the data acquisition system for processing, thereby realizing the relevant operations. In an industrial scenario, field data is generally collected by a PLC (Programmable Logic Controller) and then transmitted to the data acquisition system, which is responsible for data calculation, storage, distribution, and other tasks. The northward direction of the data acquisition system is various business platforms, which generally receive data through the pull or push mode of a message queue and can also query data. Then, in the process of ensuring high availability in a series of links such as data acquisition, calculation, storage, distribution, and query, the following problems may occur: If an abnormality occurs in the southward data source part, data loss will occur. If an abnormality occurs during data transmission within the data processing system, data loss will occur. Data loss occurs when the northward business system obtains data. If an abnormality occurs during data storage in the data processing system, it will also cause data loss for the northward business system to use. If an abnormality occurs in the northward business system, after the abnormality is recovered, it is expected to continue consuming data from the abnormality point; otherwise, an abnormality will occur. Therefore, to solve the aforementioned problems and taking into account the cost issues in the industrial field, the dual-machine backup method is more appropriate in terms of the balance between cost and availability.

[0050] The traditional high-availability solution for dual-machine backup can be divided into two parts: data and programs. Program high availability is to make the program stateless, store the state data in the database, and the program achieves high availability through a cluster method. In the case of two machines, the programs also exist in a similar mirroring manner, that is, the programs of one machine are the same as those of the other machine. After one machine fails, the other machine can completely replace it and continue to work. Data high availability is generally solved by mutual backup of two databases and shared storage. After one database fails, the other database also has the full amount of data. If the mirrored data sources are respectively linked to the dual-machine edge data acquisition and run simultaneously, during the data processing process, the dual-machine edge data acquisition system will store two copies of data and distribute two paths of data. This may cause the problem of wasted storage resources, that is, it is equivalent to storing one piece of data twice. Moreover, the two machines also need to synchronize backups, which puts pressure on both network IO and disk IO. If the business platform only connects to one path of the edge data acquisition system, there will be a switching time during fault switching in case of an abnormality and it cannot be hot standby in real time. If the business platform connects to two paths of the edge data acquisition system, there will be two paths of duplicate data, and the business side needs to deduplicate it by itself. For the two paths of data, if deduplication is performed, since there may be multiple pieces of field data received, there are still problems in identifying which data are mirror data of the same point.

[0051] Exemplarily, Figure 1It is an application schematic diagram of the data processing process in the prior art of this application. As Figure 1 shown, two-way mirror data can be collected from the mirror data source through a programmable logic controller. There are two edge data acquisition systems (which can also correspond to two servers), namely edge data acquisition system A (i.e., the first server) and edge data acquisition system B (i.e., the second server). 1, 2, and 3 represent the order of data generation. 1', 2', and 3' are the mirrored data. In the actual data, there is no such sequence label to identify the sequence relationship between data. During the process of edge data acquisition system A and edge data acquisition system B acquiring data, processing data, and transmitting data, the order of each data may change. If the data received by the service platform is out of order, it is impossible to effectively identify which data are duplicate data, reducing the accuracy of deduplication processing, and thus reducing the effectiveness of data backup and affecting the normal realization of the service.

[0052] Based on the above technical problems, this application associates the same serial number with the same data, so that after receiving duplicate data sent by multiple servers, it is possible to effectively identify which data are duplicate data according to the serial number and perform deduplication operations, achieving the technical effects of both improving the accuracy of deduplication processing and the effectiveness of data backup, and thus ensuring the normal realization of the service.

[0053] Figure 2 It is an architecture schematic diagram of the application system of the data processing method provided by the embodiment of this application. As Figure 2 shown, in this application system, it can include a first server 101, a second server 102, a programmable logic controller 103, and a mirror data source 104. The first server 101 and the second server 102 can be the servers corresponding to the edge data acquisition system, that is, they can implement the data calculation, storage, distribution, etc. work responsible by the edge data acquisition system. The programmable logic controller 103 is a device for docking industrial equipment in the industrial field. It collects data through an industrial protocol in the south and docks with the edge data acquisition system in the north.

[0054] The first server 101 and the second server 102 can respectively receive data acquisition requests containing serial numbers, and then collect target mirror data corresponding to the service to be executed from the mirror data source 104 through the programmable logic controller 103, and associate the serial number with the target mirror data. Exemplarily, if the serial number is 8, the identifiers of the target mirror data obtained by the first server 101 and the second server 102 this time are both set to 8.

[0055] In addition, the application system may further include a third-party service platform 105. After the first server 101 and the second server 102 respectively process the target mirror data, the first target mirror data and the second target mirror data can be obtained. Then, the first target mirror data and the second target mirror data can be sent to the third-party service platform 105 so that the third-party service platform 105 can implement the to-be-executed service according to the first target mirror data and the second target mirror data. Among them, the third-party service platform 105 can implement the to-be-executed service according to the first target mirror data and the second target mirror data in an existing manner, and no detailed limitation will be provided here.

[0056] In addition, the application system may further include a metronome (or called a beat model) 106, which is a timing synchronization module. It marks the mirror data by periodically sending frame synchronization signals. The role of the metronome is to establish a mapping relationship for the data of the mirror data source, that is, the data generated under the same beat has the same beat serial number. It can periodically send frame synchronization signals. The frame synchronization signal carries an incrementing serial number. After receiving the frame synchronization signal, the two data acquisition systems can generate data acquisition requests and collect data from the data source. The collected data is transmitted subsequently using a serial number. Through the metronome, two mutually related mirror data generated by the mirror data source can be identified, that is, the data with the same serial number can be considered as mirror data.

[0057] The technical solution of the present application will be described in detail below with specific embodiments. These specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0058] Figure 3 It is a schematic flowchart of the data processing method provided by the embodiment of the present application. The method of this embodiment can be executed by the first server 101. As Figure 3 shown, the method of this embodiment may include:

[0059] S301: Receive a data acquisition request, where the data acquisition request includes a serial number.

[0060] In this embodiment, a data acquisition request periodically sent by the beat module can be received, and then the corresponding target mirror data can be collected according to the data acquisition request, and relevant services can be implemented according to the target mirror data.

[0061] Further, the receiving of the data acquisition request may specifically include:

[0062] Control the beat module to generate a frame synchronization signal every preset duration, where the frame synchronization signal includes a serial number.

[0063] Receive the frame synchronization signal and generate a data acquisition request including the serial number according to the frame synchronization signal.

[0064] Specifically, the beat module can generate a frame synchronization signal every preset duration. After the acquisition driver of the first server receives the frame synchronization signal, it can generate a data acquisition request including a serial number. Among them, the serial number can be any one or more of Arabic numerals, capital letters, lowercase letters, and special symbols, as long as the serial numbers generated each time are different.

[0065] S302: Obtain the target mirror data corresponding to the business to be executed from the data source according to the data acquisition request, and associate the serial number with the target mirror data.

[0066] In this embodiment, after the first server receives the data acquisition request, it can obtain the target mirror data corresponding to the business to be executed from the data source. Among them, the data source can be a mirror data source. After obtaining the target mirror data, the serial number can be associated with the target mirror data. Similarly, the second server can also receive the data acquisition request, then obtain the target mirror data corresponding to the business to be executed from the data source, and after obtaining the target mirror data, associate the serial number with the target mirror data. Through the serial number, a mapping relationship can be established between the target mirror data obtained by the first server and the target mirror data obtained by the second server, that is, the data generated at the same beat has the same serial number. In addition, there can be one or more second servers. In this embodiment, there is one second server.

[0067] S303: Process the target mirror data associated with the serial number to obtain the first target mirror data, so as to implement the business to be executed according to the first target mirror data and the second target mirror data, where the second target mirror data is obtained by the second server processing the target mirror data associated with the serial number.

[0068] In this embodiment, after the target mirror data is associated with the serial number, the first server can process the target mirror data associated with the serial number to obtain the first target mirror data, and the second server can also process the target mirror data associated with the serial number to obtain the second target mirror data. Among them, the processing process can be storage processing, distribution processing, deduplication processing, etc. The processing processes performed by the first server and the second server can be the same or different.

[0069] After the processing is completed, the first server can implement the to-be-executed service based on the first target mirror data and the second target mirror data. In addition, the second server can also implement the to-be-executed service based on the first target mirror data and the second target mirror data. Alternatively, the first target mirror data and the second target mirror data can also be sent to a third-party service platform, so that the third-party service platform can implement the to-be-executed service based on the first target mirror data and the second target mirror data.

[0070] After adopting the above solution, the first server can first receive a data collection request containing a serial number, then obtain target mirror data corresponding to the to-be-executed service from the data source according to the data collection request, and associate the serial number with the target mirror data. The first server can also process the target mirror data associated with the serial number to obtain the first target mirror data. The second server can process the target mirror data associated with the serial number to obtain the second target mirror data, and then implement the to-be-executed service based on the first target mirror data and the second target mirror data. By associating the same serial number with the same data, after receiving duplicate data sent by multiple servers, it is possible to effectively identify which data are duplicate data according to the serial number and perform a deduplication operation, improving the accuracy of the deduplication process, thereby improving the effectiveness of data backup and ensuring the normal implementation of the service.

[0071] Based on Figure 3 the method, the embodiments of this specification also provide some specific implementation schemes of the method, which will be described below.

[0072] In another embodiment, processing the target mirror data associated with the serial number to obtain the first target mirror data, so as to implement the to-be-executed service based on the first target mirror data and the second target mirror data may specifically include:

[0073] Perform storage processing on the target mirror data associated with the serial number to obtain the first target mirror data.

[0074] Distribute the first target mirror data to a third-party service platform, so that the third-party service platform can perform deduplication processing on the first target mirror data and the second target mirror data according to the serial number associated with the first target mirror data and the serial number associated with the second target mirror data, and implement the to-be-executed service according to the target mirror data after the deduplication processing.

[0075] Specifically, the target mirror image data of the same batch obtained by different servers has been associated with the same serial number, that is, it is possible to determine which target mirror image data is duplicate data. After different servers distribute the target mirror image data to the third-party service platform, the third-party service platform can perform deduplication processing on the first target mirror image data and the second target mirror image data according to the serial number associated with the first target mirror image data and the serial number associated with the second target mirror image data, obtain the deduplicated target mirror image data without duplicate mirror image data, and implement the to-be-executed service based on the deduplicated target mirror image data.

[0076] In addition, the first server or the second server can perform deduplication processing on the first target mirror image data and the second target mirror image data, and implement the to-be-executed service based on the deduplicated target mirror image data.

[0077] In summary, by associating the target mirror image data obtained by different servers with the same serial number, the third-party service platform can directly perform deduplication processing on the first target mirror image data and the second target mirror image data according to the serial number associated with the first target mirror image data and the serial number associated with the second target mirror image data, improving the efficiency and accuracy of the deduplication processing.

[0078] In another embodiment, processing the target mirror image data associated with the serial number to obtain first target mirror image data to implement the to-be-executed service according to the first target mirror image data and the second target mirror image data may specifically include:

[0079] After obtaining the resource lock permission, process the target mirror image data associated with the serial number to obtain first target mirror image data.

[0080] Perform bit setting processing on the bitmap corresponding to the serial number according to the first target mirror image data and the target mirror image data associated with the serial number.

[0081] Release the resource lock permission to enable the second server to obtain the resource lock permission, and process the target mirror image data associated with the serial number according to the bitmap after the bit setting processing to obtain second target mirror image data.

[0082] Send the first target mirror image data to the third-party service platform to enable the third-party service platform to implement the to-be-executed service according to the first target mirror image data and the second target mirror image data, where the second target mirror image data is sent by the second server to the third-party service platform.

[0083] In this embodiment, after associating the target mirror data of the same batch obtained from different servers with the same serial number, if the servers all send the associated target mirror data to the third-party service platform, the third-party service platform needs to perform a deduplication operation first after receiving the duplicate data, and then implement the relevant pending operations. However, each serial number can only be associated with one piece of data. If there are multiple pieces of data, different serial numbers need to be assigned, otherwise the deduplication operation for duplicate data cannot be achieved. The data processing process is cumbersome and the processing time is increased.

[0084] In the prior art, the mirror data source can also be docked with two edge data acquisition systems (i.e., two servers), but one edge data acquisition system operates while the other edge data acquisition system remains on standby, and then switches after a problem occurs in one system. Figure 4 The following is an application schematic diagram of the data processing process in another prior art of the present application, as Figure 4 shown, there are two edge data acquisition systems (which can also correspond to two servers), namely edge data acquisition system A (i.e., the first server) and edge data acquisition system B (i.e., the second server). Edge data acquisition system A operates while edge data acquisition system B remains on standby. After a problem occurs in edge data acquisition system A, it switches to edge data acquisition system B. However, this method requires relying on dual-machine health monitoring and performing dual-machine switching when an abnormality is detected. If some problems cannot be detected, the backup function cannot be achieved. For example: If a packet loss occurs on one machine but is not detected, and the dual machines are not switched at this time, the packet loss abnormality will still occur, resulting in the inability to make full use of the dual-machine resources. One machine is operating while the resources of the other machine are basically in a silent state, and the service platform needs to sense the status of the dual machines when docking with them. When an abnormality occurs and the dual machines are switched, the service platform also needs to switch. At this time, the failure switchover takes time and cannot perform real-time hot backup. If only one of the mirror data sources is docked with one edge data acquisition system and the other mirror source is connected when a problem occurs, there will also be a problem of long failure switchover time.

[0085] The present application sets an alternating loader. For the data of the same frame synchronization signal, the two servers process alternately, avoiding concurrent processing of the same target mirror data. When processing alternately, it can be judged whether there is any loss or other abnormality before and after the mirror data, and repair the abnormality (for example, when there is data loss, redistribute or store the lost data. Or, when there is no data loss, only distribute a complete target mirror data to the third-party service platform to achieve the deduplication operation of the data), improving the timeliness of remedying abnormal data. And multiple target mirror data of the same batch can be assigned the same serial number, without being limited to assigning different serial numbers to each piece of data, simplifying the processing process of business implementation.

[0086] Among them, the role of the alternating loader is to alternately process the data of the same frame synchronization signal by multiple servers to avoid concurrent processing of the same mirror data. When processing alternately, it is possible to first determine whether there is any loss or other abnormality before and after the mirror data, and repair the abnormality. The alternating loader consists of a resource lock and a bitmap (also known as a bitmap). The resource lock is a token for each server to process alternately. Whoever obtains this token has the right to process the mirror data of a frame synchronization signal. The party that does not obtain it needs to wait for the other party to release this lock. The target mirror data is transmitted with the frame synchronization signal. After which server obtains this mirror data first, it acquires the mutex lock of this frame synchronization signal. The server that obtains the resource lock starts to initiate operations such as data calculation, storage, and distribution. The other server that receives the target mirror data later waits until the previous server unlocks or releases it after a timeout, and then the latter server can continue to process the target mirror data.

[0087] In addition, after the second server obtains the resource lock permission, it can first determine whether there is already a bitmap corresponding to the sequence number. If not, the second server will create a bitmap corresponding to the current sequence number and set the bitmap corresponding to the sequence number according to the target mirror data of the associated sequence number. Subsequently, if the first server resumes operation, it can process the target mirror data of the associated sequence number according to the set bitmap to obtain the second target mirror data. That is, the identities of the two servers are not fixed. It is not always the first server that obtains the resource lock and the second server that compares the information. The opportunities for the first server and the second server to obtain the resource lock are equal. It is also possible for the second server to obtain the resource lock first and the first server to compare the information.

[0088] In summary, through the setting of the alternating loader, the two servers can alternately process the target mirror data, avoid concurrent processing of the same mirror data, and can first determine whether there is any loss or other abnormality before and after the mirror data and repair the abnormality, improving the accuracy of data processing and thus ensuring the normal implementation of the service.

[0089] Further, processing the target mirror data associated with the sequence number to obtain the first target mirror data may specifically include:

[0090] Performing storage processing and / or data distribution processing on the target mirror data associated with the sequence number to obtain the first target mirror data, where the storage processing and the distribution processing correspond to different bitmaps.

[0091] Specifically, the processing procedure can be only storage processing, or only data distribution processing, or can include both storage processing and data distribution processing, and different processing procedures each correspond to a bitmap separately. For the second server, the principle is similar and will not be specifically defined here.

[0092] In addition, the data distribution processing can be that the first server or the second server distributes data to a third-party service platform through a message queue. Among them, the user can also specify the data to be concerned and distribute it as needed.

[0093] Further, performing a bit setting process on the bitmap corresponding to the serial number according to the first target mirror data and the target mirror data associated with the serial number may specifically include:

[0094] Determining the data with processing failures according to the first target mirror data and the target mirror data associated with the serial number.

[0095] Setting the bit corresponding to the data with processing failures in the bitmap to zero, and setting the bit corresponding to the data with successful processing in the bitmap to one.

[0096] Specifically, when performing a bit setting process on the bitmap corresponding to the serial number according to the first target mirror data and the target mirror data associated with the serial number, the data with processing failures can be determined first. Exemplarily, it can be the data with storage failures or distribution failures, and then the bit corresponding to this data is set to zero, and the bits corresponding to other data are set to 1.

[0097] Among them, different serial numbers can correspond to different bitmaps. After the serial number is associated with the target mirror data, the bitmap corresponding to this serial number can be generated, and the number of bits in this bitmap is the same as the number of data included in the target mirror data.

[0098] In addition, releasing the resource lock permission so that the second server obtains the resource lock permission, and processing the target mirror data associated with the serial number according to the bitmap after the bit setting process to obtain the second target mirror data may specifically include:

[0099] Releasing the resource lock permission so that the second server acquires the resource lock permission, and obtaining target sub-mirror data from the target mirror data associated with the serial number to obtain the second target mirror data. Among them, the target mirror data associated with the serial number is the target mirror data corresponding to the business to be executed obtained by the second server from the data source according to the data collection request, and the serial number is associated with the target mirror data, and the target sub-mirror data is the data corresponding to when the bit in the bitmap is zero obtained from the target mirror data associated with the serial number.

[0100] Specifically, after the first server finishes processing the target mirror data, it can release the resource lock permission. The second server obtains the resource lock permission and then determines whether the bitmap of the target mirror data already exists. If it exists, it compares the received data with the set data in the bitmap. If there are differences, it reprocesses the different data to ensure that when the previous server has an exception, the other server can provide a remedy. Exemplarily, taking storage as an example, after the first server calls the database interface to complete the write operation, it sets each bit in the bitmap corresponding to this data identity ID (which can be a serial number). After the second server obtains the resource lock permission, it can first determine whether there is a bitmap corresponding to this serial number. If it exists, it compares the received data with the set data in the bitmap. If there are differences, it performs a write operation again on the different data.

[0101] In summary, by combining the use of the resource lock permission and the bitmap, it is possible to achieve a differential comparison of the mirror data stream and perform real-time remediation for the different data. This not only realizes the real-time hot backup of the two servers, enabling seamless switching between the two servers when one server fails, but also ensures that the data is basically not lost during an abnormal switch. Additionally, it does not rely on keep-alive and health detection, avoiding the situation in the prior art where it is necessary to monitor for exceptions and perform a switch when an exception is detected, resulting in the inability to provide services during the switch time or data loss during the switch time.

[0102] In addition, the method may further include:

[0103] Determine the quantity of the target sub-mirror data.

[0104] If the quantity of the target sub-mirror data is zero, generate a no-abnormality-found prompt and release the resource lock permission.

[0105] If the quantity of the target sub-mirror data is not zero, set the target sub-mirror data as the second target mirror data and send it to the third-party service platform.

[0106] Specifically, after obtaining the target sub-mirror data, it is possible to first determine the quantity of data included in the target sub-mirror data. If the quantity of data included is zero, it indicates that when the first server processes the data, there is no abnormal data. In this case, a no-abnormality-found prompt can be generated and the resource lock permission can be released, thereby reminding the user that no data processing abnormality has occurred and enabling real-time notification to the management personnel, improving the timeliness of data processing.

[0107] If the number of data contained in the target sub-image data is not zero, it indicates that abnormal data has occurred when the first server processes the data. Then, the target sub-image data can be set as the second target image data and sent to the third-party service platform, that is, the abnormal data is resent to the third-party service platform, ensuring the integrity of the data.

[0108] In summary, this application not only improves the timeliness of data processing but also ensures the integrity of the data.

[0109] In addition, releasing the resource lock permission may include:

[0110] After the bit setting process is completed or the processing duration exceeds the preset duration threshold, release the resource lock permission.

[0111] Specifically, releasing the resource lock permission can include two cases. One is after the first server completes the bit setting process, and the other is when the bit setting process of the first server exceeds the preset duration threshold. Among them, the preset duration threshold can be custom-set according to the actual application scenario.

[0112] Exemplarily, Figure 5 is an application schematic diagram of the data processing method provided by the embodiment of this application, as Figure 5As shown in the figure, there are two edge data acquisition systems (which can also correspond to two servers), namely edge data acquisition system A (i.e., the first server) and edge data acquisition system B (i.e., the second server), and both edge data acquisition systems are in an operating state. The data processing process can include data storage and data distribution, and data storage corresponds to a bit map, and data distribution corresponds to a bit map. Through the frame synchronization signal sent by the metronome, the two edge data acquisition systems collect a pair of target mirror data from the mirror data source. As shown in the following figure, the frame synchronization signal sequence is 8 (i.e., the serial number is 8), and the target mirror data contains 3 data, namely A, B, and C. When the edge data acquisition system performs data storage, it first obtains the resource lock of the storage alternate loader, and the party that obtains it initiates the write operation of this frame synchronization mirror data. For example, the left edge data acquisition system A first obtains the resource lock permission. When writing the target mirror data with the frame synchronization serial number of 8, it is found that the write of data B is abnormal, then the bit of data B is set to 0 in the bit map (also called bitmap) with the serial number of 8. Since the settings of A and C are successful, the corresponding bits of A and C are set to 1. The right edge data acquisition system B obtains the resource lock permission later. After determining that the bitmap with the serial number of 8 already exists, it initiates a comparison. After comparison, it is found that B in the received mirror data is abnormal in the bitmap, then it reinitiates the write operation of data B, and releases the resource lock and bitmap with the serial number of 8 after the operation is completed. In addition, when the edge data acquisition system performs data distribution, taking the push-type message queue as an example here, it first obtains the resource lock of the distribution alternate loader, and the party that obtains it initiates the distribution of the frame synchronization mirror data. For example, the left edge data acquisition system A first obtains the resource lock permission. When distributing the target mirror data with the serial number of 8, it is found that the distribution of data A fails or the response is abnormal, then the bit corresponding to data A is set to 0 in the bitmap with the serial number of 8. Since the distributions of B and C are successful, their corresponding bits are set to 1. The right edge data acquisition system B obtains the resource lock permission later. After determining that the bitmap with the serial number of 8 already exists, it initiates a comparison. After comparison, it is found that the bitmap bit of A in the received mirror data is abnormal, then it reinitiates the distribution of data A, and releases the resource lock and bitmap with the serial number of 8 after the operation is completed. The pull mode of data distribution is similar to this and will not be repeated here.

[0113] In summary, through the method of dual-machine real-time hot backup, the abnormal switching time is realized to be 0. Through the metronome and the alternate loader, etc., the frame synchronization alternate loading method is realized, which solves the problem of long switching time in the traditional dual-machine cold backup or warm backup scheme, solves the problem of data loss during abnormal switching, and truly realizes the integrity of data under dual machines.

[0114] Based on the same idea, the embodiments of this specification also provide a device corresponding to the above method. Figure 6The following is a schematic structural diagram of the data processing device provided by the embodiment of the present application. As Figure 6 shown, it may include:

[0115] A receiving module 601, configured to receive a data acquisition request, where the data acquisition request includes a serial number.

[0116] In this embodiment, the receiving module 601 is further configured to:

[0117] Control the beat module to generate a frame synchronization signal every preset duration, where the frame synchronization signal includes a serial number.

[0118] Receive the frame synchronization signal, and generate a data acquisition request including the serial number according to the frame synchronization signal.

[0119] A processing module 602, configured to obtain target mirror data corresponding to the to-be-executed service from a data source according to the data acquisition request, and associate the serial number with the target mirror data.

[0120] The processing module 602 is further configured to process the target mirror data associated with the serial number to obtain first target mirror data, so as to implement the to-be-executed service according to the first target mirror data and the second target mirror data, where the second target mirror data is obtained by the second server processing the target mirror data associated with the serial number.

[0121] In another embodiment, the processing module 602 is further configured to:

[0122] Perform a storage process on the target mirror data associated with the serial number to obtain first target mirror data.

[0123] Distribute the first target mirror data to a third-party service platform, so that the third-party service platform performs a deduplication process on the first target mirror data and the second target mirror data according to the serial numbers associated with the first target mirror data and the second target mirror data, and implements the to-be-executed service according to the target mirror data after the deduplication process.

[0124] In another embodiment, the processing module 602 is further configured to:

[0125] After obtaining the resource lock permission, process the target mirror data associated with the serial number to obtain first target mirror data.

[0126] Perform a bitwise setting process on the bitmap corresponding to the serial number according to the first target mirror data and the target mirror data associated with the serial number.

[0127] Release the resource lock permission, so that the second server obtains the resource lock permission, and processes the target mirror data associated with the serial number according to the bit map after the setting process to obtain the second target mirror data.

[0128] Send the first target mirror data to the third-party service platform, so that the third-party service platform implements the to-be-executed service according to the first target mirror data and the second target mirror data, where the second target mirror data is sent by the second server to the third-party service platform.

[0129] Further, the processing module 602 is further configured to:

[0130] Perform storage processing and / or data distribution processing on the target mirror data associated with the serial number to obtain the first target mirror data, where the storage processing and the distribution processing correspond to different bit maps.

[0131] In addition, the processing module 602 is further configured to:

[0132] Determine the data with processing failures according to the first target mirror data and the target mirror data associated with the serial number.

[0133] Set the bit corresponding to the data with processing failures in the bit map to zero, and set the bit corresponding to the data with successful processing in the bit map to one.

[0134] In addition, the processing module 602 is further configured to:

[0135] Release the resource lock permission, so that the second server acquires the resource lock permission, and obtains the target sub-mirror data from the target mirror data associated with the serial number to obtain the second target mirror data, where the target mirror data associated with the serial number is obtained by the second server from the data source according to the data collection request to obtain the target mirror data corresponding to the to-be-executed service, and associates the serial number with the target mirror data, and the target sub-mirror data is the data corresponding to when the bit in the bit map is zero obtained from the target mirror data associated with the serial number.

[0136] In addition, the processing module 602 is further configured to:

[0137] Determine the quantity of the target sub-mirror data.

[0138] If the quantity of the target sub-mirror data is zero, generate a no-abnormality-found prompt and release the resource lock permission.

[0139] If the number of the target sub-image data is not zero, set the target sub-image data as the second target image data and send it to the third-party service platform.

[0140] In addition, the processing module 602 is further configured to:

[0141] After the setting process is completed or the processing duration exceeds a preset duration threshold, release the resource lock permission.

[0142] The device provided by the embodiment of the present application can implement the method of the above-mentioned embodiment as Figure 3 shown. The implementation principle and technical effect are similar, and will not be elaborated here.

[0143] Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present application. As Figure 7 shown, the device 700 provided in this embodiment includes: a processor 701 and a memory communicatively connected to the processor. Among them, the processor 701 and the memory 702 are connected through a bus 703.

[0144] In a specific implementation process, the processor 701 executes the computer execution instructions stored in the memory 702, so that the processor 701 executes the method in the above method embodiment.

[0145] The specific implementation process of the processor 701 can be referred to the above method embodiment. The implementation principle and technical effect are similar, and will not be elaborated here in this embodiment.

[0146] In the above-mentioned Figure 7 shown embodiment, it should be understood that the processor may be a central processing unit (English: Central Processing Unit, abbreviated: CPU), and may also be other general-purpose processors, digital signal processors (English: Digital Signal Processor, abbreviated: DSP), application specific integrated circuits (English: Application Specific Integrated Circuit, abbreviated: ASIC), etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the invention can be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0147] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory.

[0148] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the buses in the drawings of the present application are not limited to only one bus or one type of bus.

[0149] The embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the processor executes the computer-executable instructions, the data processing method of the above method embodiment is implemented.

[0150] The embodiment of the present application also provides a computer program product, including a computer program. When the computer program is executed by the processor, the data processing method as described above is implemented.

[0151] The above computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disc. The readable storage medium can be any available medium accessible by a general-purpose or special-purpose computer.

[0152] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an Application Specific Integrated Circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0153] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above method embodiments are executed; and the foregoing storage medium includes various media that can store program codes, such as ROM, RAM, a magnetic disk, or an optical disc.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that, Applied to a first server, the method includes: Receiving a data collection request, wherein the data collection request contains a serial number; Obtaining target mirror data corresponding to the business to be executed from a data source according to the data collection request, and associating the serial number with the target mirror data; data generated at the same time node has the same serial number; Processing the target mirror data associated with the serial number to obtain first target mirror data, so as to implement the business to be executed according to the first target mirror data and the second target mirror data, wherein the second target mirror data is obtained by the second server processing the target mirror data associated with the serial number; Wherein, the receiving the data collection request includes: Controlling a beat module to generate a frame synchronization signal every preset duration, wherein the frame synchronization signal contains a serial number; Receiving the frame synchronization signal and generating a data collection request containing the serial number according to the frame synchronization signal.

2. The method according to claim 1, wherein The processing the target mirror data associated with the serial number to obtain first target mirror data, so as to implement the business to be executed according to the first target mirror data and the second target mirror data, includes: Performing storage processing on the target mirror data associated with the serial number to obtain first target mirror data; Distributing the first target mirror data to a third-party service platform, so that the third-party service platform performs deduplication processing on the first target mirror data and the second target mirror data according to the serial number associated with the first target mirror data and the serial number associated with the second target mirror data, and implements the business to be executed according to the target mirror data after deduplication processing, wherein the second target mirror data is sent by the second server to the third-party service platform.

3. The method according to claim 1, wherein The processing the target mirror data associated with the serial number to obtain first target mirror data, so as to implement the business to be executed according to the first target mirror data and the second target mirror data, includes: After obtaining the resource lock permission, processing the target mirror data associated with the serial number to obtain first target mirror data; Performing a bit setting process on the bit map corresponding to the serial number according to the first target mirror data and the target mirror data associated with the serial number; Releasing the resource lock permission, so that the second server obtains the resource lock permission and processes the target mirror data associated with the serial number according to the bit map after the bit setting process to obtain second target mirror data; 4. The method according to claim 3, wherein Sending the first target mirror data to a third-party service platform, so that the third-party service platform implements the business to be executed according to the first target mirror data and the second target mirror data, wherein the second target mirror data is sent by the second server to the third-party service platform. The processing the target mirror data associated with the serial number to obtain first target mirror data includes: Perform storage processing and / or data distribution processing on the target mirror data associated with the serial number to obtain first target mirror data, where the storage processing and the distribution processing correspond to different bitmaps.

5. The method according to claim 3, characterized in that, The step of setting the bitmap corresponding to the serial number according to the first target mirror data and the target mirror data associated with the serial number includes: Determine the data for which the processing fails according to the first target mirror data and the target mirror data associated with the serial number; Set the bit corresponding to the data for which the processing fails in the bitmap to zero, and set the bit corresponding to the data for which the processing is successful in the bitmap to one.

6. The method according to claim 5, characterized in that, The step of releasing the resource lock permission so that the second server obtains the resource lock permission and processes the target mirror data associated with the serial number according to the bitmap after the setting process to obtain second target mirror data includes: Release the resource lock permission so that the second server obtains the resource lock permission and obtains target sub-mirror data from the target mirror data associated with the serial number to obtain second target mirror data, where the target mirror data associated with the serial number is obtained by the second server from the data source according to the data collection request and is associated with the serial number and the target mirror data, and the target sub-mirror data is the data corresponding to the bit in the bitmap being zero in the target mirror data associated with the serial number.

7. The method according to claim 6, characterized in that, The method further includes: Determine the quantity of the target sub-mirror data; If the quantity of the target sub-mirror data is zero, generate a prompt indicating no abnormality found and release the resource lock permission; If the quantity of the target sub-mirror data is not zero, set the target sub-mirror data as the second target mirror data and send it to the third-party service platform.

8. The method according to claim 3, wherein The step of releasing the resource lock permission includes: Release the resource lock permission after the setting process is completed or after the processing duration exceeds a preset duration threshold.

9. A data processing device, characterized in that, The apparatus includes: A receiving module, configured to receive a data collection request, where the data collection request includes a serial number; A processing module, configured to obtain target mirror data corresponding to the business to be executed from a data source according to the data collection request and associate the serial number with the target mirror data; data generated at the same time node has the same serial number; The processing module is further configured to process the target mirror data associated with the serial number to obtain first target mirror data to implement the business to be executed according to the first target mirror data and the second target mirror data, where the second target mirror data is obtained by the second server processing the target mirror data associated with the serial number; The receiving module is specifically configured to control the beat module to generate a frame synchronization signal every preset duration, where the frame synchronization signal includes a serial number; receive the frame synchronization signal and generate a data collection request including the serial number according to the frame synchronization signal.

Images