Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of big data processing. The method comprises the following steps: receiving first message data sent by a first model system and second message data sent by a second model system; in the first message data and the second message data, determining the first received message data as first target message data, and encrypting the first target message data to obtain first encrypted message data; when it is determined that a historical encryption value corresponding to the first primary key value exists in a first cache region, historical message data corresponding to the first primary key value and stored in a second cache region is obtained; and when it is determined that the first encrypted message data is inconsistent with the historical message data, updating a field inconsistent with the first encrypted message data in the historical message data, and transmitting the updated field in the historical message data to a message queue to form new message data.

Description

Technical Field

[0001] The present disclosure relates to the technical field of big data processing, and more specifically, to a data processing method, device, electronic device, storage medium and program product. Background Art

[0002] In the current real-time calculation of big data, in order to improve development efficiency and resource costs, some common data will be pre-calculated in the form of models, and then a new message queue will be generated for downstream users to use.

[0003] At present, a model generally uses two systems, the primary and the backup, to execute in parallel. If an abnormality occurs in the primary system during data processing, the backup system's message can be started to replace the primary system's message, or the backup system's data can be written into the primary system after the primary system resumes normal use.

[0004] In the process of implementing the concept of the present disclosure, the inventors found that there are at least the following problems in the related technology: when switching the primary and standby model message queues, there will be a long switching delay, and the program needs to be modified during the switching process, which is prone to errors in the modification, resulting in data errors and causing potential online failures. Summary of the invention

[0005] In view of this, the present disclosure provides a data processing method, device, electronic device, storage medium and program product.

[0006] One aspect of the present disclosure provides a data processing method, comprising: receiving first message data sent by the first model system and second message data sent by the second model system, when a sending direction sends message data with the same data content to a first model system and a second model system respectively, wherein there is a time difference between the first message data and the second message data in a timing sequence;

[0007] In the first message data and the second message data, the first received message data is determined as the first target message data, and the first target message data is encrypted to obtain the first encrypted message data, wherein the first encrypted message data includes the first primary key value;

[0008] When it is determined that there is a historical encrypted value corresponding to the first primary key value in the first cache area, obtaining historical message data corresponding to the first primary key value stored in the second cache area;

[0009] When it is determined that the first encrypted message data is inconsistent with the historical message data, the fields in the historical message data that are inconsistent with the first encrypted message data are updated, and the updated fields in the historical message data are transmitted to the message queue to form new message data.

[0010] According to an embodiment of the present disclosure, the first encrypted message data further includes a first encrypted value corresponding to the first primary key value, and the method further includes:

[0011] When determining that there is no historical encrypted value corresponding to the first primary key value in the first cache area, storing the first primary key value and the first encrypted value in the first cache area;

[0012] The first primary key value and first encrypted message data corresponding to the first encryption value are stored in the second buffer area, and the first encrypted message data is transmitted to the message queue.

[0013] According to an embodiment of the present disclosure, encrypting the first target message data to obtain the first encrypted message data includes: acquiring field information of the first target message data; and encrypting the field information based on an encryption algorithm to obtain the first encrypted message data.

[0014] According to an embodiment of the present disclosure, before determining that there is a historical encryption value corresponding to the first primary key value in the first cache area, it also includes: querying the historical encryption value corresponding to the first primary key value in the first cache area according to the first primary key value of the first encrypted message data.

[0015] According to an embodiment of the present disclosure, before obtaining the historical message data corresponding to the first primary key value stored in the second cache area, it also includes: storing the first primary key value and the first encrypted value corresponding to the first primary key value in the first cache area, and overwriting the historical encrypted value.

[0016] According to an embodiment of the present disclosure, when it is determined that the first encrypted message data is inconsistent with the historical message data, updating the fields in the historical message data that are inconsistent with the first encrypted message data, and transmitting the updated fields in the historical message data to the message queue to form new message data, including:

[0017] Obtain historical message data corresponding to the first primary key value in the second cache area; determine the first encrypted message data corresponding to the first encrypted value based on the first encrypted value; compare the field information in the historical message data with the field information of the first encrypted message data to determine the field information in the historical message data that is inconsistent with the first encrypted message data; update the field information in the historical message data that is inconsistent with the first encrypted message data, and transmit the updated field information in the historical message data to the message queue to form new message data.

[0018] According to an embodiment of the present disclosure, the above method further includes: storing new message data in the second buffer area.

[0019] According to an embodiment of the present disclosure, the method further includes: determining the last received message data in the first message data and the second message data as the second target message data, encrypting the second target message data to obtain second encrypted message data, wherein the second encrypted message data includes a second primary key value;

[0020] When it is determined that there is a historical encrypted value corresponding to the second primary key value in the first cache area, obtaining the historical message data corresponding to the second primary key value stored in the second cache area;

[0021] When it is determined that the second encrypted message data is consistent with the historical message data, the second encrypted message data is deleted.

[0022] Another aspect of the present disclosure provides a data processing device, comprising: a receiving module, configured to receive first message data sent by a first model system and second message data sent by a second model system, when a sending direction sends message data with the same data content to the first model system and the second model system respectively, wherein there is a time difference between the first message data and the second message data in a timing sequence;

[0023] A first encryption module is used to determine the first received message data as the first target message data in the first message data and the second message data, and encrypt the first target message data to obtain first encrypted message data, wherein the first encrypted message data includes a first primary key value;

[0024] A first acquisition module, configured to acquire, when determining that a historical encrypted value corresponding to the first primary key value exists in the first cache area, historical message data corresponding to the first primary key value stored in the second cache area;

[0025] The update module is used to update the fields in the historical message data that are inconsistent with the first encrypted message data when it is determined that the first encrypted message data is inconsistent with the historical message data, and transmit the updated fields in the historical message data to the message queue to form new message data.

[0026] Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are configured to implement the above method.

[0027] Another aspect of the present disclosure further provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0028] Another aspect of the present disclosure provides a computer program product, which includes computer executable instructions, and the instructions are used to implement the above method when executed.

[0029] According to the embodiments of the present disclosure, a model data integration system is used to receive the first message data sent by the first model system and the second message receipt sent by the second model system, and the first received message data in the first message data and the second message data with the same data content is determined as the first target message data, and then the first buffer area and the second buffer area in the model data integration system are used to process the first target message data, so that only the changed field information is transmitted to the message queue, thereby forming new message data. Therefore, it at least partially overcomes the technical problem that when one of the model systems fails, it is necessary to start another model system to switch the message data, which causes the switching time delay and the data problems that are easy to occur, thereby causing the potential of online failures. It further achieves the guarantee that the message data with the same data content is only sent once and there is no need to switch the model system, thereby achieving the technical effect of reducing the time delay, reducing the size of message data transmission, reducing the risk of data errors, and reducing the potential of online failures without modifying the original program. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0031] Figure 1 The application scenario diagram of the data processing method and device according to the embodiment of the present disclosure is schematically shown;

[0032] Figure 2 A flowchart schematically shows a data processing method according to an embodiment of the present disclosure;

[0033] Figure 3 A flowchart of a method for forming new message data according to an embodiment of the present disclosure is schematically shown;

[0034] Figure 4 A schematic diagram schematically shows a data processing method according to an embodiment of the present disclosure;

[0035] Figure 5 A block diagram schematically shows a data processing device according to an embodiment of the present disclosure; and

[0036] Figure 6 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0037] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0038] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0039] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0040] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0041] In the embodiments of the present disclosure, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage, etc. of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security, and national security.

[0042] In the embodiments of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0043] An embodiment of the present disclosure provides a data processing method, including: in a case where a first model system and a second model system send message data with the same data content respectively, receiving first message data sent by the first model system and second message data sent by the second model system, wherein there is a time difference in timing between the first message data and the second message data; in the first message data and the second message data, determining the first received message data as the first target message data, encrypting the first target message data to obtain first encrypted message data, wherein the first encrypted message data includes a first primary key value; in a case where it is determined that there is a historical encryption value corresponding to the first primary key value in the first cache area, obtaining historical message data corresponding to the first primary key value stored in the second cache area; in a case where it is determined that the first encrypted message data is inconsistent with the historical message data, updating the fields in the historical message data that are inconsistent with the first encrypted message data, and transmitting the updated fields in the historical message data to the message queue to form new message data.

[0044] Figure 1 The following schematically shows an application scenario diagram of the data processing method and device according to an embodiment of the present disclosure. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0045] like Figure 1 As shown, the application scenario 100 according to this embodiment may include: a first model system 101, a second model system 102, a model data integration system 103, and a downstream user 104. After the first model system 101 and the second model system 102 can receive message data with the same data content sent from the same sender, the model data integration system 103 integrates the message data with the same data content received from the first model system 101 and the second model system 102, respectively, to ensure that only one of the two identical message data sent by the first model system 101 and the second model system 102 is sent, and that a message is only sent once, and one of the message data is transmitted to the message queue 105, so that the downstream user 104 only receives the fastest message data among the message data with the same data content in the first model system 101 and the second model system 102.

[0046] It should be noted that the data processing method provided in the embodiment of the present disclosure can generally be executed by the model data integration system 103. Accordingly, the data processing device provided in the embodiment of the present disclosure can generally be arranged in the model data integration system 103. The data processing method provided in the embodiment of the present disclosure can also be executed by a model data integration system or a model data integration system group that is different from the model data integration system 103 and can interact and communicate with the downstream user 104 and / or the model data integration system 103. Accordingly, the data processing device provided in the embodiment of the present disclosure can also be arranged in a model data integration system or a model data integration system group that is different from the model data integration system 103 and can interact and communicate with the downstream user 104 and / or the model data integration system 103.

[0047] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0048] Figure 2 The flowchart of the data processing method according to the embodiment of the present disclosure is schematically shown.

[0049] like Figure 2 As shown, the method 200 includes operations S210 to S240.

[0050] In operation S210, when message data with the same data content are respectively sent by the sending direction to the first model system and the second model system, first message data sent by the first model system and second message data sent by the second model system are received, wherein there is a time difference in timing between the first message data and the second message data.

[0051] According to an embodiment of the present disclosure, the first model system may be a main model system, and the second model system may be a backup model system. The message data received by the main model system and the backup model system are message data with the same data content sent by the same sender.

[0052] According to an embodiment of the present disclosure, the execution subject of the present application may be a model data integration system. The first message data sent by the first model system has the same data content as the second message data sent by the second model system. The first message data and the second message data with the same data content may be sent to the model data integration system respectively.

[0053] According to an embodiment of the present disclosure, there is a time difference in timing when the model data integration system receives the first message data and the second message data, which can be the time when the model data integration system receives the first message data sent by the first model system and the second message data sent by the second model system within a set time.

[0054] According to an embodiment of the present disclosure, there is a time difference in timing when the model data integration system receives the first message data and the second message data. For example, when the model data integration system receives the message data, it may first receive the first message data sent by the first model system and then receive the second message data sent by the second model system; or it may first receive the second message data sent by the second model system and then receive the first message data sent by the first model system.

[0055] According to an embodiment of the present disclosure, if the first model system fails and fails to send the first message data within a set time, the model data integration system can receive the second message data sent by the second model system, thereby eliminating the need for data switching between models, reducing operation and maintenance costs, and improving system stability.

[0056] According to an embodiment of the present disclosure, the model data integration system may be a computing framework such as flink, spark, storm, etc., which is used to integrate and process the first message data received from the main model system and the second message data received from the backup model system.

[0057] In operation S220, the first received message data among the first message data and the second message data is determined as the first target message data, and the first target message data is encrypted to obtain the first encrypted message data, wherein the first encrypted message data includes the first primary key value.

[0058] According to an embodiment of the present disclosure, the first target message data may be encrypted by determining the first received message data among the received first message data and the second message data as the first target message data to be encrypted, thereby obtaining the first encrypted message data.

[0059] According to an embodiment of the present disclosure, the first encrypted message data may include a first primary key value. The first primary key value may represent a unique identifier of the first message data. For example, when the first message data contains information such as a waybill number, order quantity, and order status, the waybill number may be used as the primary key value of the first message data, indicating that the waybill number is a unique identifier of the first message data.

[0060] In operation S230, when it is determined that there is a historical encryption value corresponding to the first primary key value in the first cache area, the historical message data corresponding to the first primary key value stored in the second cache area is obtained.

[0061] According to an embodiment of the present disclosure, the model data integration system can set two types of caches, namely, a first cache area and a second cache area. The first cache area is used to store the encrypted value of the received message data, which can be a historical encrypted value, and is used to quickly determine whether the received first encrypted message data has been sent within a set time; the second cache area is used to store the received message data, which can be historical message data.

[0062] According to an embodiment of the present disclosure, the first primary key value of the first encrypted message data can be used to query whether there is a historical encryption value corresponding to the first primary key value in the first cache area. If there is a historical encryption value corresponding to the first primary key value in the first cache area, it means that the first encrypted message data has been received by the model data integration system within the set time.

[0063] According to an embodiment of the present disclosure, the historical encryption value corresponding to the first primary key value is reused to obtain the historical message data corresponding to the historical encryption value stored in the second cache area. Since the historical encryption value corresponds to the first primary key value, the historical message data can be the historical message data corresponding to the first primary key value.

[0064] In operation S240, when it is determined that the first encrypted message data is inconsistent with the historical message data, the fields in the historical message data that are inconsistent with the first encrypted message data are updated, and the updated field information in the historical message data is transmitted to the message queue to form new message data.

[0065] According to an embodiment of the present disclosure, historical message data corresponding to the first primary key value is obtained from the second cache area, and all field information in the historical message data is compared with all field information in the first encrypted message data. If there is field information in the historical message data that is inconsistent with the field information in the first encrypted message data, the field information in the historical message data that is different from the field information in the first encrypted message data is updated, and the updated field information in the historical message data is transmitted to the message queue, thereby forming new message data.

[0066] According to an embodiment of the present disclosure, for example, taking order information as an example, the waybill number is used as the primary key value, and the primary key value of the historical message data is consistent with the first primary key value of the first encrypted message data, indicating that the historical message data and the first encrypted message data are orders with the same waybill number. If the order status of the historical message data at a previous time point is collection, and the order status of the first encrypted message data at the time point when the first encrypted message data is received is sorting, then the field information representing the collection status in the historical message data is updated to the field information representing the sorting status in the first encrypted message data, and the updated field information of the sorting status in the historical message data is transmitted to the message queue, thereby forming new message data.

[0067] According to an embodiment of the present disclosure, the updated field information in the historical message data is transmitted to the message queue, which can reduce the size of the message data transmission.

[0068] According to an embodiment of the present disclosure, new message data is stored in the second buffer area at the same time. The new message data represents the latest data feature state of the message data corresponding to the first primary key value.

[0069] According to an embodiment of the present disclosure, when the model data integration system determines the first received message data as the first target message data and integrates the first target message data, it can use the historical message data stored in the second cache area to compare with the first encrypted message data to determine whether the field information of the historical message data is consistent with the field information of the first encrypted message data, which can ensure that the model data integration system only sends the message data that is received faster, without sending both identical message data.

[0070] According to the embodiments of the present disclosure, a model data integration system is used to receive the first message data sent by the first model system and the second message receipt sent by the second model system, and the first received message data in the first message data and the second message data with the same data content is determined as the first target message data, and then the first buffer area and the second buffer area in the model data integration system are used to process the first target message data, so that only the changed field information is transmitted to the message queue, thereby forming new message data. Therefore, it at least partially overcomes the technical problem that when one of the model systems fails, it is necessary to start another model system to switch the message data, which causes the switching time delay and the data problems that are easy to occur, thereby causing the potential of online failures. It further achieves the guarantee that the message data with the same data content is only sent once and there is no need to switch the model system, thereby achieving the technical effect of reducing the time delay, reducing the size of message data transmission, reducing the risk of data errors, and reducing the potential of online failures without modifying the original program.

[0071] According to an embodiment of the present disclosure, the first encrypted message data also includes a first encrypted value corresponding to the first primary key value, and the above data processing method may further include:

[0072] When it is determined that there is no historical encryption value corresponding to the first primary key value in the first cache area, the first primary key value and the first encryption value are stored in the first cache area; the first primary key value and the first encrypted message data corresponding to the first encryption value are stored in the second cache area, and the first encrypted message data is transmitted to the message queue.

[0073] According to an embodiment of the present disclosure, since certain fields in the first target message data received by the model data integration system change each time, the encrypted value after encrypting the changed first target message data will also change. Therefore, the first encrypted value can be the encrypted value of the received first target message data.

[0074] According to an embodiment of the present disclosure, it can be determined whether there is a historical encryption value corresponding to the first primary key value in the first cache area according to the first primary key value of the received first encrypted message data. If there is no historical encryption value corresponding to the first primary key value in the first cache area, it can be explained that the first target message data corresponding to the first encrypted message data is sent to the model data integration system for processing for the first time.

[0075] According to an embodiment of the present disclosure, the first primary key value of the first encrypted message data and the first encryption value corresponding to the first primary key value may be stored in a first cache area.

[0076] According to an embodiment of the present disclosure, the first primary key value and the first encrypted message data corresponding to the first encryption value corresponding to the first primary key value can be stored together in the second cache area, and the first encrypted message data can be transmitted to the message queue.

[0077] According to an embodiment of the present disclosure, for example, the model data integration system receives the first message data sent by the first model system and the second message data sent by the second model system, and determines the first received message data as the first target message data in the first message data and the second message data with the same data content, and encrypts the first target message data. The encrypted first target message data, that is, the first encrypted message data may include a first primary key value and a first encryption value. If it is determined that there is no historical encryption value corresponding to the first primary key value in the first cache area, the first encryption value and the first primary key value of the base clothing encrypted message data are stored together in the first cache area. At the same time, the first primary key value and the first encrypted message data corresponding to the first encryption value are stored together in the second cache area, and the first encrypted message data is transmitted to the message queue.

[0078] According to an embodiment of the present disclosure, encrypting first target message data to obtain first encrypted message data includes: acquiring field information of the first target message data; and encrypting the field information based on an encryption algorithm to obtain the first encrypted message data.

[0079] According to an embodiment of the present disclosure, the field information of the first target message data can be obtained, and the field information can be encrypted by the MD5 encryption algorithm to obtain the first encrypted message data. The encryption form of the first encrypted message data can be the first primary key value and the first encrypted value.

[0080] According to an embodiment of the present disclosure, before determining that there is a historical encryption value corresponding to the first primary key value in the first cache area, it also includes: querying the historical encryption value corresponding to the first primary key value in the first cache area according to the first primary key value of the first encrypted message data.

[0081] According to an embodiment of the present disclosure, before obtaining the historical message data corresponding to the first primary key value stored in the second cache area, it also includes: storing the first primary key value and the first encryption value corresponding to the first primary key value in the first cache area, and overwriting the historical encryption value.

[0082] According to an embodiment of the present disclosure, after the received first target message data is encrypted, a first primary key value and a first encryption value of the first encrypted message data are obtained.

[0083] According to an embodiment of the present disclosure, based on the first primary key value, it is possible to query from the first cache whether there is a historical encryption value corresponding to the first primary key value to determine whether the message data corresponding to the first primary key value has been sent.

[0084] According to an embodiment of the present disclosure, when it is determined that there is a historical encryption value corresponding to the first primary key value in the first cache, it is determined that the message data has been sent within a set time. At the same time, the historical encryption values ​​are deduplicated and the first encryption value and the first primary key value are stored in the first cache area.

[0085] Figure 3 The flowchart of the method for forming new message data according to the embodiment of the present disclosure is schematically shown.

[0086] like Figure 3 As shown, the method 300 may include operations S310 to S340.

[0087] In operation S310, historical message data corresponding to the first primary key value in the second cache area is obtained.

[0088] In operation S320, first encrypted message data corresponding to the first encrypted value is determined according to the first encrypted value.

[0089] In operation S330, field information in the historical message data is compared with field information in the first encrypted message data to determine field information in the historical message data that is inconsistent with the first encrypted message data.

[0090] In operation S340, field information in the historical message data that is inconsistent with the first encrypted message data is updated, and the updated field information in the historical message data is transmitted to the message queue to form new message data.

[0091] According to an embodiment of the present disclosure, based on the first primary key value of the first encrypted message data, the historical encryption value corresponding to the first primary key value is determined, and the historical message data corresponding to the historical encryption value stored in the second cache area is obtained, that is, the historical message data corresponding to the first primary key value.

[0092] According to an embodiment of the present disclosure, the field information in the first encrypted message data is compared one by one with the field information in the historical message data to determine whether the field information in the first encrypted message data is the same as the field information in the historical message data.

[0093] According to an embodiment of the present disclosure, if all field information is the same, it means that the field information in the first encrypted message data is the same as the field information in the historical message data, and the first encrypted message data can be discarded.

[0094] According to an embodiment of the present disclosure, if there is different field information in all field information, it is determined that there is field information in the first encrypted message data that is different from the historical message data, and the field information in the historical message data that is different from the first encrypted message data is updated to the field information in the first encrypted message data, and the updated field information is transmitted to the message queue. At the same time, the field information in the first encrypted message data that is different from the historical message data is merged into the historical message data, and the updated historical message data is stored as new message data in the second cache area.

[0095] According to an embodiment of the present disclosure, the inconsistent field information of the first encrypted message data and the historical message data stored in the second cache area is merged and updated through the second cache area, and transmitted to the message queue, so that only the changed field information is transmitted, reducing the size of message data transmission and improving data processing efficiency.

[0096] According to an embodiment of the present disclosure, the above-mentioned data processing method may also include: in the first message data and the second message data, determining the last received message data as the second target message data, encrypting the second target message data to obtain second encrypted message data, wherein the second encrypted message data includes a second primary key value.

[0097] When it is determined that there is a historical encryption value corresponding to the second primary key value in the first cache area, the historical message data corresponding to the second primary key value stored in the second cache area is obtained.

[0098] When it is determined that the second encrypted message data is consistent with the historical message data, the second encrypted message data is deleted.

[0099] According to an embodiment of the present disclosure, a model data integration system receives first message data sent by a first model system and second message data sent by a second model system, wherein the data contents of the first message data and the second message data are the same, and there is a time difference in timing between receiving the first message data and the second message data.

[0100] According to an embodiment of the present disclosure, the second target message data may be encrypted by encrypting the last received message data among the received first message data and the second message data, thereby obtaining the second encrypted message data.

[0101] According to an embodiment of the present disclosure, the data content of the second encrypted message data is the same as that of the first encrypted message data, and the second primary key value of the second encrypted message data is the same as the first primary key value of the first encrypted message data.

[0102] According to an embodiment of the present disclosure, when the model data integration system processes the second encrypted message data, if it is determined that there is a historical encryption value corresponding to the second primary key value in the first cache area, the historical message data corresponding to the second primary key value in the second cache area will be obtained, and the second encrypted message data will be compared with the historical message data. If it is determined that the field information in the second encrypted message data is completely consistent with that in the historical message data, the second encrypted message data will be deleted.

[0103] According to an embodiment of the present disclosure, the method for processing the second encrypted message data is the same as the method for processing the first encrypted message data, except that the first encrypted message data is first received and processed by the model data integration system, and the second encrypted message data is last received and processed by the model data integration system. This disclosure will not be described in detail here.

[0104] Figure 4 A schematic diagram of a data processing method according to an embodiment of the present disclosure is schematically shown.

[0105] like Figure 4As shown, in the schematic diagram 400, the first model system 401 and the second model system 402 send the first message data 401-1 and the second message data 402-1 to the model data integration system within the preset time, respectively. The first message data 401-1 and the second message data 402-1 are message data with the same data content, and the first received message data of the first message data and the second message data is determined as the first target message data 403. The model data integration system encrypts the first target message data 403 to obtain the first encrypted message data 404, and the first encrypted message data includes the first primary key value 404-1 and the first encrypted value 404-2. Determine whether there is a historical encrypted value 405 corresponding to the first primary key value 404-1 in the first cache area. If there is a historical encrypted value corresponding to the first primary key value in the first cache area, obtain the historical message data corresponding to the historical encrypted value in the second cache area according to the first primary key value 406. Determine whether the field information of the first encrypted message data is consistent with that of the historical message data 407. If the field information is consistent, the first encrypted message data 408 is deleted, and if the field information is inconsistent, the field information inconsistent with the first encrypted message data in the historical message data is updated 409, and the inconsistent field information is transmitted to the message queue 410. If there is no historical encrypted value corresponding to the first primary key value in the first cache area, the first primary key value 404-1 and the first encrypted value 404-2 are stored in the first cache area 411, the first encrypted message data 403 is stored in the second cache area 412, and the first encrypted message data is transmitted to the message queue 410.

[0106] Figure 5 The block diagram schematically shows a data processing device according to an embodiment of the present disclosure.

[0107] like Figure 5 As shown, the device 500 may include: a receiving module 510 , a first encryption module 520 , a first acquisition module 530 and an update module 540 .

[0108] The receiving module 510 is used to receive first message data sent by the first model system and second message data sent by the second model system when message data with the same data content are respectively sent by the sending direction to the first model system and the second model system, wherein there is a time difference in timing between the first message data and the second message data.

[0109] The first encryption module 520 is used to determine the first received message data as the first target message data in the first message data and the second message data, encrypt the first target message data to obtain the first encrypted message data, wherein the first encrypted message data includes the first primary key value.

[0110] The first acquisition module 530 is used to acquire the historical message data corresponding to the first primary key value stored in the second cache area when it is determined that there is a historical encryption value corresponding to the first primary key value in the first cache area.

[0111] The updating module 540 is used to update the fields in the historical message data that are inconsistent with the first encrypted message data when it is determined that the first encrypted message data is inconsistent with the historical message data, and transmit the updated fields in the historical message data to the message queue to form new message data.

[0112] According to an embodiment of the present disclosure, the above-mentioned device 500 may further include: a first storage module and a second storage module.

[0113] The first storage module is used to store the first primary key value and the first encrypted value in the first cache area when it is determined that there is no historical encrypted value corresponding to the first primary key value in the first cache area.

[0114] The second storage module stores the first primary key value and the first encrypted message data corresponding to the first encryption value in the second buffer area, and transmits the first encrypted message data to the message queue.

[0115] According to an embodiment of the present disclosure, the encryption module 520 may include: a first acquisition submodule and a processing submodule.

[0116] The first acquisition submodule is used to acquire field information of the first target message data.

[0117] The processing submodule is used to encrypt the field information based on the encryption algorithm to obtain the first encrypted message data. According to the embodiment of the present disclosure, the above-mentioned device 500 may also include: a query module.

[0118] The query module is used to query the historical encryption value corresponding to the first primary key value in the first cache area according to the first primary key value of the first encrypted message data.

[0119] According to an embodiment of the present disclosure, the above-mentioned device 500 may further include: a third storage module.

[0120] The third storage module is used to store the first primary key value and the first encrypted value corresponding to the first primary key value in the first cache area, and overwrite the historical encrypted value.

[0121] According to an embodiment of the present disclosure, the updating module 540 may include: a second acquiring submodule, a first determining submodule, a second determining submodule, and an updating submodule.

[0122] The second acquisition submodule is used to acquire the historical message data corresponding to the first primary key value in the second buffer area. The first determination submodule is used to determine the first encrypted message data corresponding to the first encrypted value according to the first encrypted value.

[0123] The second determination submodule is used to compare the field information in the historical message data with the field information in the first encrypted message data, and determine the field information in the historical message data that is inconsistent with the first encrypted message data.

[0124] The updating submodule is used to update the field information in the historical message data that is inconsistent with the first encrypted message data, and transmit the updated field information in the historical message data to the message queue to form new message data.

[0125] According to an embodiment of the present disclosure, the above-mentioned device 500 may further include: a fourth storage module.

[0126] The fourth storage module is used to store new message data in the second buffer area.

[0127] According to an embodiment of the present disclosure, the above-mentioned device 500 may further include: a second encryption module, a second acquisition module and a deletion module.

[0128] The second encryption module is used to determine the last received message data in the first message data and the second message data as the second target message data, encrypt the second target message data, and obtain the second encrypted message data, wherein the second encrypted message data includes a second primary key value.

[0129] The second acquisition module is used to acquire the historical message data corresponding to the second primary key value stored in the second cache area when determining that there is a historical encryption value corresponding to the second primary key value in the first cache area.

[0130] The deleting module is used to delete the second encrypted message data when it is determined that the second encrypted message data is consistent with the historical message data.

[0131] According to the modules and submodules of the embodiments of the present disclosure, any multiple or at least part of the functions of any multiple thereof can be implemented in one module. According to the modules and submodules of the embodiments of the present disclosure, any one or more thereof can be split into multiple modules for implementation. According to the modules and submodules of the embodiments of the present disclosure, any one or more thereof can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware of any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, according to the modules and submodules of the embodiments of the present disclosure, one or more thereof can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0132] For example, any multiple of the receiving module 510, the first encryption module 520, the first acquisition module 530, and the update module 540 can be combined in one module / submodule for implementation, or any one of the modules / submodules can be split into multiple modules / submodules. Alternatively, at least part of the functions of one or more of these modules / submodules can be combined with at least part of the functions of other modules / submodules and implemented in one module / submodule. According to an embodiment of the present disclosure, at least one of the receiving module 510, the first encryption module 520, the first acquisition module 530, and the update module 540 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a suitable combination of any of them. Alternatively, at least one of the receiving module 510, the first encryption module 520, the first acquisition module 530 and the updating module 540 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be performed.

[0133] It should be noted that the data processing device part in the embodiments of the present disclosure corresponds to the data processing method part in the embodiments of the present disclosure. The description of the data processing device part specifically refers to the data processing method part, which will not be repeated here.

[0134] Figure 6 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure is schematically shown. Figure 6The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0135] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0136] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0137] According to an embodiment of the present disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The system 600 may further include one or more of the following components connected to the input / output (I / O) interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0138] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0139] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0140] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0141] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 602 and / or the RAM 603 described above and / or one or more memories other than the ROM 602 and the RAM 603 .

[0142] The embodiments of the present disclosure also include a computer program product, which includes a computer program, and the computer program contains program code for executing the method provided by the embodiments of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the method provided by the embodiments of the present disclosure.

[0143] When the computer program is executed by the processor 601, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0144] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0145] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0146] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments and / or claims of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0147] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data processing method, comprising: In the case where the sending direction sends message data with the same data content to the first model system and the second model system respectively, receiving the first message data sent by the first model system and the second message data sent by the second model system, wherein there is a time difference between the first message data and the second message data in the timing sequence; Among the first message data and the second message data, the first received message data is determined as the first target message data, and the first target message data is encrypted to obtain the first encrypted message data, wherein the first encrypted message data includes the first primary key value; When it is determined that there is a historical encrypted value corresponding to the first primary key value in the first cache area, obtaining historical message data corresponding to the first primary key value stored in the second cache area; When it is determined that the first encrypted message data is inconsistent with the historical message data, the fields in the historical message data that are inconsistent with the first encrypted message data are updated, and the updated fields in the historical message data are transmitted to the message queue to form new message data.

2. The method according to claim 1, wherein: The first encrypted message data also includes a first encrypted value corresponding to the first primary key value, and the method further includes: Determining that if there is no historical encrypted value corresponding to the first primary key value in the first cache area, storing the first primary key value and the first encrypted value in the first cache area; The first primary key value and the first encrypted message data corresponding to the first encryption value are stored in the second buffer area, and the first encrypted message data is transmitted to the message queue.

3. The method according to claim 1, wherein: Encrypting the first target message data to obtain first encrypted message data includes: Acquire field information of the first target message data; Based on an encryption algorithm, the field information is encrypted to obtain the first encrypted message data.

4. The method according to claim 1, before determining that there is a historical encrypted value corresponding to the first primary key value in the first cache area, further comprising: According to the first primary key value of the first encrypted message data, a historical encryption value corresponding to the first primary key value in the first cache area is queried.

5. The method according to claim 2, before acquiring the historical message data corresponding to the first primary key value stored in the second cache area, further comprises: The first primary key value and the first encrypted value corresponding to the first primary key value are stored in the first cache area, and overwrite the historical encrypted value.

6. The method according to claim 2, wherein: When it is determined that the first encrypted message data is inconsistent with the historical message data, updating the fields in the historical message data that are inconsistent with the first encrypted message data, and transmitting the updated fields in the historical message data to the message queue to form new message data, including: Obtaining historical message data corresponding to the first primary key value in the second cache area; Determining, according to the first encrypted value, the first encrypted message data corresponding to the first encrypted value; Comparing the field information in the historical message data with the field information in the first encrypted message data, and determining the field information in the historical message data that is inconsistent with the first encrypted message data; The field information in the historical message data that is inconsistent with the first encrypted message data is updated, and the updated field information in the historical message data is transmitted to the message queue to form new message data.

7. The method according to claim 6, further comprising: The new message data is stored in the second buffer area.

8. The method according to claim 1, further comprising: Among the first message data and the second message data, determine the last received message data as the second target message data, encrypt the second target message data to obtain second encrypted message data, wherein the second encrypted message data includes a second primary key value; When it is determined that the historical encrypted value corresponding to the second primary key value exists in the first cache area, acquiring the historical message data corresponding to the second primary key value stored in the second cache area; When it is determined that the second encrypted message data is consistent with the historical message data, the second encrypted message data is deleted.

9. The method according to any one of claims 2 to 7, wherein: The first buffer area is used to store the historical encrypted value and the first encrypted value; the second buffer area is used to store the historical message data and the new message data.

10. A data processing device, comprising: A receiving module, configured to receive first message data sent by the first model system and second message data sent by the second model system when the sending direction sends message data with the same data content to the first model system and the second model system respectively, wherein there is a time difference between the first message data and the second message data in timing; A first encryption module is used to determine the first received message data as the first target message data in the first message data and the second message data, and encrypt the first target message data to obtain first encrypted message data, wherein the first encrypted message data includes a first primary key value; A first acquisition module, configured to acquire, when determining that a historical encrypted value corresponding to the first primary key value exists in the first cache area, historical message data corresponding to the first primary key value stored in the second cache area; An updating module is used to update the fields in the historical message data that are inconsistent with the first encrypted message data when it is determined that the first encrypted message data is inconsistent with the historical message data, and transmit the updated fields in the historical message data to the message queue to form new message data.

11. An electronic device, comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images