A business data processing system, method, device and storage medium

Abstract

An embodiment of the present invention provides a service data processing system, method, device, and storage medium. The method includes: receiving a second service data set processed by a first processing link, where the second service data set is data obtained by processing a first service data set, and the first service data set is obtained from a database node on the first processing link through a service intermediate node on the first processing link; receiving service update data processed by a second processing link, where the service update data is obtained by processing target service supplementary data, and the target service supplementary data is data determined according to a difference comparison between the first service data set and the second service data set. The first service data set is obtained from the database node, and the second service data set is obtained from a target search node on the first processing link, which can improve the writing efficiency and the accuracy of data transmission.

Description

Technical Field

[0001] The present invention relates to the field of Internet technologies, and in particular, to a service data processing system, method, device, and storage medium. Background Art

[0002] With the development of Internet technologies, in various service processes, a large amount of service data will be generated, and all these service data need to be transmitted to corresponding places for data storage, so as to facilitate subsequent operations such as querying the service data. For example, taking the mobile payment service as an example, with the rapid growth of commercial payments and the diversification of payment scenarios, the payment service faces higher requirements both in terms of data volume and response latency.

[0003] Currently, mainly synchronous dual writing is adopted to save payment service data in a relational database and at the same time write the payment service data into an elastic search engine, so as to facilitate subsequent querying of service data through the elastic search engine. However, in this case, synchronous dual writing requires strong data consistency, and it is necessary for the service data to be successfully written into both the relational database and the elastic search engine. However, due to various resources and resource consumptions between the devices where the database is located and the devices where the elastic search engine is located, such as inconsistent bandwidth resources, processor resources, memory resources, etc., it leads to low writing efficiency when performing synchronous dual writing. Summary of the Invention

[0004] Embodiments of the present invention provide a service data processing system, method, device, and storage medium, which can improve writing efficiency and data transmission accuracy.

[0005] In a first aspect, an embodiment of the present application provides a service data processing method, and the method includes:

[0006] Obtaining, by a service intermediate node, a first service data set stored in a database node, where the first service data set is data for a target service stored by the database node based on a data processing request, and the service intermediate node is a node configured according to the database protocol of the database node and serving as a subordinate device of the database node;

[0007] Performing data processing on the data in the first service data set to obtain a second service data set for the target service;

[0008] Sending the second service data set to the target search node, where the data in the second service data set belongs to the first service data set.

[0009] In one embodiment, a plurality of service data stored in the database node are allowed to be recorded in the service log node. Obtaining the first service data set from the database node through the service intermediate node includes:

[0010] Performing data parsing on a plurality of service data stored in the service log node through the service intermediate node to obtain a plurality of parsed service data;

[0011] Obtaining the first service data set from the plurality of parsed service data.

[0012] In one embodiment, the plurality of parsed service data are stored in the message queue of the service message queue node, and the first service data set is read from the message queue of the service message queue node by using a read thread.

[0013] In one embodiment, sending the second service data set to the target search node includes:

[0014] Obtaining connection resources for accessing the target search node from the Transmission Control Protocol (TCP) long connection pool; the connection resources include one or both of an available port number and an available communication address;

[0015] Establishing a TCP connection according to the connection resources for accessing the target search node;

[0016] Using a write thread and through the established TCP connection, sending the second service data set to the target search node.

[0017] In one embodiment, a service proxy node is configured for the target search node; establishing a TCP connection according to the connection resources for accessing the target search node includes:

[0018] Establishing a TCP connection with the service proxy node according to the connection resources for accessing the target search node;

[0019] Using a write thread and through the established TCP connection, sending the second service data set to the target search node includes:

[0020] Based on the established TCP connection, using a write thread to write the second service data set into the service proxy node to trigger the service proxy node to process the second service data set and save the processed second service data set in the target search node.

[0021] In one embodiment, the method further includes:

[0022] When the transmission of the second service data set to the target search node fails, perform service rewriting processing on the second service data set at a first time interval;

[0023] Obtain the number of failures in transmitting the second service data set to the target search node during the service rewriting process;

[0024] If the number of failures exceeds the threshold number of times, add the second service data set to the service polling queue;

[0025] Traverse the service data in the service polling queue at a second time interval, and use a write thread to rewrite the currently traversed service data to the target search node;

[0026] The second time interval is greater than the first time interval.

[0027] In one embodiment, the method further includes:

[0028] If the service data in the service polling queue includes target service data whose survival duration exceeds a preset duration, delete the target service data from the service polling queue.

[0029] In a second aspect, an embodiment of the present application provides a service data processing method, and the method includes:

[0030] Obtain target service supplementary data from a database supplementary node through a service supplementary intermediate node;

[0031] Perform data processing on the target service supplementary data to obtain service update data;

[0032] Send the service update data to a target search node;

[0033] Wherein, the target service supplementary data is data determined by a data comparison node after comparing the differences between a first service data set and a second service data set stored in the database supplementary node. The first service data set is obtained from a database node, and the second service data set is obtained from the target search node.

[0034] In one embodiment, a plurality of service supplementary data stored in the database supplementary node are allowed to be recorded in a service supplementary log node. The obtaining of the target service supplementary data from the database supplementary node through the service supplementary intermediate node includes:

[0035] Perform data parsing on a plurality of service supplementary data stored in the service supplementary log node through the service supplementary intermediate node to obtain the parsed plurality of service supplementary data;

[0036] Obtain the target service supplementary data from the parsed multiple service supplementary data.

[0037] In one embodiment, the parsed multiple service supplementary data are stored in the message queue of the service supplementary message queue node, and the target service supplementary data is read from the message queue of the service supplementary message queue node by a read thread.

[0038] In one embodiment, sending the service update data to the target search node includes:

[0039] Obtain connection resources for accessing the target search node from the Transmission Control Protocol (TCP) long connection pool; the connection resources include one or both of an available port number and an available communication address;

[0040] Establish a TCP connection according to the connection resources for accessing the target search node;

[0041] Use a write thread and send the service update data to the target search node through the established TCP connection.

[0042] In one embodiment, a service supplementary proxy node is configured for the target search node; establishing a TCP connection according to the connection resources for accessing the target search node includes:

[0043] Establish a TCP connection with the service supplementary proxy node according to the connection resources for accessing the target search node;

[0044] The step of using a write thread and sending the service update data to the target search node through the established TCP connection includes:

[0045] Based on the established TCP connection, use a write thread to write the service update data to the service supplementary proxy node to trigger the service supplementary proxy node to process the service update data, and save the processed service update data in the target search node.

[0046] In one embodiment, the method further includes:

[0047] When the sending of the service update data to the target search node fails, perform supplementary service rewriting processing on the service update data at a first time interval;

[0048] Obtain the number of failures of sending the service update data to the target search node during the supplementary service rewriting processing;

[0049] If the number of failures exceeds the number threshold, add the service update data to the supplementary service polling queue;

[0050] Traverse the service update data in the supplementary service polling queue according to a second time interval, and use a write thread to rewrite the traversed service update data to the target search node;

[0051] The second time interval is greater than the first time interval.

[0052] In one embodiment, the method further includes:

[0053] If the service update data in the supplementary service polling queue includes target service update data whose survival duration exceeds a preset duration, delete the target service update data from the supplementary service polling queue.

[0054] In a third aspect, an embodiment of the present application provides a service data processing method, and the method includes:

[0055] Receive a second service data set processed by a first processing link, where the second service data set is data obtained by processing a first service data set, and the first service data set is obtained by a service intermediate node on the first processing link from a database node on the first processing link. The service intermediate node is a node configured according to the database protocol of the database node and serving as a subordinate device of the database node;

[0056] Receive service update data processed by a second processing link, where the service update data is obtained by processing target service supplementary data, and the target service supplementary data is data determined according to a difference comparison between the first service data set and the second service data set. The first service data set is obtained from the database node, and the second service data set is obtained from a target search node on the first processing link.

[0057] In one embodiment, multiple service data stored in the database node are allowed to be recorded in a service log node, and the first service data set is read by a service transmission node using a read thread from a message queue of a service message queue node; multiple parsed service data are stored in the message queue of the service message queue node, and the multiple parsed service data are obtained by the service intermediate node parsing the multiple service data stored in the service log node; the service log node, the service transmission node, and the service message queue node belong to the first processing link;

[0058] Multiple service supplementary data stored in the database supplementary node are allowed to be recorded in the service supplementary log node. The target service supplementary data is obtained by the supplementary service transmission node using a read thread to read from the message queue of the service supplementary message queue node. The message queue of the service supplementary message queue node stores multiple parsed service supplementary data, and the multiple parsed service supplementary data are obtained by the service supplementary intermediate node parsing the multiple service supplementary data stored in the service supplementary log node. The database supplementary node, the service supplementary log node, the supplementary service transmission node, the service message queue node, and the service supplementary intermediate node belong to the second processing link.

[0059] In one embodiment, the first processing link includes a TCP connection. The TCP connection is established according to the connection resources of the target search node accessing the first processing link. The connection resources include one of an available port number or an available communication address. The connection resources of the target search node accessing the first processing link are obtained from the TCP long connection pool. The second service data set is sent by the service transmission node on the first processing link using a write thread and through the TCP connection.

[0060] In one embodiment, a service proxy node is configured for the target search node. The TCP connection is established with the service proxy node according to the connection resources for accessing the target search node. The second service data set is sent by the service transmission node using a write thread and through the TCP connection to the service proxy node.

[0061] In one embodiment, the second processing link includes a TCP connection. The TCP connection is established according to the connection resources of the target search node accessing it. The connection resources of the target search node accessing it are obtained from the TCP long connection pool. The service update data is sent by the supplementary service transmission node on the second processing link using a write thread and through the TCP connection.

[0062] In one embodiment, a service supplementary proxy node is configured for the target search node. The TCP connection is established with the service supplementary proxy node according to the connection resources for accessing the target search node. The service update data is sent by the supplementary service transmission node using a write thread and through the TCP connection to the service supplementary proxy node.

[0063] Fourthly, an embodiment of the present application provides a service data processing device, and the device includes:

[0064] A service processing unit, configured to obtain a first service data set stored in a database node through a service intermediate node. The first service data set is data for a target service stored by the database node based on a data processing request. The service intermediate node is a node configured according to the database protocol of the database node and serving as a subordinate device of the database node;

[0065] The service processing unit is further configured to process the data in the first service data set to obtain a second service data set for the target service;

[0066] A service communication unit, configured to send the second service data set to the target search node, where the data in the second service data set belongs to the first service data set.

[0067] In a fifth aspect, an embodiment of the present application provides a service data processing device, including:

[0068] A service supplementary processing unit, configured to obtain target service supplementary data from a database supplementary node through a service supplementary intermediate node;

[0069] The service supplementary processing unit is further configured to process the target service supplementary data to obtain service update data;

[0070] A service supplementary communication unit, configured to send the service update data to a target search node;

[0071] Wherein, the target service supplementary data is data stored in the database supplementary node and determined by a data comparison node after comparing the differences between the first service data set and the second service data set. The first service data set is obtained from a database node, and the second service data set is obtained from the target search node.

[0072] In a sixth aspect, an embodiment of the present application provides a service data processing device, including:

[0073] A communication unit, configured to send and receive data;

[0074] A processing unit, configured to receive, through the communication unit, a second service data set processed by a first processing link. The second service data set is data obtained by processing a first service data set. The first service data set is obtained from a database node on the first processing link through a service intermediate node on the first processing link. The service intermediate node is a node configured according to the database protocol of the database node and serving as a subordinate device of the database node;

[0075] The processing unit is further configured to receive, through the communication unit, service update data processed by a second processing link, where the service update data is obtained by processing target service supplementary data, and the target service supplementary data is data determined according to a difference comparison between the first service data set and the second service data set. The first service data set is obtained from the database node, and the second service data set is obtained from a target search node on the first processing link.

[0076] In a seventh aspect, an embodiment of the present application provides a service data processing system, where the system includes:

[0077] The database node is configured to store service data for a target service based on a data processing request, where the stored service data for the target service includes a first service data set;

[0078] The service intermediate node is configured to obtain the first service data set from the database node;

[0079] The service transmission node is configured to obtain the first service data set from the database node through the service intermediate node, perform data processing on the data in the first service data set to obtain a second service data set of the target service; and send the second service data set to a target search node, where the data in the second service data set belongs to the first service data set.

[0080] In an eighth aspect, an embodiment of the present application provides an intelligent device, where the intelligent device includes: a storage device and a processor; the storage device is configured to store a computer program; the processor is configured to execute the computer program to implement the method described in the first aspect above or to implement the method described in the second aspect above or to implement the method described in the third aspect above.

[0081] In a ninth aspect, an embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed, the method described in the first aspect above or the method described in the second aspect above or the method described in the third aspect above will be implemented.

[0082] In a tenth aspect, an embodiment of the present application further provides a computer program product or a computer program, where the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of an intelligent device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the intelligent device executes the method described above.

[0083] In the embodiment of the present application, a second set of service data processed by a first processing link is received, where the second set of service data is data obtained by processing a first set of service data, and the first set of service data is obtained by a service intermediate node on the first processing link from a database node on the first processing link; through the first processing link, the data in the database node can be directly saved to the target search node without the participation of other links, improving the accuracy of data transmission; a service update data processed by a second processing link is received, and the service update data is obtained by processing target service supplementary data, and the target service supplementary data is data determined according to the difference comparison between the first set of service data and the second set of service data. Through the second processing link, the data in the database node and the data in the target search node are compared for data differences. When it is determined that there are differences in the data, the service update data corresponding to the different data is saved to the target search node again, so that the data in the database node and the search device can be better made consistent, which is beneficial to the accuracy of subsequent data queries. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings.

[0085] Figure 1 It is an architecture diagram of a service data processing system provided by an embodiment of the present invention;

[0086] Figure 2 It is a schematic flowchart of a service data processing method provided by an embodiment of the present invention;

[0087] Figure 3 It is a schematic structural diagram of a service transmission node provided by an embodiment of the present invention;

[0088] Figure 4 It is a schematic flowchart of a service data processing method provided by an embodiment of the present invention;

[0089] Figure 5 It is a schematic flowchart of a service data processing method provided by an embodiment of the present invention;

[0090] Figure 6a It is a schematic flowchart of a service data processing solution provided by an embodiment of the present invention;

[0091] Figure 6bIt is a schematic flowchart of a service data processing solution provided by an embodiment of the present invention;

[0092] Figure 6c It is a schematic flowchart of a service data processing solution provided by an embodiment of the present invention;

[0093] Figure 7 It is a schematic structural diagram of a service data processing device provided by an embodiment of the present invention;

[0094] Figure 8 It is a schematic structural diagram of a service data processing device provided by an embodiment of the present invention;

[0095] Figure 9 It is a schematic structural diagram of a service data processing device provided by an embodiment of the present invention;

[0096] Figure 10 It is a schematic structural diagram of an intelligent node provided by an embodiment of the present invention. Detailed implementation manners

[0097] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0098] The embodiment of the present application provides a business data processing solution. This business data processing solution mainly realizes data transmission and data storage of business data through two processing links. By using the two processing links, while improving the efficiency and accuracy of data writing and data transmission, it also better ensures the accuracy of data storage and subsequent data query. Among them, the first processing link mainly sets multiple nodes between the database node and the target search node, and can directly and relatively accurately transmit the business data saved in the database to the target search node. Among the multiple nodes in the first processing link, there are database nodes, business intermediate nodes, business transmission nodes, and business message queue nodes. The business intermediate node can disguise itself as a node of a subordinate device in the database node to obtain the business data in the database node. Then, the business intermediate node can parse the obtained business data and push the parsed business data into the business message queue node. Then, the business transmission node reads the parsed business data in the business message queue node, processes the read parsed business data, and sends it to the target search node. Through the first processing link, when directly writing business data into the database node, the business data in the database node can be directly transmitted to the target search node, making data transmission simple, efficient, and at the same time, better improving the accuracy of data transmission and the efficiency of data writing.

[0099] The second processing link mainly checks the data in the target search node and the data in the database node. If it is found that the data in the target search node is missing compared with the data in the database node, or if it is found that the data in the target search node is inconsistent with the data in the database node, then the data in the database node shall prevail, and target business supplementary data shall be obtained from the supplementary database node of the second processing link. Remedial measures shall be initiated for the target business supplementary data through the second processing link, and then the target business supplementary data shall be transmitted to the target search node through each node on the second processing link. By correcting the data, the data in the database node and the search device can be better made consistent, thereby improving the accuracy of business data query. In one embodiment, a Socket pool (socket pool) (subsequently referred to as a TCP (Transmission Control Protocol) long connection pool) and a delayed failure queue are pre-constructed in the business transmission node and the supplementary business transmission node, etc., to reduce latency problems and improve the efficiency of data writing and the efficiency of data transmission.

[0100] Based on the above-provided business data processing solution, please refer to Figure 1 , Figure 1An architecture diagram of a service data processing system provided by an embodiment of the present invention. The service data processing system may include a service system, a database node 101, a service intermediate node 103, a service transmission node 105, and a target search node 106. In one embodiment, the service data processing system may further include a data comparison node 107, a database supplement node 108, a service supplement intermediate node 110, and a supplementary service transmission node 112. Among them, the service intermediate node 103 is configured according to the database protocol of the database node 101 and is a node subordinate to the database node 101; in one embodiment, the service processing system may further include a service log node 102, a service supplement log node 109, a service message queue node 104, and a service supplement message queue node 111. In one embodiment, the service data processing system further includes a service proxy node 1061 and a service supplement proxy node 1062 configured for the target search node 106. The above service intermediate node 103 and service supplement intermediate node 110 may be the same intermediate node or different intermediate nodes; the above service transmission node 105 and the above supplementary service transmission node 112 may be the same transmission node or different transmission nodes; the above service log node 102 and service supplement log node 109 may be the same log node or different log nodes; the above service message queue node 104 and service supplement message queue node 110 may be the same message queue node or different message queue nodes, and the service proxy node 1061 and service supplement proxy node 1062 may also be the same proxy node or different proxy nodes; the embodiments of the present application do not limit this.

[0101] The aforementioned database node 101, business log node 102, business intermediate node 103, business message queue node 104, business transmission node 105, and target search node 106 can be communicatively connected via a mobile communication network, a computer network, etc., and form a first processing link; the aforementioned database node 101, data comparison node 107, database supplement node 108, business supplement log node 109, business supplement intermediate node 110, business supplement message queue node 111, supplement business transmission node 112, and target search node 106 are also communicatively connected via a mobile communication network, a computer network, etc., and form a second processing link. It should be noted that: the nodes involved in the embodiments of the present application may be components (middleware), that is, the nodes involved mainly refer to some middleware for corresponding processing. These middleware are deployed to corresponding intelligent devices to execute corresponding steps to implement corresponding functions. Each node involved in the embodiments of the present application may also be an intelligent device, etc. The intelligent device may be a terminal or a server; the aforementioned terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The aforementioned server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0102] The business system can send a data processing request to the database node 101, and the data processing request carries the business data of the target business. The data processing request may be a data storage request, a data modification request, etc. for the target business. The target business may be a payment business, a refund business, etc., and of course, it may also be some other businesses, such as a business for recording product information in an e-commerce platform, a business for recording services that a store can provide, etc.

[0103] The database node 101 can be used to receive the data processing request sent by the business system and store the business data for the target business based on the data processing request. Among them, the business data for the target business stored in the database 101 includes a first business data set. Among them, the database node 101 may be a relational database MySQL; the database node 101 is used to process and store the business data for the target business according to the data processing request. The processing here may be, for example: converting the format of the business data of the target business so that the business data set of the target business can be stored in the database node 101.

[0104] The service log node 102 can be a binary log node (such as the binlog node of MySQL), and this service log node 102 is used to record multiple service data stored in the database node 101. Specifically, the service log node is used to store multiple service data as the first log information, and the multiple service data can be part of or all of the data stored in the database node 101. The part of the data can refer to some service data in the database node 101 that meets the preset time range. Among them, the preset time can be set according to requirements. For example, when the database node 101 is used to store service data for the target service based on a data processing request, the first log information of the service log node 102 is used to record the service data for the target service stored in the database node 101.

[0105] The service intermediate node 103 is used to obtain multiple service data recorded by the service log node 102, parse the recorded multiple service data to obtain the parsed multiple service data, and send the parsed multiple service data to the service message queue node 104. The service intermediate node 103 can be a data synchronization and backup tool DBSync (a database synchronization and backup tool). DBSync can be used to obtain the service data that has changed in the database node 101 recorded by the service log node 102. At this time, it can be understood that: the multiple service data are the service data that has changed in the database node 101. In specific implementation, the service intermediate node 103 can be used to send a dump command (a command for backup) to the service log node to obtain multiple service data with changes recorded by the service log node. In one embodiment, since the service data recorded in the service log all have timestamps; therefore, the service intermediate node 103 can also be used to obtain multiple service data that meet the preset time range from the service log node 102 according to the timestamp of each service data, parse the multiple service data that meet the preset time range, and store the parsed multiple service data that meet the preset time range in the message queue of the service message queue.

[0106] The service message queue node 104 is used to store the parsed multiple service data. The service message queue node 104 can be HIPPO. HIPPO is a new generation of message system (or message queue), which can meet the service requirements of application scenarios with high reliability and high availability, and is used to support services of high-value data such as advertising billing and transaction flow. Among them, the parsed multiple service data can be obtained by parsing the multiple service data that meet the preset time range.

[0107] The service transmission node 105 is used to obtain the first service data set from the database node 101 through the service intermediate node 103, then process the first service data set to obtain the second service data set of the target service, and then send the second service data set to the target search node 106. Among them, the data in the second service data set belongs to the first service data set; the data processing here can be, for example, the conversion of the data format, so that the format of the service data in the database node is converted into a data format that the target search node can store. It should be noted that the above data processing can also be other processing methods, which are not limited in the embodiments of the present application. The service transmission node 105 can be a DTS (Data Transmission Service) device, which can be used for data migration between homogeneous or heterogeneous storage nodes.

[0108] In one embodiment, when obtaining the first service data set from the database node 101 through the service intermediate node 103, the service transmission node 105 is used to parse multiple service data stored in the service log node 102 through the service intermediate node 103 to obtain the parsed multiple service data, and then the service transmission node 105 obtains the first service data set from the parsed multiple service data. In one embodiment, the first service data set may include all or part of the multiple service data after parsing. In one embodiment, the service transmission node 105 can obtain the first service data set within a target time range from the parsed multiple service data. In specific implementation, the parsed multiple service data carry timestamps, and the parsed service data whose timestamps indicate times within the target time range are added to the first service data set. For example, the parsed multiple service data are service data 1 and service data 2 respectively, the time indicated by the timestamp of service data 1 is 11 o'clock, and the time indicated by the timestamp of service data 2 is 11:20; the preset time range is 10 o'clock - 12 o'clock; then the service transmission node 105 determines that both service data 1 and service data 2 are added to the first service data set.

[0109] In one embodiment, the parsed multiple service data are stored in the message queue of the service message queue node 104, and the service transmission node 105 can be used to obtain the first service data set from the message queue of the service message queue node 104 by using a read thread.

[0110] In one embodiment, when the service transmission node 105 sends the second service data set to the target search node 106, it is used to obtain connection resources for accessing the target search node from the TCP long connection pool, and the connection resources include one of an available port number or an available communication address. Then, the service transmission node 105 establishes a TCP connection according to the connection resources for accessing the target search node, and uses the write thread in the service transmission node 105 to send the second service data set to the target search node 106 through the established TCP connection. Among them, the TCP long connection pool includes many connection resources for accessing the target search node 106. In practical applications, any connection resource for accessing the target search node 106 can be selected from the TCP connection pool. The available communication address can be an IP address (Internet Protocol Address), a MAC address (Media Access Control Address), and so on.

[0111] In one embodiment, as can be seen from the foregoing, a service proxy node 1601 is configured for the target search node. When the service transmission node 105 establishes a TCP connection according to the connection resources for accessing the target search node, it can specifically be used to: establish a TCP connection with the service proxy node 1601 according to the connection resources for accessing the target search node. Then, using the write thread and through the established TCP connection, send the second service data set to the target search node, specifically used to: based on the established TCP connection, use the write thread to write the second service data set to the service proxy node 1601 to trigger the service proxy node 1601 to process the second service data set, and save the processed second service data set in the target search node 106.

[0112] In one embodiment, the service transmission node 105 is configured to, when the transmission of the second service data set to the target search node 106 fails, perform service rewriting processing on the second service data set at a first time interval, and obtain the number of failures in transmitting the second service data set to the target search node 106 during the service rewriting process; then the service transmission node 105 may determine whether the number of failures exceeds a threshold number. If it is determined that the number of failures exceeds the threshold number, the second service data set is added to the service polling queue, and the service data in the service polling queue is traversed at a second time interval, and the current traversed service data is rewritten to the target search node using a write thread. Among them, the second time interval is greater than the first time interval; the first time interval can be understood as a very short time interval, and the first time interval can be 1S, 4S, etc. The second time interval can be set according to requirements, and the second time interval can be 30 minutes, 40 minutes, etc. In one embodiment, the service transmission node 105 is configured to, if the current traversed service data is successfully rewritten to the target search node 106 using the write thread, remove the current traversed service data from the service polling queue.

[0113] In one embodiment, the service transmission node 105 is further configured to determine whether the service data in the service polling queue includes target service data whose survival duration exceeds a preset duration. If the service data in the service polling queue includes target service data whose survival duration exceeds the preset duration, the target service data is deleted from the service polling queue. Among them, the preset duration can be set according to requirements. For example, the preset duration can be 1 day, 2 days, etc. The survival duration can be understood as the effective duration of the service data in the service polling queue. The survival duration of any service data in the service polling queue refers to the duration from the time when any service data is added to the service polling queue to the current time. For example, any service data is data A; the time when data A is added to the service polling queue is 11:20, and the current time is 12:00; the survival duration of data A is 40 minutes.

[0114] In one embodiment, if it is determined that the service data in the service polling queue does not include target service data whose survival duration exceeds the preset duration, the service transmission node 105 continuously traverses the service data in the service polling queue, so that the service data in the service polling queue can be rewritten to the target search node within the validity period; of course, there may be a situation where some or all of the service data in the service polling queue fails to be rewritten within the validity period, and the part of the data that fails to be rewritten within the validity period will be deleted (that is, the validity period refers to within the preset duration).

[0115] The service proxy node 1061 is used to receive the second service data set sent by the service transmission node 105 based on a TCP connection, process the service data in the received second service data set, and save the processed service data set in the target search node 106. Here, the processing can be, for example, slicing and table partitioning the service data in the second service data set according to the timestamps of the service data in the second service data set. The TCP connection means: establishing a long-term communication connection with the service transmission node 105 based on the connection resources accessing the target search node.

[0116] The target search node 106 is used to receive the second service data set sent by the service transmission node 105, or the target search node 106 is used to save the second service data set processed by the service proxy node 1061; in one embodiment, the target search node 106 is used to receive the service update data sent by the supplementary service transmission node, or the target search node 106 is used to save the service update data processed by the service supplementary proxy node 1062. Among them, the target search node 106 can be ES (Elasticsearch, a distributed full-text search engine).

[0117] The data comparison node 107 is used to check whether the data in the target search node 106 and the database node 101 is consistent; the data comparison node 107 can be ESCheck (a node for data comparison). In one embodiment, the data comparison node 107 is used to obtain the first service data set and the second service data set according to the timestamps; and perform a data difference comparison on the first service data set and the second service data set to obtain the target service supplementary data. Among them, the first service data set is obtained from the database node 101, and the second service data set is obtained from the target search node 106. There are two reasons for data differences: (1) The data in the second service data set is inconsistent with the data in the first service data set, and it is possible that the data in the second service data set has an abnormality during transmission; (2) The data in the first service data set is not stored in the second service data set. In this case, the target service supplementary data is based on the service data in the first service data set; for example, the service data in the first service data set includes payment A; the service data in the second service data set includes payment B, and the data comparison node 107 is used to perform a data difference comparison on payment A and payment B to obtain the target service supplementary data as payment A. Another example is that the service data in the first service data set includes payment A; the service data in the second service data set does not include payment A, and the data comparison node 107 is used to perform a data difference comparison on the first service data set and the second service data set to obtain the target service supplementary data as payment A.

[0118] In one embodiment, since the formats of the service data in the first service data set and the service data in the second service data set are different, it is difficult to perform data difference comparison. Therefore, when the data comparison node 107 performs data difference comparison on the first service data set and the second service data set to obtain the target service supplementary data, the data comparison node 107 is used to convert the format of the first service data set into the Json format, convert the format of the second service data set into the Json format, and perform MD5 (Message-Digest Algorithm) calculation on the first service data set converted into the Json format to obtain the first MD5 value, and perform MD5 calculation on the second service data set converted into the Json format to obtain the second MD5 value. Then, the data comparison node 107 compares whether the first MD5 value and the second MD5 value are consistent. If the first MD5 value and the second MD5 value are inconsistent, data difference comparison is performed on the first service data set and the second service data set to determine the target service supplementary data.

[0119] The database supplementary node 108 is used to store multiple service supplementary data so that the subsequent supplementary service transmission node 112 can obtain the service data to be supplemented from the database supplementary node 108. The database supplementary node 108 can be MySQL. Among them, the service data to be supplemented is consistent with the corresponding service data in the database node. In one embodiment, the database supplementary node 108 is used to store the above-mentioned target service supplementary data. As in the above example, the database supplementary node 108 is used to store Payment A.

[0120] The service supplementary log node 109 is used to record multiple service supplementary data stored by the database supplementary node 108. The service supplementary log node 109 can be a binary log node (such as a binlog node). Specifically, the service supplementary log node is used to store multiple service supplementary data in the second log information. The multiple service supplementary data can be part of or all of the data in the database supplementary node 108. For example, the multiple service supplementary data can be some service supplementary data in the database supplementary node 108 that meet the preset time range.

[0121] The service supplement intermediate node 110 is used to obtain multiple service supplement data records by the service supplement log node 109, parse the multiple service supplement data records obtained, obtain the parsed multiple service supplement data, and send the parsed multiple service supplement data to the service supplement message queue node 111. Among them, the service supplement intermediate node 110 can be DBSync (database synchronization tool). DBSync can be used to obtain the service supplement data that has changed in the database supplement node 108 recorded by the service supplement log node 109. At this time, it can be understood that: the multiple service supplement data is the service supplement data that has changed in the database supplement node 108. In specific implementation, the service supplement intermediate node 110 can send a dump command to the service supplement log node 109 to obtain the multiple service supplement data that has changed recorded by the service supplement log node 109. Among them, the multiple service supplement data can be part of the data in the service supplement log node 109. For example, the service supplement intermediate node 110 can obtain multiple service supplement data that meet the preset time range from the service supplement log node 109 according to the time stamp. The specific implementation method for the service supplement intermediate node 110 to obtain multiple service supplement data that meet the preset time range from the service supplement log 109 according to the time stamp can refer to the specific implementation method of the above service intermediate node 103 to obtain multiple service data that meet the preset time range from the service log node 102 according to the time stamp of each service data, which will not be elaborated here.

[0122] The service supplement message queue node 111 is used to store the parsed multiple service supplement data. The service supplement message queue node 111 can be HIPPO.

[0123] The supplementary service transmission node 112 is used to parse multiple service supplementary data stored in the service supplementary log node 109 through the service supplementary intermediate node 110 to obtain the parsed multiple service supplementary data, and obtain the target service supplementary data from the parsed multiple service supplementary data. The target service supplementary data can be one or more. In one embodiment, the supplementary service transmission node 112 can obtain the target service supplementary data within the target time range from the parsed multiple service supplementary data. Specifically, each parsed service supplementary data corresponds to a timestamp, and then according to the time indicated by the timestamp of each parsed service supplementary data, the target service supplementary data within the target time range is obtained from the parsed multiple service supplementary data. The target time range can be set according to actual needs. It should be noted that if the above parsed multiple service supplementary data is obtained by parsing the service supplementary data that meets the preset time range, then the target time range is less than or equal to the preset time range. The supplementary service transmission node 112 can also be a DTS device. The supplementary service transmission node 112 and the service transmission node can be the same node or different nodes.

[0124] In one embodiment, the parsed multiple service supplementary data is stored in the message queue of the service supplementary message queue 111, and the supplementary service transmission node 112 is used to use a read thread to read the target service supplementary data from the message queue of the service supplementary message queue 111.

[0125] In one embodiment, the supplementary service transmission node 112 is used to obtain connection resources for accessing the target search node 106 from the Transmission Control Protocol (TCP) long connection pool. The connection resources include one of an available port number or an available communication address. A TCP connection is established according to the connection resources for accessing the target search node. Then, using a write thread and through the established TCP connection, service update data is sent to the target search node 106.

[0126] In one embodiment, when establishing a TCP connection based on the connection resources for accessing the target search node, the supplementary service transmission node 112 is used to: establish a TCP connection with the service supplementary proxy node according to the connection resources for accessing the target search node. When using a write thread and through the established TCP connection to send service update data to the target search node, the supplementary service transmission node 112 is used to: based on the established TCP connection, use a write thread to write the service update data to the service supplementary proxy node, so that the service supplementary proxy node processes the service update data and saves the processed service update data in the target search node 106.

[0127] In one embodiment, the supplementary service transmission node 112 is further configured to: when the transmission of service update data to the target search node fails, perform supplementary service rewriting processing on the service update data at a first time interval; obtain the number of failures in sending the service update data to the target search node 106 during the supplementary service rewriting process; and determine whether the number of failures exceeds a failure threshold; if the supplementary service transmission node 112 determines that the number of failures exceeds the number threshold, add the service update data to the supplementary service polling queue; and traverse the service update data in the supplementary service polling queue at a second time interval, and use a write thread to rewrite the currently traversed service update data to the target search node 106. Wherein, the second time interval is greater than the first time interval; the first time interval can be understood as a very short time interval, and the first time interval can be 1 second, 4 seconds, etc. The second time interval can be set according to requirements, and the second time interval can be 30 minutes, 40 minutes, etc. If the rewriting of the currently traversed service update data to the target search node 106 exceeds a certain time threshold, such as 24 hours, the currently traversed service update data can be directly discarded, and different preset durations can be set according to the attribute information corresponding to the currently traversed service update data. When the attribute information corresponding to the currently traversed service update data indicates that the service to which the currently traversed service update data belongs is marked as an important service, the preset duration set is the first duration. When the attribute information corresponding to the currently traversed service update data indicates that the service to which the currently traversed service update data belongs is marked as an ordinary service, the preset duration set is the second duration, and the first duration is greater than the second duration.

[0128] In one embodiment, the supplementary service transmission node 112 is further configured to: determine whether the service update data in the supplementary service polling queue includes target service update data whose survival duration exceeds a preset duration; if the supplementary service transmission node 112 determines that the service update data in the supplementary service polling queue includes target service update data whose survival duration exceeds the preset duration, then delete the target service update data from the supplementary service polling queue. The survival duration here can be understood as the effective duration of the service update data in the supplementary service polling queue. The survival duration of the service update data in the supplementary service polling queue refers to the duration from the time when the service update data is added to the supplementary service polling queue to the current time. For example, the service update data is data A; the time when data A is added to the service polling queue is 11:20, and the current time is 12:00; the survival duration of data A is 40 minutes. If the survival duration of the target service update data exceeds a certain time threshold, such as 24 hours, the target service update data can be directly discarded. Different preset durations can be set according to the attribute information corresponding to the target service update data. When the attribute information corresponding to the target service update data indicates that the service to which the target service update data belongs is marked as an important service, the preset duration set is the first duration. When the attribute information corresponding to the target service update data indicates that the service to which the target service update data belongs is marked as an ordinary service, the preset duration set is the second duration, and the first duration is greater than the second duration. At this time, it can be understood that if the survival duration of the target service update data exceeds the preset duration, it can be discarded in advance.

[0129] In one embodiment, if the supplementary service transmission node 112 determines that the service update data in the supplementary service polling queue does not include target service update data whose survival duration exceeds the preset duration, then the supplementary service transmission node 112 can continue to poll the service update data in the supplementary service polling queue at a second time interval, so that the service update data in the supplementary service polling queue can be rewritten into the target search node within the validity period; of course, there will be a situation where some or all of the service update data in the supplementary service polling queue fails to be rewritten within the validity period, and the partially failed data will be deleted.

[0130] The service supplementary proxy node 1062 is configured to receive the service update data sent by the supplementary service transmission node based on a TCP connection, process the received service update data, and save the processed service update data in the target search node. Here, the processing can be, for example, slicing and tabling the service update data according to the timestamp of the service update data. The TCP connection means: establishing a long-term communication connection with the supplementary service transmission node 112 based on the connection resources used to access the target search node.

[0131] It should be noted that the business log nodes and business supplementary log nodes in the embodiments of the present application; the business intermediate nodes and business supplementary intermediate nodes; the business message queue nodes and business supplementary message queue nodes; the functions of the business transmission nodes and supplementary business transmission nodes are similar and can be referred to each other.

[0132] In one embodiment, since a large amount of business data will be generated for the target business. For example, if the target business is the mobile payment business, a large number of users will conduct mobile payments every day, which is likely to cause problems in system performance. Therefore, in order to save the time for obtaining business data and solve the bottleneck of system performance, the embodiments of the present application adopt concurrent clocks at each node. A copy of the thread-local variable can be created for each thread through ThreadLocal (thread-local variable) in Java. When accessing, each thread can only access its own copy variable, thereby implementing concurrent clocks. Asynchronous logging means that when writing business data into the business log of the corresponding node, all business data should be saved as much as possible when meeting business requirements, and at the same time, data irrelevant to business requirements can be not saved.

[0133] In the embodiments of the present application, the business intermediate node 103 obtains the first business data set stored in the database node 101, and processes the data in the first business data set to obtain the second business data set of the target business. Then, the second business data set is sent to the target search node 106. Instead of writing to the target search node 106 simultaneously when writing to the database node 101, after writing to the database node 101, the data of the database node is directly written to the target search node 106 through the business intermediate node 103 and the business transmission node 105. While improving the data writing efficiency, it also better ensures the accuracy of data transmission. In addition, the present application also uses a data comparison node to compare the data differences between the first business data set and the second business data set, and obtains the target business supplementary data from the database supplementary node according to the data differences. Then, the target business supplementary data is processed to obtain business update data, and the business update data is sent to the target search node. Through data difference comparison, when it is determined that there are data differences, and the business update data is re-saved to the target search node, the consistency and accuracy of the data in the database node and the data in the target search node can be made, which is beneficial to the accuracy of subsequent data queries.

[0134] Based on the above business data processing system, please refer to Figure 2 , Figure 2Schematic flowchart of a service data processing method provided by an embodiment of the present invention. The service data processing method can be executed by the service transmission node 105 in the above service data processing system, and the service transmission node 105 can be a DTS device. The service data processing method described in this embodiment includes the following steps S201 - S203:

[0135] S201: Obtain a first set of service data stored in the database node through the service intermediate node. The first set of service data is data for the target service stored by the database node based on a data processing request. The service intermediate node is a node configured according to the database protocol of the database node and serving as a subordinate device of the database node. Here, the service intermediate node can be understood as a node that disguises itself as a subordinate device of the database node to obtain multiple sets of service data stored in the database node. From the perspective of the main device of the database node, the service intermediate node is only a node of the subordinate device of the database node.

[0136] In one embodiment, multiple sets of service data stored in the database node are allowed to be recorded in the service log node. The specific implementation manner of step S201 is as follows: The service transmission node can parse multiple sets of service data stored in the service log node through the service intermediate node to obtain the parsed multiple sets of service data. Then, obtain the first set of service data from the parsed multiple sets of service data. Specifically, the service intermediate node can pull multiple sets of service data recorded in the service log node, and then parse the multiple sets of service data stored in the service log node through the database log protocol. Among them, if the database node is MySQL, the database log protocol is the binlog protocol of MySQL. Then the service transmission node can obtain the parsed multiple sets of service data and obtain the first set of service data from the parsed multiple sets of service data.

[0137] In the embodiment of the present application, for traditional message queue reading, generally the same thread is used to handle read and write operations. One disadvantage of this is that reading and writing restrict each other, and it is easy to encounter bottlenecks in scenarios where the read and write performance is inconsistent. Moreover, in the embodiment of the present application, the writing time consumption is much greater than the reading time consumption. This is because there are flow limiting measures in the target search node (such as the bill ES), and the underlying server does not use SSD (Solid State Disk or Solid State Drive) but a mechanical disk, resulting in many jitter phenomena, and the upstream needs to keep retrying. Therefore, in the embodiment of the application, the architecture of the service transmission node is optimized. The structure of the service transmission node is as Figure 3 shown. The service transmission node uses a read-write separation design, separating the read thread and the write thread. The read thread is only responsible for data reading, and the write thread is only responsible for data writing. This design greatly improves the efficiency and can achieve quasi-real-time timeliness.

[0138] In one embodiment, the parsed multiple service data are stored in the message queue of the service message queue node, and the service transmission node can use a read thread to read a first service data set from the message queue of the service message queue node. For example, in Figure 3 , the message queue of the service message queue node includes Queue 1, Queue 2, Queue 3... Queue n; then use a read thread to read a first service data set from the message queue of the service message queue node (for example, the first service data set includes Queue 1 and Queue 2).

[0139] S202: Perform data processing on the data in the first service data set to obtain a second service data set for the target service. The data processing can be, for example, format conversion processing, which mainly refers to converting the service data format in the database node into a data format that the target search node can store. The first service data set can be one or more service data for the target service. The data in the second service data set belongs to the first service data set.

[0140] S203: Send the second service data set to the target search node. In one embodiment, in Figure 3 , a TCP long connection pool is stored in the service transmission node. The TCP long connection pool stores connection resources for accessing the target search node. The connection resources can be one of an available port number and an available communication address. Then the service transmission node can obtain the connection resources for accessing the target search node from the TCP long connection pool, establish a TCP connection according to the connection resources of the target search node, and then use the write thread in the service transmission node to send the second service data set to the target search node through the established TCP connection. Among them, the available communication address can be an IP address, a MAC address, etc. In a specific implementation, the service transmission node can obtain the connection resources for accessing the target search node from the TCP long connection pool, then initiate a TCP request to the target search node. The TCP request carries an Svrkit (an application layer protocol) protocol packet, and then a TCP connection is established based on the TCP request.

[0141] The above TCP connection pool can interact with cl5 (a device implementing load balancing) at system startup to obtain the available communication addresses and available ports for accessing the target search node, thereby caching a TCP (or Socket) long connection pool with a maximum quantity limit, and simultaneously setting a method for verifying the validity of TCP (or Socket). Among them, the method for verifying the validity of TCP (or Socket) can be: when each machine accesses, by recording the access information of each machine, where the access information includes the current status of the machine, as well as the available ports and available communication addresses; then determining the available ports and available communication addresses of each machine based on the access information of each machine; and generating a TCP (or Socket) long connection pool with a maximum quantity limit according to the available ports and available communication addresses of each machine. Then, continuously and regularly verify the validity of the available ports or available communication addresses in the TCP (or Socket) long connection pool. If it is detected that a certain machine fails or the traffic of a certain machine is relatively large, the service transmission node determines that the available ports and available communication addresses corresponding to this machine in the TCP (or Socket) long connection pool are invalid, and promptly removes the ports and communication addresses corresponding to this machine. In one embodiment, it is also possible to periodically poll and detect whether the TCP (or Socket) long connection pool is healthy. If the TCP (or Socket) long connection pool is not healthy, indicating that there are unavailable ports or communication addresses, then the unavailable ports or communication addresses are removed, thereby ensuring the effectiveness of the established TCP connection. Among them, the health here refers to whether the ports or communication addresses in the TCP long connection pool are available. The machines involved above can be the devices corresponding to the service proxy nodes or service supplementary proxy nodes in the embodiments of the present application; they can also be the devices corresponding to the target search nodes.

[0142] In one embodiment, taking the target service as the payment service, there are some differences between the target search node (such as the bill ES) in the embodiments of the present application and the external open-source search node (such as the open-source ES). There is a service proxy node (ES Proxy) adapted for payment. When communicating with the bill ES, it is necessary to open the port of the service transmission node and establish a TCP connection with the service proxy node ES Proxy. In this case, a service proxy node can be configured for the target search node. The specific implementation method for the service transmission node to establish a TCP connection according to the connection resources for accessing the target search node is: establish a TCP connection with the service proxy node according to the connection resources for accessing the target search node.

[0143] In one embodiment, the specific implementation of the service transmission node for sending the second service data set to the target search node by using a write thread and through the established TCP connection is as follows: Based on the established TCP connection, use the write thread to write the second service data set to the service proxy node to trigger the service proxy node to process the second service data set and save the processed second service data set in the target search node. Among them, the above TCP connection refers to a long-term communication connection maintained between the service transmission node and the service proxy node.

[0144] In one embodiment, due to various problems always occurring in the scenario of continuous writing of a high concurrency data volume. For example, downstream disk failures, flow limiting, glitches, etc. may all cause data writing failures. Therefore, in order to prevent data writing failures and reduce data transmission latency problems, in Figure 3 the architecture of the provided service transmission node, an exception failure queue and an exception handling thread are introduced. Among them, the exception failure queue may include service data that fails to be written to the target search node by using the write thread; the exception handling thread is used to process the service data that fails to be written to the target search node by the write thread. In this case, when the sending of the second service data set to the target search node fails, that is, it indicates at this time that the service transmission node fails to write the second service data set to the target search node based on the TCP connection, the service transmission node can add the second service data set to the exception failure queue and perform the first-layer real-time retry. In a specific implementation, when the sending of the second service data set to the target search node fails, the service transmission node can perform service rewriting processing on the second service data set at a first time interval.

[0145] Then, the service transmission node can obtain the number of failures in sending the second service data set to the target search node during the service rewriting process; and determine whether the number of failures exceeds the number threshold. If it is determined that the number of failures exceeds the number threshold, the second service data set is degraded, that is, the second-layer queue retry is performed. The entire second-layer queue is traversed every once in a while to retry the service data in the second-layer queue in sequence. In one embodiment, if the service transmission node determines that the number of failures exceeds the number threshold, it can add the second service data set to the service polling queue, and then traverse the service data in the service polling queue at the second time interval, and use the writing thread to rewrite the currently traversed service data to the target search node; among them, the second time interval can be set according to requirements. For example, the number threshold can be 3 times, 5 times, etc.; the second time interval can be 1 hour, 30 minutes, etc. It should be noted that the obtained number of failures mentioned above refers to the number of consecutive failures in sending the second service data set to the target search node. It should be understood that the service polling queue belongs to the abnormal failure queue, and the second-layer queue is the service polling queue mentioned in the embodiments of the present application. In one embodiment, when the currently traversed service data is successfully rewritten to the target search node by the writing thread, the service transmission node can delete the currently traversed service data from the service polling queue.

[0146] For example, taking the second service data set as the bill data set, the target search node as the bill ES, and the number threshold as 3 as an example; when the bill data set fails to be written to the bill ES through the writing thread based on the TCP connection, the service transmission node performs real-time service rewriting processing on the bill data set; and obtains the number of failures in sending the bill data set to the ES during the service rewriting process; if the number of failures in sending the bill to the ES exceeds the number threshold 3, the bill data set is degraded and added to the service polling queue; then the bill data in the service polling queue is traversed every 30 minutes, and the currently traversed bill data is rewritten to the target search node by the writing thread.

[0147] In one embodiment, the service transmission node can determine whether the service data in the service polling queue includes target service data whose survival duration exceeds the preset duration. If the service data in the service polling queue includes target service data whose survival duration exceeds the preset duration, the target service data is deleted from the service polling queue. If the service data in the service polling queue does not include target service data whose survival duration exceeds the preset duration, the service data in the service polling queue is continuously traversed. Among them, the survival duration of the service data in the service polling queue refers to the duration from the time of being added to the service polling queue to the current time; the preset duration can be set according to requirements. For example, the preset duration can be 12 hours, 24 hours, etc.

[0148] In the embodiment of the present application, the service transmission node obtains the first service data set stored in the database node through the service intermediate node, processes the data in the first service data set to obtain the second service data set of the target service; and sends the second service data set to the target search node, where the data in the second service data set belongs to the first service data set. Instead of writing to the target search node simultaneously when writing to the database node, after writing to the database node, the data of the database node is directly written to the target search node through the service intermediate node and the service transmission node, which can improve the data writing efficiency and better ensure the accuracy of data transmission.

[0149] Based on the above service data processing system, please refer to Figure 4 , Figure 4 which is a schematic flowchart of a service data processing method provided by an embodiment of the present application. This service data processing method can be executed by the supplementary service transmission node 112 in the above service data processing system, and the supplementary service transmission node 112 can be a DTS device. The structure of the supplementary service transmission node 112 can be as above Figure 3 shown. In the embodiment of the present application, this service data processing method includes the following steps S401 - S403:

[0150] S401: Obtain the target service supplementary data from the database supplementary node through the service supplementary intermediate node. The target service supplementary data is the data determined by the data comparison node after comparing the differences between the first service data set and the second service data set and stored in the database supplementary node. The first service data set is obtained from the database node, and the second service data set is obtained from the target search node.

[0151] In one embodiment, multiple service supplementary data stored in the database supplementary node are allowed to be recorded in the service supplementary log node. The specific implementation manner of the above step S401 is: the service supplementary intermediate node parses the multiple service supplementary data stored in the service supplementary log node to obtain the parsed multiple service supplementary data. In specific implementation, the service supplementary intermediate node can parse the multiple service supplementary data stored in the service supplementary log node to obtain the parsed multiple service supplementary data, and then the service supplementary intermediate node sends the parsed multiple service supplementary data to the supplementary service transmission node, and the supplementary service transmission node obtains the target service supplementary data from the parsed multiple service supplementary data. Among them, the multiple service supplementary data are all the difference data between the service data in the database node and the service data in the target search node, and the multiple service supplementary data are all the same as the corresponding service data in the database node.

[0152] In one embodiment, multiple parsed service supplementary data can be stored in the message queue of the service supplementary message queue node. The service supplementary intermediate node sends the multiple parsed service supplementary data to the service supplementary message queue node, and then the supplementary service transmission node uses a read thread to read the target service supplementary data from the message queue of the service supplementary message queue node.

[0153] S402: Perform data processing on the target service supplementary data to obtain service update data. Among them, the data processing can be, for example, format conversion processing.

[0154] S403: Send the service update data to the target search node.

[0155] In one embodiment, obtain connection resources for accessing the target search node from the Transmission Control Protocol (TCP) long connection pool; the connection resources include one of an available port number or an available communication address; then establish a TCP connection according to the connection resources for accessing the target search node; and use a write thread to send the service update data to the target search node through the established TCP connection. Among them, a service supplementary proxy node is configured for the target search node, and the specific implementation of establishing a TCP connection according to the connection resources for accessing the target search node is: establish a TCP connection with the service supplementary proxy node according to the connection resources for accessing the target search node. Among them, the TCP connection means: establish a long-term communication connection between the supplementary service transmission node and the service supplementary proxy node.

[0156] In one embodiment, the specific implementation of the supplementary service transmission node using a write thread and the established TCP connection to send the service update data to the target search node is: based on the established TCP connection, use a write thread to write the service update data to the service supplementary proxy node to trigger the service supplementary proxy node to process the service update data and save the processed service update data in the target search node.

[0157] In one embodiment, when the sending of the service update data to the target search node fails, perform supplementary service rewrite processing on the service update data at a first time interval; and obtain the number of failures of writing the service update data to the target search node during the supplementary service rewrite processing; if the number of failures exceeds the number threshold, add the service update data to the supplementary service polling queue; traverse the service update data in the supplementary service polling queue at a second time interval, and use a write thread to rewrite the currently traversed service update data to the target search node. In one embodiment, when the currently traversed service update data is successfully rewritten to the target search node using a write thread, the currently traversed service update data can be deleted from the supplementary service polling queue.

[0158] In one embodiment, the supplementary service transmission node determines whether the service update data in the supplementary service polling queue includes target service update data whose survival duration exceeds a preset duration. Here, the survival duration of the service update data in the supplementary service polling queue refers to the duration from the time when it is added to the supplementary service polling queue until the current time. If the supplementary service transmission node determines that the service update data in the supplementary service polling queue includes target service update data whose survival duration exceeds the preset duration, the currently traversed target service update data is deleted from the supplementary service polling queue. If the supplementary service transmission node determines that the service update data in the supplementary service polling queue does not include target service update data whose survival duration exceeds the preset duration, the service data in the supplementary service polling queue can be continuously traversed.

[0159] It should be noted that for the embodiments involved in the embodiments of the present application, reference can be made to the above relevant content, and details will not be repeated here.

[0160] In the embodiments of the present application, the service supplementary intermediate node obtains target service supplementary data from the database supplementary node; performs data processing on the target service supplementary data to obtain service update data; and sends the service update data to the target search node. Among them, the target service supplementary data is the data determined by the data comparison node after comparing the differences between the first service data set and the second service data set and stored in the database supplementary node. The first service data set is obtained from the database node, and the second service data set is obtained from the target search node. Through data difference comparison, when it is determined that there are data differences, the target supplementary data corresponding to the data differences can be retrieved from the database supplementary node again and saved to the target search node, so that the data in the database node and the search device can be made consistent and accurate, which is conducive to improving the accuracy of data query.

[0161] Based on the above service data processing system, please refer to Figure 5 , Figure 5 which is a schematic flowchart of a service data processing method provided by the embodiments of the present application. This service data processing method can be executed by the target search node 106 in the above service data processing system. The target search node 106 can be a full-text search engine ES device. In the embodiments of the present application, this service data processing method may include the following steps S501 - S503:

[0162] S501: Receive the second set of service data processed by the first processing link. The second set of service data is data obtained by processing the first set of service data. The first set of service data is obtained by a service intermediate node on the first processing link from a database node on the first processing link. The service intermediate node is a node configured according to the database protocol of the database node and serving as a subordinate device of the database node.

[0163] In one embodiment, multiple pieces of service data stored in the database node are allowed to be recorded in the service log node. The first set of service data is obtained by a service transmission node using a read thread to read from the message queue of the service message queue node. The message queue of the service message queue node stores multiple pieces of parsed service data, and the multiple pieces of parsed service data are obtained by a service intermediate node parsing the multiple pieces of service data stored in the service log node. The service log node, the service transmission node, and the service message queue node belong to the first processing link.

[0164] In one embodiment, the first processing link includes a TCP connection. The TCP connection is established according to the connection resources of a target search node accessing the first processing link. The connection resources include one of an available port number or an available communication address. The connection resources of the target search node accessing the first processing link are obtained from a TCP long connection pool of the Transmission Control Protocol (TCP). The second set of service data is sent by a service transmission node on the first processing link using a write thread through the TCP connection.

[0165] In one embodiment, a service proxy node is configured for the target search node. The TCP connection is established with the service proxy node according to the connection resources used to access the target search node. The second set of service data is sent by the service transmission node using a write thread through the TCP connection to the service proxy node.

[0166] In one embodiment, the second set of service data may include currently traversed service data. The currently traversed service data is obtained by a service transmission node traversing the service data in the service polling queue at a second time interval. The service polling queue includes service data for which the number of failures to send the second set of service data to the target search node during the service rewriting process exceeds a threshold.

[0167] S502: Receive service update data processed by the second processing link. The service update data is obtained by processing target service supplementary data. The target service supplementary data is determined based on a difference comparison between the first set of service data and the second set of service data. The first set of service data is obtained from the database node, and the second set of service data is obtained from the target search node on the first processing link.

[0168] Multiple business supplementary data stored in the database supplementary node are allowed to be recorded in the business supplementary log node. The target business supplementary data is obtained by the supplementary service transmission node using a read thread to read from the message queue of the business supplementary message queue node. The message queue of the business supplementary message queue node stores multiple parsed business supplementary data, and the multiple parsed business supplementary data are obtained by the business supplementary intermediate node parsing the multiple business supplementary data stored in the business supplementary log node. The database supplementary node, the business supplementary log node, the supplementary service transmission node, and the business message queue node belong to the second processing link.

[0169] The second processing link includes a TCP connection. The TCP connection is established according to the connection resources of the target search node accessing the second processing link. The connection resources of the target search node accessing the second processing link are obtained from the TCP long connection pool. The service update data is sent by the supplementary service transmission node on the second processing link using a write thread and through the TCP connection.

[0170] In one embodiment, a business supplementary proxy node is configured for the target search node. The TCP connection is established with the business supplementary proxy node according to the connection resources used to access the target search node. The second service data set is sent by the supplementary service transmission node using a write thread and through the TCP connection to the business supplementary proxy node.

[0171] In one embodiment, the service update data includes the currently traversed service update data. The currently traversed service update data is obtained by the supplementary service transmission node traversing the supplementary service polling queue at a second time interval. The supplementary service polling queue includes service update data for which the number of failures to send service update data to the target search node during the supplementary service rewriting process exceeds a threshold.

[0172] It should be noted that the embodiments involved in the above steps S501 - S502 can be referred to the foregoing related content and will not be elaborated here.

[0173] In the embodiment of the present application, a second set of service data processed by a first processing link is received, where the second set of service data is data obtained by processing a first set of service data, and the first set of service data is obtained by a service intermediate node on the first processing link from a database node on the first processing link; a service update data processed by a second processing link is received, where the service update data is obtained by processing target service supplementary data, and the target service supplementary data is data determined according to a difference comparison between the first set of service data and the second set of service data. The first set of service data is obtained from a database node on the second processing link, and the second set of service data is obtained from a target search node on the second processing link. Through the first processing link, the data in the database node can be directly saved to the target search node without the participation of other links, improving the accuracy of data transmission; through the second processing link, the data in the database node and the data in the target search node are compared for data differences. When data differences are determined, the service update data corresponding to the difference data can be saved to the target search node again, so that the data in the database node and the search device can be better made consistent, which is beneficial to the accuracy of subsequent data queries.

[0174] The service data processing method provided by the embodiment of the present application can be applied to a service refund bill, can realize the vertical search requirement of the service refund bill, and provide a heterogeneous data transmission service from a database node MySQL to a target search node ES. It can solve the problems of long-distance and second-level asynchronous data transmission in intra-domain and extra-domain scenarios, which is beneficial to the accuracy of data transmission. Among them, extra-domain means that users can access the service data processing system using the external network; intra-domain means that the service data processing system can be directly accessed without passing through the external network. Among them, in the service refund bill scenario, the service database node is service MySQL; the service log node is service MySQL binlog; the service intermediate node is service DBSync; the service message queue node is service Hippo; the service transmission node is the first DTS device and the target search node is the bill ES device; the service supplementary log node is service supplementary MySQLbinlog; the service supplementary intermediate node is service supplementary DBSync; the service supplementary message queue node is service supplementary Hippo; the supplementary service transmission node is the second DTS device and the target search node is the bill ES device; the data comparison node is ESCheck, and the database supplementary node is supplementary MySQL; the service data processing method includes the following steps:

[0175] (1) The business system writes billing data to the business MySQL. The business DBSync pretends to be a slave node of the business MySQL master node, pulls the billing data in the business MySQL binlog by parsing the MySQL Binlog protocol, parses the pulled billing data, and pushes the parsed billing data to the business Hippo.

[0176] (2) The first DTS device reads the billing data in the message queue of the business Hippo (corresponding to the above first business data set), processes the read billing data, and then sends the processed billing data to the billing ES device (corresponding to the above second business data set).

[0177] (3) ESCheck pulls the data in the business MySQL and compares it with the data in the billing ES device. If it finds that the data in the billing ES device is missing / inconsistent, it will initiate data supplementation. In a specific implementation, ESCheck pulls the billing data in the business MySQL and compares it with the billing data in the billing ES device (data comparison is data difference comparison) to determine the target business supplementary data.

[0178] (4) There is the same link between the supplementary MySQL and the billing ES device as between the business MySQL and the billing RS device, which is specifically used to process supplementary data. Finally, the billing supplementary data flows to the target billing ES device. In a specific implementation, the business supplementary DBSync pretends to be a slave node of the supplementary MySQL master node, pulls the billing supplementary data in the business supplementary MySQL binlog by parsing the MySQL Binlog protocol, parses the pulled billing supplementary data, and pushes the parsed billing supplementary data to the business supplementary Hippo.

[0179] (5) The second DTS device reads the target billing supplementary data in the message queue of the business supplementary Hippo (corresponding to the above target business supplementary data), processes the target billing supplementary data, and then sends the processed target billing supplementary data to the billing ES device (corresponding to the above business update data).

[0180] The business data processing method implemented by various nodes can solve the complex problem of building a binlog system, improve the efficiency of data writing, and better ensure the accuracy of data. Each node in the above business data processing system generates a simple, efficient, and accurate componentized program, which has been used in Tenpay. The current services include orders (hundreds of library and ten-table orders, date orders, historical orders), recharge orders, etc. Through the business data processing method, it is possible to achieve a data volume migration of nearly 1.5 billion per day. The system throughput TPS is close to 20,000 / s, supporting payment refund bills. Currently, the query requests have been switched from the open-source database HBase to ES. In the future, a new business of KV merchant bills with a daily level of nearly 3 billion is also being prepared for access.

[0181] An embodiment of the present application provides a business data processing solution. This business data processing solution can be as Figure 6a shown. This business data processing solution mainly uses asynchronous dual writing to process business data and introduces MQ (Message Queue). Its general principle is as follows: Implement the asynchronous dual writing process through two links. One link is to write business data into a database node (such as the business relational database in Figure 6a ), and the other link is to first store the business data in MQ (Message Queue), and then through consuming the message queue MQ, realize storing the business data in the target search node (such as the full-text search engine in Figure 6a ).

[0182] In one embodiment, the embodiment of the present application also provides a business data processing solution. This business data processing solution can be as Figure 6b shown. This business data processing solution mainly uses asynchronous dual writing - worker to process business data. Its general principle is as follows: Add a field with a timestamp in the relevant tables of the database node. Any crud operations (a type of create, read, update, and delete operation) such as addition, deletion, and update will cause the time of this field to change, while the crud operations in the original program remain unchanged. And add a timer program to let this timer program scan the specified relevant tables at a certain time interval, and then extract the data that has changed during this time period. And write the changed data into ES one by one.

[0183] In one embodiment, the embodiment of the present application also provides a business data processing solution. This data processing solution can be as Figure 6cAs shown in the figure, the general principle of the business data processing solution is as follows: The business log node in the database node (such as the binary log binlog of the relational database mysql) is used to implement business data processing. The specific steps are as follows: 1) Read the binlog of mysql to obtain the log information of the specified table; 2) Convert the read business data into a message queue MQ; 3) Write a message queue MQ consumer program; 4) Continuously read the message queue MQ, and for each message read, write the message into ES.

[0184] Please refer to Figure 7 , Figure 7 which is a schematic structural diagram of a business data processing device according to an embodiment of the present application. The business data processing device according to an embodiment of the present application can be set on an intelligent device, and the intelligent device can be the above Figure 1 or Figure 2 in the business transmission node. The business data processing device includes the following units:

[0185] The service processing unit 701 is configured to obtain a first business data set stored in the database node through a service intermediate node. The first business data set is data for a target service stored by the database node based on a data processing request. The service intermediate node is a node configured according to the database protocol of the database node and serving as a subordinate device of the database node;

[0186] The service processing unit 701 is further configured to perform data processing on the data in the first business data set to obtain a second business data set for the target service;

[0187] The communication unit 702 is configured to send the second business data set to the target search node, where the data in the second business data set belongs to the first business data set.

[0188] In one embodiment, multiple business data stored in the database node are allowed to be recorded in the business log node. When the service processing unit 701 obtains the first business data set from the database node through the service intermediate node, it can be specifically used for:

[0189] Perform data parsing on multiple business data stored in the business log node through the service intermediate node to obtain the parsed multiple business data;

[0190] Obtain the first business data set from the parsed multiple business data.

[0191] In one embodiment, the parsed multiple business data are stored in the message queue of the business message queue node, and the first business data set is read from the message queue of the business message queue node by a read thread.

[0192] In one embodiment, the service processing unit 701 is configured to obtain connection resources for accessing the target search node from a Transmission Control Protocol (TCP) long connection pool; the connection resources include one or both of an available port number and an available communication address; and establish a TCP connection according to the connection resources for accessing the target search node.

[0193] The communication unit 702 is configured to send the second service data set to the target search node by using a write thread and through the established TCP connection.

[0194] In one embodiment, a service proxy node is configured for the target search node; when the service processing unit 701 establishes a TCP connection according to the connection resources for accessing the target search node, it may specifically be configured to: establish a TCP connection with the service proxy node according to the connection resources for accessing the target search node.

[0195] When sending the second service data set to the target search node by using a write thread and through the established TCP connection, the service processing unit 701 is configured to write the second service data set into the service proxy node based on the established TCP connection by using the write thread, so as to trigger the service proxy node to process the second service data set, and save the processed second service data set in the target search node.

[0196] In one embodiment, the service processing unit 701 is further configured to:

[0197] When the sending of the second service data set to the target search node fails, perform service rewriting processing on the second service data set at a first time interval;

[0198] Obtain the number of failures of sending the second service data set to the target search node during the service rewriting processing;

[0199] If the number of failures exceeds a threshold number, add the second service data set to a service polling queue;

[0200] Traverse the service data in the service polling queue at a second time interval, and rewrite the currently traversed service data to the target search node by using a write thread;

[0201] The second time interval is greater than the first time interval.

[0202] In one embodiment, the service processing unit 701 is further configured to:

[0203] If the service data in the service polling queue includes target service data whose survival duration exceeds a preset duration, the target service data is deleted from the service polling queue.

[0204] It can be understood that the functions of the functional units of the service data processing device in this embodiment can be specifically implemented according to the methods in the above method embodiments, and the specific implementation process can refer to the relevant descriptions in the above method embodiments Figure 1 Or Figure 2 and will not be elaborated here.

[0205] In the embodiments of the present application, the service transmission node obtains the first service data set stored in the database node through the service intermediate node, processes the data in the first service data set to obtain the second service data set of the target service; and sends the second service data set to the target search node, where the data in the second service data set belongs to the first service data set. Instead of writing to the target search node simultaneously when writing to the database node, after writing to the database node, the data of the database node is directly written to the target search node through the service intermediate node and the service transmission node, which can improve the data writing efficiency and better ensure the accuracy of data transmission.

[0206] Please refer to Figure 8 , Figure 8 which is a schematic structural diagram of a service data processing device according to an embodiment of the present application. The service data processing device according to the embodiment of the present application can be set on an intelligent device, and the intelligent device can be the above Figure 1 Or Figure 4 or the supplementary service transmission node among them. The service data processing device includes the following units:

[0207] A service supplement processing unit 801, configured to obtain target service supplement data from a database supplement node through a service supplement intermediate node;

[0208] The service supplement processing unit 801 is further configured to process the target service supplement data to obtain service update data;

[0209] A service supplement communication unit 802, configured to send the service update data to a target search node;

[0210] Wherein, the target service supplement data is data determined by a data comparison node after comparing the differences between the first service data set and the second service data set and stored in the database supplement node. The first service data set is obtained from the database node, and the second service data set is obtained from the target search node.

[0211] Obtain target service supplementary data from a database supplementary node through a service supplementary intermediate node;

[0212] Perform data processing on the target service supplementary data to obtain service update data;

[0213] Send the service update data to a target search node;

[0214] Wherein, the target service supplementary data is data determined by a data comparison node after performing a difference comparison on a first service data set and a second service data set and stored in the database supplementary node. The first service data set is obtained from a database node, and the second service data set is obtained from the target search node.

[0215] In one embodiment, multiple service supplementary data stored in the database supplementary node are allowed to be recorded in a service supplementary log node. When the service supplementary processing unit 801 obtains target service supplementary data from the database supplementary node through the service supplementary intermediate node, it specifically is used for:

[0216] Perform data parsing on multiple service supplementary data stored in the service supplementary log node through the service supplementary intermediate node to obtain the parsed multiple service supplementary data;

[0217] Obtain the target service supplementary data from the parsed multiple service supplementary data.

[0218] In one embodiment, the parsed multiple service supplementary data are stored in a message queue of a service supplementary message queue node, and the target service supplementary data is read from the message queue of the service supplementary message queue node by a read thread.

[0219] In one embodiment, when sending the service update data to the target search node, the service supplementary processing unit 801 is used to obtain connection resources for accessing the target search node from a Transmission Control Protocol (TCP) long connection pool; the connection resources include one or both of an available port number or an available communication address; establish a TCP connection according to the connection resources for accessing the target search node;

[0220] The service supplementary communication unit 801 is used to send the service update data to the target search node by using a write thread and through the established TCP connection.

[0221] In one embodiment, a service supplement proxy node is configured for the target search node; when establishing a TCP connection according to the connection resources for accessing the target search node, the service supplement processing unit 801 is specifically configured to: establish a TCP connection with the service supplement proxy node according to the connection resources for accessing the target search node;

[0222] When the service supplement processing unit 801 sends the service update data to the target search node by using a write thread through the established TCP connection, it may be specifically configured to:

[0223] Based on the established TCP connection, use the write thread to write the service update data to the service supplement proxy node to trigger the service supplement proxy node to process the service update data, and save the processed service update data in the target search node.

[0224] In one embodiment, the service supplement processing unit 801 is further configured to:

[0225] When the sending of the service update data to the target search node fails, perform a supplementary service rewrite process on the service update data at a first time interval;

[0226] Obtain the number of failures in sending the service update data to the target search node during the supplementary service rewrite process;

[0227] If the number of failures exceeds a threshold number, add the service update data to a supplementary service polling queue;

[0228] Traverse the service update data in the supplementary service polling queue at a second time interval, and use the write thread to rewrite the previously traversed service update data to the target search node;

[0229] The second time interval is greater than the first time interval.

[0230] In one embodiment, the service supplement processing unit 801 is further configured to:

[0231] If the service update data in the supplementary service polling queue includes target service update data whose survival duration exceeds a preset duration, delete the target service update data from the supplementary service polling queue.

[0232] It can be understood that the functions of the functional units of the service data processing device in this embodiment can be specifically implemented according to the methods in the above method embodiments, and the specific implementation process can refer to the relevant descriptions of the above method embodiments Figure 1 Or Figure 4 and will not be elaborated here.

[0233] In an embodiment of the present application, a target service supplementary data is obtained from a database supplementary node through a service supplementary intermediate node; the target service supplementary data is processed to obtain service update data; and the service update data is sent to a target search node. The target service supplementary data is data determined by a data comparison node after comparing the differences between a first service data set and a second service data set and stored in the database supplementary node. The first service data set is obtained from a database node, and the second service data set is obtained from the target search node. Through data difference comparison, when it is determined that there are data differences, the target supplementary data corresponding to the data differences is re-obtained from the database supplementary node and saved to the target search node, so that the data in the database node and the search device can be made consistent and accurate, which is conducive to improving the accuracy of data query.

[0234] Please refer to Figure 9 , Figure 9 which is a schematic structural diagram of a service data processing device according to an embodiment of the present application. The service data processing device according to an embodiment of the present application can be set on an intelligent device, and the intelligent device can be the above Figure 1 or Figure 5 target search device in. The service data processing device includes the following units:

[0235] A communication unit 901, configured to send and receive data;

[0236] A processing unit 902, configured to receive, through the communication unit, a second service data set processed by a first processing link. The second service data set is data obtained by processing a first service data set. The first service data set is obtained from a database node on the first processing link through a service intermediate node on the first processing link. The service intermediate node is a node configured according to the database protocol of the database node and serving as a subordinate device of the database node;

[0237] The processing unit 902 is further configured to receive, through the communication unit, service update data processed by a second processing link. The service update data is obtained by processing target service supplementary data. The target service supplementary data is data determined by comparing the differences between the first service data set and the second service data set. The first service data set is obtained from the database node, and the second service data set is obtained from the target search node on the first processing link.

[0238] In one embodiment, a plurality of service data stored in the database node are allowed to be recorded in the service log node. The first service data set is obtained by the service transmission node using a read thread to read from the message queue of the service message queue node. A plurality of parsed service data are stored in the message queue of the service message queue node, and the plurality of parsed service data are obtained by the service intermediate node parsing the plurality of service data stored in the service log node. The service log node, the service transmission node, and the service message queue node belong to the first processing link.

[0239] A plurality of service supplementary data stored in the database supplementary node are allowed to be recorded in the service supplementary log node. The target service supplementary data is obtained by the supplementary service transmission node using a read thread to read from the message queue of the service supplementary message queue node. A plurality of parsed service supplementary data are stored in the message queue of the service supplementary message queue node, and the plurality of parsed service supplementary data are obtained by the service supplementary intermediate node parsing the plurality of service supplementary data stored in the service supplementary log node. The database supplementary node, the service supplementary log node, the supplementary service transmission node, the service message queue node, and the service supplementary intermediate node belong to the second processing link.

[0240] In one embodiment, the first processing link includes a TCP connection. The TCP connection is established according to the connection resources of the target search node accessing the first processing link. The connection resources include one of an available port number or an available communication address. The connection resources of the target search node accessing the first processing link are obtained from the TCP long connection pool. The second service data set is sent by the service transmission node on the first processing link using a write thread through the TCP connection.

[0241] In one embodiment, a service proxy node is configured for the target search node. The TCP connection is established with the service proxy node according to the connection resources for accessing the target search node. The second service data set is sent by the service transmission node using a write thread through the TCP connection to the service proxy node.

[0242] In one embodiment, the second processing link includes a TCP connection. The TCP connection is established according to the connection resources of the target search node accessing it. The connection resources of the target search node accessing it are obtained from the TCP long connection pool. The service update data is sent by the supplementary service transmission node on the second processing link using a write thread through the TCP connection.

[0243] In one embodiment, a service supplement agent node is configured for the target search node; the TCP connection is established with the service supplement agent node according to the connection resources used to access the target search node; the service update data is sent by the supplementary service transmission node to the service supplement agent node using a write thread and through the TCP connection.

[0244] It can be understood that the functions of the functional units of the service data processing device in this embodiment can be specifically implemented according to the methods in the above method embodiments, and the specific implementation process can refer to the relevant descriptions of the above method embodiments Figure 1 Or Figure 5 and will not be elaborated here.

[0245] In the embodiment of the present application, a second service data set processed by a first processing link is received, where the second service data set is data obtained by processing a first service data set, and the first service data set is obtained by a service intermediate node on the first processing link from a database node on the first processing link; service update data processed by a second processing link is received, and the service update data is obtained by processing target service supplement data, and the target service supplement data is data determined according to a difference comparison between the first service data set and the second service data set, the first service data set is obtained from a database node on the second processing link, and the second service data set is obtained from a target search node on the second processing link; through the first processing link, the data in the database node can be directly saved to the target search node without the participation of other links, improving the accuracy of data transmission; through the second processing link, the data in the database node and the data in the target search node are compared for data differences, and when it is determined that there are differences in the data, the service update data corresponding to the different data can be re-saved to the target search node, so that the data in the database node and the search device can be better made consistent, which is beneficial to the accuracy of subsequent data queries.

[0246] Further, please refer to Figure 10 , Figure 10 which is a schematic structural diagram of an intelligent device provided by an embodiment of the present application. The intelligent device may be the service transmission node in the above Figure 1 or Figure 2 and may include: a processor 1001, an input device 1002, an output device 1003, and a memory 1004. The above-mentioned processor 1001, input device 1002, output device 1003, and memory 1004 are connected through a bus 1005. The memory 1004 is used to store a computer program, and the computer program includes program instructions, and the processor 1001 is used to execute the program instructions stored in the memory 1004.

[0247] In an embodiment of the present application, the processor 1001 performs the following operations by running the executable program code in the memory 1004:

[0248] Obtain a first service data set stored in the database node through a service intermediate node, where the first service data set is data for a target service stored by the database node based on a data processing request, and the service intermediate node is a node configured according to the database protocol of the database node and serving as a subordinate device of the database node;

[0249] Process the data in the first service data set to obtain a second service data set for the target service;

[0250] Send the second service data set to the target search node, where the data in the second service data set belongs to the first service data set.

[0251] In one embodiment, multiple service data stored in the database node are allowed to be recorded in the service log node. When the processor 1001 obtains the first service data set from the database node through the service intermediate node, it may specifically be used for:

[0252] Perform data parsing on multiple service data stored in the service log node through the service intermediate node to obtain the parsed multiple service data;

[0253] Obtain the first service data set from the parsed multiple service data.

[0254] In one embodiment, the parsed multiple service data are stored in the message queue of the service message queue node, and the first service data set is read from the message queue of the service message queue node by a read thread.

[0255] In one embodiment, when the processor 1001 sends the second service data set to the target search node, it may specifically be used for:

[0256] Obtain connection resources for accessing the target search node from a Transmission Control Protocol (TCP) long connection pool; the connection resources include one or both of an available port number or an available communication address;

[0257] Establish a TCP connection according to the connection resources for accessing the target search node;

[0258] Use a write thread and send the second service data set to the target search node through the established TCP connection.

[0259] In one embodiment, a service proxy node is configured for the target search node; the processor 1001, when establishing a TCP connection according to the connection resources for accessing the target search node, is specifically configured to:

[0260] Establish a TCP connection with the service proxy node according to the connection resources for accessing the target search node;

[0261] When the processor 1001 sends the second service data set to the target search node by using a write thread and through the established TCP connection, it is specifically configured to:

[0262] Based on the established TCP connection, use the write thread to write the second service data set into the service proxy node to trigger the service proxy node to process the second service data set, and save the processed second service data set in the target search node.

[0263] In one embodiment, the processor 1001 is further configured to:

[0264] When the sending of the second service data set to the target search node fails, perform service rewriting processing on the second service data set at a first time interval;

[0265] Obtain the number of failures of sending the second service data set to the target search node during the service rewriting processing;

[0266] If the number of failures exceeds a threshold number of times, add the second service data set to a service polling queue;

[0267] Traverse the service data in the service polling queue at a second time interval, and use a write thread to rewrite the currently traversed service data to the target search node;

[0268] The second time interval is greater than the first time interval.

[0269] In one embodiment, the processor 1001 is further configured to:

[0270] If the service data in the service polling queue includes target service data whose survival duration exceeds a preset duration, delete the target service data from the service polling queue.

[0271] It should be understood that in the embodiments of the present application, the so-called processor 1001 may be a central processing unit (CPU), and this processor 1001 may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0272] The memory 1004 may include a read-only memory and a random access memory, and provide instructions and data to the processor 1001. A part of the memory 1004 may also include a non-volatile random access memory.

[0273] The input device 1002 may include a keyboard, etc., and input service data to the processor 1001; the output device 1003 may include a display, etc.

[0274] In a specific implementation, the processor 1001, input device 1002, output device 1003, and memory 1004 described in the embodiments of the present application may execute the implementation manners described in the above embodiments Figure 1 and Figure 2 described in the above, and may also execute the implementation manners described in the above device, which will not be elaborated here.

[0275] In the embodiments of the present application, the service transmission node obtains the first service data set stored in the database node through the service intermediate node, processes the data in the first service data set to obtain the second service data set of the target service; and sends the second service data set to the target search node, where the data in the second service data set belongs to the first service data set. Instead of writing to the target search node simultaneously when writing to the database node, after writing to the database node, the data of the database node can be directly written to the target search node through the service intermediate node and the service transmission node, which can improve the efficiency of data writing and better ensure the accuracy of data transmission.

[0276] Furthermore, the embodiments of the present application also provide a schematic structural diagram of an intelligent device, and the schematic structural diagram of the intelligent device can be referred to Figure 10 . The intelligent device may be the above Figure 1 or Figure 4The supplementary service transfer node therein, the intelligent device may include: a processor 1001, an input device 1002, an output device 1003, and a memory 1004. The above-mentioned processor 1001, input device 1002, output device 1003, and memory 1004 are connected through a bus 1005. The memory 1004 is used to store a computer program, and the computer program includes program instructions. The processor 1001 is used to execute the program instructions stored in the memory 1004.

[0277] In an embodiment of the present application, the processor 1001 performs the following operations by running the executable program code in the memory 1004:

[0278] Obtain target service supplementary data from the database supplementary node through the service supplementary intermediate node;

[0279] Perform data processing on the target service supplementary data to obtain service update data;

[0280] Send the service update data to the target search node;

[0281] Among them, the target service supplementary data is the data determined by the data comparison node after comparing the differences between the first service data set and the second service data set stored in the database supplementary node. The first service data set is obtained from the database node, and the second service data set is obtained from the target search node.

[0282] In one embodiment, multiple service supplementary data stored in the database supplementary node are allowed to be recorded in the service supplementary log node. When the processor 1001 obtains the target service supplementary data from the database supplementary node through the service supplementary intermediate node, it is specifically used for:

[0283] Perform data parsing on multiple service supplementary data stored in the service supplementary log node through the service supplementary intermediate node to obtain the parsed multiple service supplementary data;

[0284] Obtain the target service supplementary data from the parsed multiple service supplementary data.

[0285] In one embodiment, the parsed multiple service supplementary data are stored in the message queue of the service supplementary message queue node, and the target service supplementary data is read from the message queue of the service supplementary message queue node by a read thread.

[0286] In one embodiment, when the processor 1001 sends the service update data to the target search node, it is specifically used for:

[0287] Obtain connection resources for accessing the target search node from the Transmission Control Protocol (TCP) long connection pool; the connection resources include one or both of an available port number or an available communication address;

[0288] Establish a TCP connection according to the connection resources for accessing the target search node;

[0289] Use a write thread and send the service update data to the target search node through the established TCP connection.

[0290] In one embodiment, a service supplementary proxy node is configured for the target search node; when the processor 1001 establishes a TCP connection according to the connection resources for accessing the target search node, it is specifically configured to:

[0291] Establish a TCP connection with the service supplementary proxy node according to the connection resources for accessing the target search node;

[0292] When the processor 1001 uses a write thread and sends the service update data to the target search node through the established TCP connection, it may be specifically configured to:

[0293] Based on the established TCP connection, use a write thread to write the service update data to the service supplementary proxy node to trigger the service supplementary proxy node to process the service update data and save the processed service update data in the target search node.

[0294] In one embodiment, the processor 1001 is further configured to:

[0295] When the sending of the service update data to the target search node fails, perform supplementary service rewriting processing on the service update data at a first time interval;

[0296] Obtain the number of failures of sending the service update data to the target search node during the supplementary service rewriting processing;

[0297] If the number of failures exceeds a threshold, add the service update data to a supplementary service polling queue;

[0298] Traverse the service update data in the supplementary service polling queue at a second time interval and use a write thread to re-write the previously traversed service update data to the target search node;

[0299] The second time interval is greater than the first time interval.

[0300] In one embodiment, the processor 1001 is further configured to:

[0301] If the service update data in the supplementary service polling queue includes target service update data whose survival duration exceeds a preset duration, the target service update data is deleted from the supplementary service polling queue.

[0302] It should be understood that in the embodiments of the present application, the so-called processor 1001 may be a central processing unit (CPU), and this processor 1001 may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0303] The memory 1004 may include a read-only memory and a random access memory, and provide instructions and data to the processor 1001. A part of the memory 1004 may also include a non-volatile random access memory.

[0304] The input device 1002 may include a keyboard, etc., and input service data to the processor 1001; the output device 1003 may include a display, etc.

[0305] In specific implementation, the processor 1001, input device 1002, output device 1003, and memory 1004 described in the embodiments of the present application may execute the implementation manners described in the above embodiments Figure 1 and Figure 4 the implementation manners described in the above device, and details are not described herein again.

[0306] In the embodiments of the present application, the target service supplementary data is obtained from the database supplementary node by the service supplementary intermediate node; the target service supplementary data is processed to obtain service update data; the service update data is sent to the target search node; wherein, the target service supplementary data is the data determined by the data comparison node after comparing the differences between the first service data set and the second service data set and stored in the database supplementary node, the first service data set is obtained from the database node, and the second service data set is obtained from the target search node. Through data difference comparison, when it is determined that there are differences in the data, the target supplementary data corresponding to the data difference is re-obtained from the database supplementary node and saved to the target search node, so that the data in the database node and the search device can be better made consistent and accurate, which is beneficial to improving the accuracy of data query.

[0307] Further, the embodiments of the present application also provide a schematic structural diagram of an intelligent device. The schematic structural diagram of the intelligent device can be seen Figure 10 . The intelligent device may be the target search node in the above Figure 1 or Figure 5 . The intelligent device may include: a processor 1001, an input device 1002, an output device 1003, and a memory 1004. The above processor 1001, input device 1002, output device 1003, and memory 1004 are connected through a bus 1005. The memory 1004 is used to store a computer program, and the computer program includes program instructions. The processor 1001 is used to execute the program instructions stored in the memory 1004.

[0308] In the embodiments of the present application, the processor 1001 performs the following operations by running the executable program code in the memory 1004:

[0309] Receiving a second service data set processed by a first processing link, wherein the second service data set is data obtained by processing a first service data set, the first service data set is obtained from a database node on the first processing link by a service intermediate node on the first processing link, and the service intermediate node is a node configured according to the database protocol of the database node and serving as a subordinate device of the database node;

[0310] Receiving service update data processed by a second processing link, the service update data is obtained by processing target service supplementary data, the target service supplementary data is data determined according to a difference comparison between the first service data set and the second service data set, the first service data set is obtained from the database node, and the second service data set is obtained from the target search node on the first processing link.

[0311] In one embodiment, multiple pieces of service data stored in the database node are allowed to be recorded in the service log node, and the first set of service data is obtained by the service transmission node using a read thread to read from the message queue of the service message queue node; multiple pieces of parsed service data are stored in the message queue of the service message queue node, and the multiple pieces of parsed service data are obtained by the service intermediate node parsing the multiple pieces of service data stored in the service log node; the service log node, the service transmission node, and the service message queue node belong to the first processing link.

[0312] Multiple pieces of service supplementary data stored in the database supplementary node are allowed to be recorded in the service supplementary log node, and the target service supplementary data is obtained by the supplementary service transmission node using a read thread to read from the message queue of the service supplementary message queue node; multiple pieces of parsed service supplementary data are stored in the message queue of the service supplementary message queue node, and the multiple pieces of parsed service supplementary data are obtained by the service supplementary intermediate node parsing the multiple pieces of service supplementary data stored in the service supplementary log node; the database supplementary node, the service supplementary log node, the supplementary service transmission node, the service message queue node, and the service supplementary intermediate node belong to the second processing link.

[0313] In one embodiment, the first processing link includes a TCP connection; the TCP connection is established according to the connection resources of the target search node accessing the first processing link; the connection resources include one of an available port number or an available communication address; the connection resources of the target search node accessing the first processing link are obtained from the TCP long connection pool; the second set of service data is sent by the service transmission node on the first processing link using a write thread and through the TCP connection.

[0314] In one embodiment, a service proxy node is configured for the target search node; the TCP connection is established with the service proxy node according to the connection resources for accessing the target search node; the second set of service data is sent by the service transmission node using a write thread and through the TCP connection to the service proxy node.

[0315] In one embodiment, the second processing link includes a TCP connection; the TCP connection is established according to the connection resources of the target search node accessing the second processing link; the connection resources of the target search node accessing the second processing link are obtained from the TCP long connection pool; the service update data is sent by the supplementary service transmission node on the second processing link using a write thread and through the TCP connection.

[0316] In one embodiment, a service supplement agent node is configured for the target search node; the TCP connection is established with the service supplement agent node according to the connection resources for accessing the target search node; and the service update data is sent by the supplementary service transmission node to the service supplement agent node by using a write thread and through the TCP connection.

[0317] It should be understood that in the embodiments of the present application, the so-called processor 1001 may be a central processing unit (CPU), and this processor 1001 may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0318] The memory 1004 may include a read-only memory and a random access memory, and provide instructions and data to the processor 1001. A part of the memory 1004 may also include a non-volatile random access memory.

[0319] The input device 1002 may include a keyboard, etc., and input service data to the processor 1001; the output device 1003 may include a display, etc.

[0320] In a specific implementation, the processor 1001, input device 1002, output device 1003, and memory 1004 described in the embodiments of the present application may execute the implementation manners described in the above embodiments Figure 1 and Figure 5 the implementation manners described in the above device, which will not be elaborated herein.

[0321] In an embodiment of the present application, a second set of service data processed by a first processing link is received, where the second set of service data is data obtained by processing a first set of service data, and the first set of service data is obtained by a service intermediate node on the first processing link from a database node on the first processing link; a service update data processed by a second processing link is received, and the service update data is obtained by processing target service supplementary data, and the target service supplementary data is data determined according to a difference comparison between the first set of service data and the second set of service data. The first set of service data is obtained from a database node on the second processing link, and the second set of service data is obtained from a target search node on the second processing link. Through the first processing link, the data in the database node can be directly saved to the target search node without the participation of other links, improving the accuracy of data transmission; through the second processing link, the data in the database node and the data in the target search node are compared for data differences. When it is determined that there are differences in the data, the service update data corresponding to the different data is saved to the target search node again, so that the data in the database node and the search device can be better made consistent, which is beneficial to the accuracy of subsequent data queries.

[0322] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, and when a processor runs the computer program, the intelligent device is enabled to execute the method provided in the foregoing embodiment.

[0323] An embodiment of the present application further provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, enabling the intelligent device to execute the method provided in the foregoing embodiment.

[0324] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the foregoing embodiments can be completed by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the foregoing method embodiments. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0325] The above-disclosed is only a preferred embodiment of the present invention. Of course, it cannot be used to limit the scope of the rights of the present invention. Those of ordinary skill in the art can understand all or part of the processes of implementing the above embodiments, and the equivalent changes made according to the claims of the present invention still fall within the scope covered by the invention.

Claims

1. A method for processing service data, characterized in that, it includes: obtaining a first set of service data stored in a database node through a service intermediate node, where the first set of service data is data for a target service stored by the database node based on a data processing request, and the service intermediate node is a node configured according to the database protocol of the database node and serving as a subordinate device of the database node; processing the data in the first set of service data to obtain a second set of service data for the target service; sending the second set of service data to a target search node, where the data in the second set of service data belongs to the first set of service data.

2. The method according to claim 1, characterized in that, multiple service data stored in the database node are allowed to be recorded in a service log node, and obtaining the first set of service data from the database node through the service intermediate node includes: performing data parsing on multiple service data stored in the service log node through the service intermediate node to obtain the parsed multiple service data; obtaining the first set of service data from the parsed multiple service data.

3. The method according to claim 1, characterized in that, sending the second set of service data to the target search node includes: obtaining connection resources for accessing the target search node from a Transmission Control Protocol (TCP) long connection pool; the connection resources include one or both of an available port number or an available communication address; establishing a TCP connection according to the connection resources for accessing the target search node; using a write thread and sending the second set of service data to the target search node through the established TCP connection.

4. The method according to any one of claims 1 - 3, characterized in that, the method further includes: when the sending of the second set of service data to the target search node fails, performing service rewriting processing on the second set of service data at a first time interval; obtaining the number of failures in sending the second set of service data to the target search node during the service rewriting processing; if the number of failures exceeds a threshold number, adding the second set of service data to a service polling queue; traversing the service data in the service polling queue at a second time interval and using a write thread to rewrite the currently traversed service data to the target search node; the second time interval is greater than the first time interval.

5. A method for processing service data, characterized in that, it includes: obtaining target service supplementary data from a database supplementary node through a service supplementary intermediate node; processing the target service supplementary data to obtain service update data; obtaining connection resources for accessing the target search node from a Transmission Control Protocol (TCP) long connection pool; the connection resources include: one or both of an available port number or an available communication address; establishing a TCP connection according to the connection resources for accessing the target search node; Using a write thread and through the established TCP connection, send the service update data to the target search node; Wherein, the target service supplementary data is the data determined by the data comparison node after comparing the differences between the first service data set and the second service data set and stored in the database supplementary node. The first service data set is obtained from the database node, and the second service data set is obtained from the target search node.

6. The method according to claim 5, characterized in that Multiple service supplementary data stored in the database supplementary node are allowed to be recorded in the service supplementary log node. The process of obtaining the target service supplementary data from the database supplementary node through the service supplementary intermediate node includes: Performing data parsing on multiple service supplementary data stored in the service supplementary log node through the service supplementary intermediate node to obtain the parsed multiple service supplementary data; Obtaining the target service supplementary data from the parsed multiple service supplementary data.

7. The method according to claim 5 or 6, the method further includes: When the sending of the service update data to the target search node fails, perform supplementary service rewriting processing on the service update data at a first time interval; Obtain the number of failures in sending the service update data to the target search node during the supplementary service rewriting process; If the number of failures exceeds the number threshold, add the service update data to the supplementary service polling queue; Traverse the service update data in the supplementary service polling queue at a second time interval, and use the write thread to re-write the previously traversed service update data to the target search node; The second time interval is greater than the first time interval.

8. A service data processing method, characterized in that includes: Receiving a second service data set processed by a first processing link, wherein the second service data set is data obtained by processing a first service data set. The first service data set is obtained from a database node on the first processing link through a service intermediate node on the first processing link. The service intermediate node is configured according to the database protocol of the database node and is a node subordinate to the database node; Receiving service update data processed by a second processing link, the service update data is obtained by processing target service supplementary data, and the target service supplementary data is determined according to the difference comparison between the first service data set and the second service data set. The first service data set is obtained from the database node, and the second service data set is obtained from a target search node on the first processing link.

9. The method according to claim 8, characterized in that Multiple service data stored in the database node are allowed to be recorded in the service log node. The first service data set is obtained by the service transmission node using a read thread to read from the message queue of the service message queue node. Multiple parsed service data are stored in the message queue of the service message queue node, and the multiple parsed service data are obtained by the service intermediate node parsing the multiple service data stored in the service log node. The service log node, the service transmission node, and the service message queue node belong to the first processing link. Multiple service supplementary data stored in the database supplementary node are allowed to be recorded in the service supplementary log node. The target service supplementary data is obtained by the supplementary service transmission node using a read thread to read from the message queue of the service supplementary message queue node. Multiple parsed service supplementary data are stored in the message queue of the service supplementary message queue node, and the multiple parsed service supplementary data are obtained by the service supplementary intermediate node parsing the multiple service supplementary data stored in the service supplementary log node. The database supplementary node, the service supplementary log node, the supplementary service transmission node, the service message queue node, and the service supplementary intermediate node belong to the second processing link.

10. The method according to claim 8, wherein, the first processing link includes a TCP connection. The TCP connection is established according to the connection resources of the target search node accessing the first processing link. The connection resources include one of an available port number or an available communication address. The connection resources of the target search node accessing the first processing link are obtained from the TCP long connection pool. The second service data set is sent by the service transmission node on the first processing link using a write thread through the TCP connection.

11. The method according to claim 10, wherein, a service proxy node is configured for the target search node. The TCP connection is established with the service proxy node according to the connection resources for accessing the target search node. The second service data set is sent by the service transmission node using a write thread through the TCP connection to the service proxy node.

12. The method according to any one of claims 8-11, wherein, the second processing link includes a TCP connection. The TCP connection is established according to the connection resources of the target search node accessing it. The connection resources of the target search node accessing it are obtained from the TCP long connection pool. The service update data is sent by the supplementary service transmission node on the second processing link using a write thread through the TCP connection.

13. An intelligent device, wherein, comprising: a storage device and a processor; the storage device is used for storing a computer program; The processor is configured to execute the computer program to implement the method according to any one of claims 1-4, or the method according to any one of claims 5-7, or the method according to any one of claims 8-12.

14. A computer-readable storage medium, wherein, the computer-readable storage medium stores a computer program, and when the computer program is executed, it implements the method according to any one of claims 1-4, or the method according to any one of claims 5-7, or the method according to any one of claims 8-12.

15. A computer program product comprising a computer program / instructions, wherein, when the computer program / instructions are executed by a processor, it implements the method according to any one of claims 1-4, or the method according to any one of claims 5-7, or the method according to any one of claims 8-12.

Images