A data processing method, system and computing device

By processing and storing the changed data of the business module in real time in the data processing server, the high communication cost and data inconsistency of the data integration solution in the existing technology are solved, real-time update and consistency of data are achieved, and the stability of the system is improved. and reliability.

CN113377791BActive Publication Date: 2024-11-08BEIJING QIERBULAITE TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110645575.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-10
Publication Date
2024-11-08
Estimated Expiration
2041-06-10

AI Technical Summary

Technical Problem

In the prior art, data integration solutions have problems such as high communication costs, delay problems and data inconsistency, especially in the process of business data integration in multiple vertical data centers.

Method used

By extracting the data of the business module in the data processing server, obtaining the changed data in real time and encapsulating it into a message body, and sending it to the message server for processing and storage, ensuring real-time updates and consistency of the data storage device, and providing it through message queues and search engines Personalized service.

Benefits of technology

Non-invasive data integration is realized, communication costs are reduced, delay problems are avoided, and data consistency and real-timeness are ensured through a timed compensation mechanism, improving the stability and reliability of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113377791B_ABST
    Figure CN113377791B_ABST
Patent Text Reader

Abstract

The present invention discloses a data processing method, including: extracting data in one or more business modules to a data acquisition device based on corresponding data tables; creating a message subject corresponding to each data table in a message server; obtaining the change data in each data table in real time, encapsulating the change data in each data table into a message body, and sending the message body to the message server based on the corresponding message subject; obtaining one or more message bodies corresponding to each message subject from the message server, and processing the message body into corresponding change data and storing it in a data storage device, so as to update the data in the data storage device; sending the change data in the data storage device to a business resource server corresponding to the corresponding business module. The present invention also discloses a corresponding data processing system and a computing device. According to the technical solution of the present invention, data integration of each business module can be realized, and the real-time and consistency of the data can be guaranteed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer and Internet technology, and in particular to a data processing method, a data processing system and a computing device. Background Art

[0002] At present, with the expansion and diversification of enterprise business lines, while actively promoting business data, various vertical businesses are also developing rapidly, thus forming multiple vertical data centers. Therefore, it is necessary to connect various data centers, integrate data and provide unified services to form a unified full business line data system.

[0003] In the prior art, there are two ways to implement data integration. One is to use an intrusive interface call method to provide a central database data integration interface, which is called by the business side to write. This solution increases communication and implementation costs because the integration of data from various business lines requires relevant departments to actively call the interface and push it. In addition, this solution will inevitably have delay problems.

[0004] The other is a non-invasive binlog parsing method that parses the real-time binlog of the business database and collects it uniformly. With this solution, when the full data is integrated for the first time, it is necessary to perform task processing or trigger the full data binlog regularly, which takes too long and there is a problem of data inconsistency between the business database and the central database.

[0005] Therefore, a data processing method and system are needed to solve the problems existing in the above technical solutions. Summary of the invention

[0006] To this end, the present invention provides a data processing method, system and computing device to solve or at least alleviate the above problems.

[0007] According to one aspect of the present invention, there is provided a data processing method, which is executed in a data processing server and includes the steps of: extracting data in one or more business modules to a data acquisition device based on corresponding data tables; creating a message subject corresponding to each data table in a message server; acquiring the changed data in each data table in real time, encapsulating the changed data in each data table into a message body, and sending the message body to the message server based on the corresponding message subject; acquiring one or more message bodies corresponding to each message subject from the message server, and processing the message bodies into corresponding changed data and storing them in a data storage device so as to update the data in the data storage device; and sending the changed data in the data storage device to a business resource server corresponding to the corresponding business module.

[0008] Optionally, the data processing method according to the present invention further includes the step of sending the changed data in the data storage device to a resource server, so as to search for the changed data in the resource server based on a search engine.

[0009] Optionally, the data processing method according to the present invention further includes the step of sending the changed data in the resource server to the search engine in real time, so that the search engine generates a corresponding index based on the changed data.

[0010] Optionally, the data processing method according to the present invention also includes the steps of: comparing the data in the data storage device and the data acquisition device at predetermined intervals to determine whether there is missing data in the data storage device; if there is missing data, generating a compensation task based on the missing data, and storing the missing data back to the data storage device by executing the compensation task.

[0011] Optionally, in the data processing method according to the present invention, the step of processing the message body into corresponding change data and storing it in the data storage device includes: parsing the message body to obtain the change data and the corresponding business module identifier in the data table corresponding to the message body; and storing the change data in the data storage device in association with the business module identifier.

[0012] Optionally, in the data processing method according to the present invention, the step of sending the changed data in the data storage device to the business resource server corresponding to the corresponding business module includes: sending the changed data in the data storage device to the corresponding message queue based on the associated business module identifier, so as to send the changed data to the business resource server corresponding to the corresponding business module based on the message queue.

[0013] Optionally, in the data processing method according to the present invention, the step of processing the message body into corresponding changed data and storing it in the data storage device includes: creating a thread pool including multiple threads; obtaining the message body from the message server based on multiple threads, and processing the message body into corresponding changed data and storing it in the data storage device.

[0014] Optionally, in the data processing method according to the present invention, before extracting data in one or more business modules to a data acquisition device based on corresponding data tables, the method includes the steps of: obtaining data models corresponding to the one or more business modules respectively; and creating data tables corresponding to each data model in the data acquisition device.

[0015] According to one aspect of the present invention, there is provided a data processing system, comprising: a data acquisition device, suitable for extracting data of one or more business modules based on corresponding data tables, and suitable for obtaining change data in each data table in real time, and encapsulating the change data in each data table as a message body; a message server, connected to the data acquisition device, suitable for creating a message subject corresponding to each data table, and suitable for receiving a message body sent by the data acquisition device based on the corresponding message subject; a data processing server, connected to the message server, suitable for obtaining one or more message bodies corresponding to each message subject from the message server, and processing the message body into corresponding change data; a data storage device, connected to the data processing server, suitable for obtaining and storing the change data from the data processing server, so as to update the data in the data storage device; and a business resource server corresponding to each business module, connected to the data storage device, suitable for obtaining the corresponding change data from the data storage device.

[0016] Optionally, the data processing system according to the present invention further includes: a resource server, connected to the data storage device, suitable for obtaining change data from the data storage device; a search engine, connected to the resource server, suitable for receiving the change data sent by the resource server in real time, and suitable for generating corresponding indexes based on the change data.

[0017] Optionally, the data processing system according to the present invention further includes: a comparison module, which is respectively connected to the data storage device and the data acquisition device, and is suitable for comparing the data in the data storage device and the data acquisition device at predetermined intervals to determine whether there is missing data in the data storage device; if there is missing data, a compensation task is generated based on the missing data, and the missing data is stored back to the data storage device by executing the compensation task.

[0018] According to one aspect of the present invention, there is provided a computing device, comprising: at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be suitable for execution by the at least one processor, and the program instructions include instructions for executing the data processing method as described above.

[0019] According to one aspect of the present invention, a readable storage medium storing program instructions is provided. When the program instructions are read and executed by a computing device, the computing device executes the data processing method as described above.

[0020] According to the technical solution of the present invention, a data processing method and system are provided, which collects data of each business module through a data acquisition device, and obtains the change data of each business module in real time, and encapsulates the change data of each business module into a message body, and sends it to a message server based on the message subject corresponding to the business module, and the message body is obtained in real time by the data processing device, and the change data obtained after parsing is stored in the data storage device in association with the business module identifier. In this way, the present invention realizes the integration of the data of each business module in a non-invasive manner, will not affect the functions of each business module, and can write the change data in the data acquisition device to the data storage device in real time, ensuring that the data in the data storage device is updated in real time and consistent with the data of the data acquisition device. In addition, the data in the data storage device can be distributed to the corresponding business resource server, so that each business resource server can provide personalized search and recommendation services.

[0021] In addition, the present invention can ensure the consistency and real-time performance of data in the data storage device and the data acquisition device by regularly compensating for data missed by the data storage device, making the system data more stable and reliable.

[0022] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] To achieve the above and related purposes, certain illustrative aspects are described herein in conjunction with the following description and accompanying drawings, which indicate various ways in which the principles disclosed herein can be practiced, and all aspects and their equivalents are intended to fall within the scope of the claimed subject matter. The above and other purposes, features and advantages of the present disclosure will become more apparent by reading the following detailed description in conjunction with the accompanying drawings. Throughout the present disclosure, the same reference numerals generally refer to the same parts or elements.

[0024] Figure 1 A schematic diagram of a data processing system 100 according to an embodiment of the present invention is shown;

[0025] Figure 2 A schematic diagram showing a computing device 200 according to one embodiment of the present invention; and

[0026] Figure 3 FIG. 3 is a flow chart of a data processing method 300 according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0028] Figure 1 FIG. 1 is a schematic diagram of a data processing system 100 according to an embodiment of the present invention.

[0029] like Figure 1 As shown, the data processing system 100 includes one or more business modules 110, a data acquisition device 120, a message server 130, a data processing server 140 and a data storage device 150. The data acquisition device 120 is respectively connected to one or more business modules 110 in communication, and the data acquisition device 120 is connected to the message server 130 in communication, for example, through a wired or wireless network connection.

[0030] The data acquisition device 120 may create a data table corresponding to the data model of each business module, and based on the corresponding data table, the data in one or more business modules 110 may be extracted into the data acquisition device 120. Specifically, the data models corresponding to one or more business modules 110 are obtained in advance, and the data tables corresponding to the data models of each business module are created in the data acquisition device 120. It can be understood that each business module 110 corresponds to a data table.

[0031] A message topic corresponding to each data table can be created in the message server 130. It can be understood that each message topic created here also corresponds to a business module. In addition, the data acquisition device 120 can obtain the change data in each data table in the data acquisition device 120 by collecting the binlog log corresponding to each data table, and can encapsulate the change data in each data table as a message body, and send the message body to the message server 130 based on the message topic corresponding to the data table, so that the message body is stored in the message server 130 based on the corresponding message topic. Here, the message server 130 can be implemented as a kafka message middleware, for example, but is not limited to this.

[0032] The data processing server 140 is respectively connected to the message server 130 and the data storage device 150 for communication. The data processing server 140 monitors whether the message server 130 includes a message body. When the message body is monitored, the data processing server 140 obtains one or more message bodies corresponding to each message topic from the message server 130, and the data processing server 140 can process the obtained message body into corresponding changed data and store it in the data storage device 150. After obtaining the changed data from the data processing server 140, the data storage device 150 can update the data in the data storage device 150.

[0033] In one embodiment, the data processing server 140 can obtain the change data and the corresponding business module identifier in the data table corresponding to the message body by parsing the message body, and store the change data in association with the business module identifier in the data storage device 150. It should be noted that the data table includes a main table and a sub-table associated with the main table. According to one embodiment, when parsing the message body, Goodsld can be used to query and obtain the sub-table information in the data collection device 120.

[0034] In one embodiment, the system 100 further includes a business resource server 160 and a resource server 170 corresponding to each business module. The data storage device 150 is communicatively connected to the business resource server 160 corresponding to each business module 110 and is communicatively connected to the resource server 170 .

[0035] The data in the data storage device 150 and the changed data can be provided to the business resource server 160 corresponding to the corresponding business module 110. Here, the changed data in the data storage device 150 can be sent to the corresponding business resource server 160 based on the associated business module identifier. In one embodiment, the system 100 also includes a message queue server 156, and the message queue server 156 includes a plurality of message queues. The data storage device 150 is connected to each business resource server 160 based on the corresponding message queue, so that the changed data in the data storage device 150 can be sent to the business resource server 160 corresponding to the business module identifier via the message queue based on the associated business module identifier. After the business resource server 160 obtains the corresponding changed data from the data storage device 150, it can be used for search recommendation of the corresponding business module based on the segmented data. In this way, each business resource server 160 can provide personalized search and recommendation services.

[0036] In addition, the data and the changed data in the data storage device 150 can also be provided to the resource server 170 without distinction. In one embodiment, the resource server 170 is connected to the search engine 175 in communication, for example, through a wired or wireless network connection. Based on the changed data obtained from the data storage device 150, the resource server 170 allows users to search for the newly added changed data in the resource server 170 based on the search engine 175. Further, the resource server 170 can send the obtained change data to the search engine 175 in real time, and the search engine 175 can generate a corresponding index based on the real-time obtained change data, so that users can search for data based on the corresponding index on the search engine page. Here, the attributes in the index are the query conditions required when searching for data. It should also be pointed out that the data obtained by the resource server 170 is the full amount of data involving multiple business modules, so that the data in the resource server 170 can be used for profiling and data analysis.

[0037] In one embodiment, the system 100 further includes a comparison module 125, which is respectively connected to the data storage device 150 and the data acquisition device 120. The comparison module 125 has a timing function, and can compare the data in the data storage device 150 and the data acquisition device 120 at predetermined intervals to determine whether there is missing data in the data storage device 150, and the missing data may specifically include data missing from the data acquisition device 120 and data different from the data acquisition device 120. If there is missing data, the comparison module 125 generates a compensation task based on the missing data, and stores the missing data back into the data storage device 150 by executing the compensation task. In this way, by compensating for the missing data of the data storage device 150 at regular intervals, the consistency and real-time performance of the data in the data storage device 150 and the data acquisition device 120 are ensured.

[0038] In an embodiment of the present invention, the data processing server 140 is adapted to execute the data processing method 300. The data processing method 300 of the present invention will be described in detail below.

[0039] According to the system 100 of the present invention, the data of each business module is collected by a data acquisition device, and the change data of each business module is obtained in real time. The change data of each business module is encapsulated into a message body, and sent to a message server based on the message subject corresponding to the business module, and the message body is obtained in real time by a data processing device, and the change data obtained after parsing is stored in the data storage device in association with the business module identifier. In this way, the present invention realizes the integration of the data of each business module in a non-invasive manner, will not affect the functions of each business module, and can write the change data in the data acquisition device to the data storage device in real time, ensure that the data in the data storage device is updated in real time and is consistent with the data of the data acquisition device. In addition, the data in the data storage device can be distributed to the corresponding business resource server, so that each business resource server can provide personalized search and recommendation services. In addition, by compensating for the data missed by the data storage device at regular intervals, the consistency and real-time of the data in the data storage device and the data acquisition device can be guaranteed, making the system data more stable and reliable.

[0040] In one embodiment, the data collection device 120, the message server 130, the data processing server 140, the data storage device 150, the business resource server 160, and the resource server 170 in the present invention can be implemented as a computing device. The data processing server 140 can be implemented as a computing device, so that the data processing method 300 of the present invention can be executed in the computing device.

[0041] Figure 2 FIG. 2 shows a structural diagram of a computing device 200 according to an embodiment of the present invention.

[0042] like Figure 2 As shown, in a basic configuration 202, computing device 200 typically includes a system memory 206 and one or more processors 204. A memory bus 208 may be used for communication between processor 204 and system memory 206.

[0043] Depending on the desired configuration, the processor 204 can be any type of processing, including but not limited to: a microprocessor (UP), a microcontroller (UC), a digital information processor (DSP), or any combination thereof. The processor 204 can include one or more levels of cache such as a primary cache 210 and a secondary cache 212, a processor core 214, and registers 216. An example processor core 214 can include an arithmetic logic unit (ALU), a floating point unit (FPU), a digital signal processing core (DSP core), or any combination thereof. An example memory controller 218 can be used with the processor 204, or in some implementations, the memory controller 218 can be an internal part of the processor 204.

[0044] Depending on the desired configuration, the system memory 206 can be any type of memory, including but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. The system memory 106 may include an operating system 220, one or more applications 222, and program data 224. The application 222 is actually a plurality of program instructions, which are used to instruct the processor 204 to perform corresponding operations. In some embodiments, the application 222 can be arranged to cause the processor 204 to operate using the program data 224 on the operating system.

[0045] The computing device 200 may also include a storage interface bus 234. The storage interface bus 234 enables communication from storage devices 232 (e.g., removable storage 236 and non-removable storage 238) to the basic configuration 202 via the bus / interface controller 230. At least a portion of the operating system 220, the application 222, and the data 224 may be stored on the removable storage 236 and / or the non-removable storage 238, and when the computing device 200 is powered on or the application 222 is to be executed, it is loaded into the system memory 206 via the storage interface bus 234 and executed by the one or more processors 204.

[0046] The computing device 200 may also include an interface bus 240 that facilitates communication from various interface devices (e.g., output devices 242, peripheral interfaces 244, and communication devices 246) to the basic configuration 202 via the bus / interface controller 230. Example output devices 242 include a graphics processing unit 248 and an audio processing unit 250. They can be configured to facilitate communication with various external devices such as a display or speakers via one or more A / V ports 252. Example peripheral interfaces 244 may include a serial interface controller 254 and a parallel interface controller 256, which may be configured to facilitate communication with external devices such as input devices (e.g., keyboards, mice, pens, voice input devices, touch input devices) or other peripherals (e.g., printers, scanners, etc.) via one or more I / O ports 258. Example communication devices 246 may include a network controller 260, which may be arranged to facilitate communication with one or more other computing devices 262 via a network communication link via one or more communication ports 264.

[0047] A network communication link can be an example of a communication medium. Communication media can generally be embodied as computer-readable instructions, data structures, program modules in a modulated data signal such as a carrier wave or other transmission mechanism, and can include any information delivery medium. A "modulated data signal" can be a signal in which one or more of its data sets or its changes can be performed in the signal in a manner that encodes information. As non-limiting examples, communication media can include wired media such as a wired network or a dedicated line network, and various wireless media such as sound, radio frequency (RF), microwave, infrared (IR) or other wireless media. The term computer-readable medium used herein can include both storage media and communication media.

[0048] In the computing device 200 according to the present invention, the application 222 includes multiple program instructions for executing the data processing method 300 , and these program instructions can instruct the processor 204 to execute the data processing method 300 of the present invention, so that the computing device 200 executes the data processing method 300 of the present invention. Figure 3 The flowchart of a data processing method 300 according to one embodiment of the present invention is shown. The method 300 is suitable for being executed in a data processing server 140 (eg, the aforementioned computing device 200).

[0049] like Figure 3 As shown, the method 300 starts at step S310 .

[0050] In step S310, data in one or more business modules are extracted to the data collection device 120 based on the corresponding data tables. Here, for example, the CDC synchronization tool can be used to extract the data of each business module to the data collection device 120, but it is not limited thereto, and all synchronization tools in the prior art are within the protection scope of the present invention.

[0051] In one embodiment, before executing step S310, data models corresponding to one or more business modules 110 may be acquired in advance, and data tables corresponding to each data model may be created in the data acquisition device 120. It is understood that each business module 110 corresponds to a data table. In this way, data in one or more business modules 110 may be extracted to the data acquisition device 120 based on the corresponding data tables.

[0052] In step S320, a message topic corresponding to each data table is created in the message server 130. It can be understood that each message topic created here also corresponds to a business module.

[0053] In step S330, by collecting the binlog log corresponding to each data table in the data acquisition device 120, the change data in each data table in the data acquisition device 120 is obtained in real time. In addition, the change data in each data table is encapsulated as a message body, and the message body is sent to the message server 130 based on the message subject corresponding to the data table, so that the message body is stored in the message server 130 based on the corresponding message subject. Here, the message server 130 can be implemented as a kafka message middleware, for example, but is not limited thereto.

[0054] In step S340, by monitoring the message body in the message server 130, when the message body is monitored, one or more message bodies corresponding to each message topic are obtained from the message server 130. Then, the obtained message body is processed into corresponding change data, and the change data is sent to the data storage device 150 and stored in the data storage device 150. In this way, after obtaining the change data, the data storage device 150 can update the corresponding data in the data storage device 150.

[0055] According to the above steps, the changed data in the data acquisition device 120 can be written into the data storage device 150 in real time, ensuring that the data in the data storage device 150 is updated in real time and is consistent with the data of the data acquisition device.

[0056] In one embodiment, when the message body is processed into corresponding change data, the message body is parsed to obtain the change data and the corresponding business module identifier in the data table corresponding to the message body. The change data is then stored in the data storage device 150 in association with the business module identifier. It should be noted that the data table includes a main table and a secondary table associated with the main table. According to an embodiment, when parsing the message body, Goodsld can be used to query and obtain the secondary table information in the data collection device 120.

[0057] In one embodiment, the message body can be processed into corresponding change data based on multi-threading, and the change data is stored in a data storage device. Specifically, a thread pool including multiple threads is first created. Subsequently, the message body is obtained from the message server 130 based on the created multiple threads, and the message body is processed into corresponding change data, and the change data is stored in the data storage device 150. Here, the present invention does not specifically limit the number of threads created. For example, in one embodiment, the number of threads to be created can be determined based on the number of CPU cores of the data processing server 140. It should be pointed out that the use of multi-threading to process the message body is conducive to improving the efficiency of data processing. When the data changes, it can ensure that the data in the data storage device 150 is updated in time to ensure the real-time consistency of the data.

[0058] In step S350 , the newly added changed data in the data storage device 150 is sent to the business resource server 170 corresponding to the corresponding business module.

[0059] In one embodiment, the change data in the data storage device 150 is sent to the corresponding message queue based on the associated business module identifier, so that the change data is sent to the business resource server 170 corresponding to the business module identifier via the message queue. It should be understood that here, when the change data is sent to the business resource server 160, the change data in the data storage device 150 is differentiated based on the associated business module identifier, and the change data associated with each business module identifier is sent to the business resource server 160 corresponding to the business module identifier. After the business resource server 160 obtains the corresponding change data, it can be used for search recommendation of the corresponding business module based on the segmented data. In this way, each business resource server 160 can provide personalized search and recommendation services.

[0060] In addition, in one embodiment, the changed data in the data storage device 150 can be sent to the resource server 180 without distinction, so that the user can search for the newly added changed data in the resource server 180 based on the search engine. Furthermore, the changed data in the resource server 180 can be sent in real time to the search engine connected thereto. The search engine can generate corresponding indexes based on the real-time acquired change data. In this way, the user can search for data based on the corresponding index on the search engine page. Here, the attributes in the index are the query conditions required for searching data. It should also be pointed out that the data acquired by the resource server 170 is the full amount of data involving multiple business modules, so the data in the resource server 170 can be used for profiling and data analysis.

[0061] In addition, the present invention also utilizes a comparison module 125 connected to the data storage device 150 and the data acquisition device 120 to periodically compare and verify whether the data is consistent. Specifically, the comparison module 125 compares the data in the data storage device 150 and the data acquisition device 120 at predetermined intervals to determine whether there is missing data in the data storage device 150. Here, the missing data may specifically include data that is missing compared to the data acquisition device 120 and data that is different from the data acquisition device 120. If there is missing data, the comparison module 125 generates a compensation task based on the missing data, and stores the missing data back into the data storage device 150 by executing the compensation task. In this way, by periodically compensating for the missing data of the data storage device 150, the consistency and real-time performance of the data in the data storage device 150 and the data acquisition device 120 can be ensured, making the system data more stable and reliable.

[0062] According to the method 300 of the present invention, the data of each business module is collected by a data acquisition device, and the change data of each business module is obtained in real time. The change data of each business module is encapsulated into a message body, and sent to a message server based on the message subject corresponding to the business module. The message body is obtained in real time by a data processing device, and the change data obtained after parsing is stored in a data storage device in association with the business module identifier. In this way, the present invention realizes the integration of the data of each business module in a non-invasive manner without affecting the functions of each business module, and can write the change data in the data acquisition device to the data storage device in real time, ensuring that the data in the data storage device is updated in real time and is consistent with the data of the data acquisition device. In addition, the data in the data storage device can be distributed to the corresponding business resource servers, so that each business resource server can provide personalized search and recommendation services.

[0063] In addition, the present invention can ensure the consistency and real-time performance of data in the data storage device and the data acquisition device by regularly compensating for data missed by the data storage device, making the system data more stable and reliable.

[0064] A8. A method as described in any one of A1-A7, wherein before extracting data in one or more business modules to a data acquisition device based on corresponding data tables, the method includes the steps of: obtaining data models corresponding to the one or more business modules respectively; and creating data tables corresponding to each data model in the data acquisition device.

[0065] B10. A system as described in B9, further comprising: a resource server connected to the data storage device and suitable for obtaining change data from the data storage device; a search engine connected to the resource server and suitable for receiving the change data sent by the resource server in real time, and suitable for generating corresponding indexes based on the change data.

[0066] B11. A system as described in B9, which further includes: a comparison module, which is respectively connected to the data storage device and the data acquisition device, and is suitable for comparing the data in the data storage device and the data acquisition device at predetermined intervals to determine whether there is missing data in the data storage device; if there is missing data, a compensation task is generated based on the missing data, and the missing data is stored back to the data storage device by executing the compensation task.

[0067] The various techniques described herein may be implemented in combination with hardware or software, or a combination thereof. Thus, the method and apparatus of the present invention, or some aspects or parts of the method and apparatus of the present invention may be in the form of program codes (i.e., instructions) embedded in a tangible medium, such as a removable hard disk, a USB flash drive, a floppy disk, a CD-ROM, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for practicing the present invention.

[0068] In the case where the program code is executed on a programmable computer, the mobile terminal generally includes a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store the program code; the processor is configured to execute the data processing method of the present invention according to the instructions in the program code stored in the memory.

[0069] By way of example and not limitation, readable media include readable storage media and communication media. Readable storage media stores information such as computer readable instructions, data structures, program modules or other data. Communication media generally embody computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and include any information transfer medium. Any combination of the above is also included within the scope of readable media.

[0070] In the description provided herein, algorithms and displays are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the examples of the present invention. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to implement the content of the present invention described herein, and the above description of specific languages ​​is for the purpose of disclosing the best mode of the present invention.

[0071] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0072] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed invention requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0073] Those skilled in the art will appreciate that the modules or units or components of the devices in the examples disclosed herein may be arranged in the devices described in the embodiment, or alternatively may be located in one or more devices different from the devices in the examples. The modules in the foregoing examples may be combined into one module or may be divided into multiple submodules.

[0074] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0075] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0076] In addition, some of the embodiments are described herein as methods or combinations of method elements that can be implemented by a processor of a computer system or by other devices that perform the functions. Therefore, a processor with necessary instructions for implementing the method or method elements forms a device for implementing the method or method elements. In addition, the elements described herein of the device embodiments are examples of devices for implementing the functions performed by the elements for the purpose of implementing the invention.

[0077] As used herein, unless otherwise specified, the use of ordinal numbers "first," "second," "third," etc. to describe common objects merely indicates that different instances of similar objects are involved, and is not intended to imply that the objects so described must have a given order in time, space, order, or in any other manner.

[0078] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative and not restrictive with respect to the scope of the present invention, and the scope of the present invention is defined by the appended claims.

Claims

1. A data processing method, executed in a data processing server, wherein the data processing server is respectively connected to a message server and a data storage device, wherein the message server is respectively connected to a data acquisition device, and wherein the data acquisition device is respectively connected to a plurality of business modules, the method comprising the steps of: Obtaining a data model corresponding to each business module, creating a data table corresponding to each data model in a data acquisition device, and extracting data from multiple business modules to the data acquisition device based on the corresponding data tables; Create a message topic corresponding to each data table in the message server; By collecting the binlog log corresponding to each data table in the data collection device, the change data in each data table is obtained in real time, the change data in each data table is encapsulated as a message body, and the message body is sent to the message server based on the corresponding message topic; Obtaining one or more message bodies corresponding to each message topic from the message server, and processing the message bodies into corresponding change data and storing them in a data storage device, so as to update the data in the data storage device; as well as Sending the changed data in the data storage device to the business resource server corresponding to the corresponding business module so that the business resource server can provide search and recommendation services; Through the comparison module, the data in the data storage device and the data acquisition device are compared at predetermined intervals to determine whether there is missing data in the data storage device; if there is missing data, a compensation task is generated based on the missing data, and the missing data is stored back to the data storage device by executing the compensation task.

2. The method according to claim 1, wherein Also includes the steps: The changed data in the data storage device is sent to a resource server, so that the changed data in the resource server is searched based on a search engine.

3. The method according to claim 2, wherein: Also includes the steps: The changed data in the resource server is sent to the search engine in real time so that the search engine can generate corresponding indexes based on the changed data.

4. The method according to any one of claims 1 to 3, wherein The steps of processing the message body into corresponding change data and storing it in the data storage device include: Parse the message body to obtain the change data and corresponding business module identifier in the data table corresponding to the message body; The change data is stored in a data storage device in association with the business module label.

5. The method according to claim 4, wherein: The step of sending the changed data in the data storage device to the business resource server corresponding to the corresponding business module includes: The changed data in the data storage device is sent to the corresponding message queue based on the associated business module identifier, so that the changed data is sent to the business resource server corresponding to the corresponding business module based on the message queue.

6. The method according to any one of claims 1 to 3, wherein The steps of processing the message body into corresponding change data and storing it in the data storage device include: Create a thread pool consisting of multiple threads; A message body is obtained from the message server based on multiple threads, and the message body is processed into corresponding changed data and stored in a data storage device.

7. A data processing system comprising: A data acquisition device, communicating with each of the multiple business modules, adapted to create data tables corresponding to the data models of each business module, extract data from the multiple business modules based on the corresponding data tables, and adapted to obtain change data in each data table in real time, and encapsulate the change data in each data table into a message body; A message server connected to the data acquisition device, adapted to create a message subject corresponding to each data table, and adapted to receive a message body sent by the data acquisition device based on the corresponding message subject; a data processing server, connected to the message server, and adapted to execute the method according to any one of claims 1 to 6; a data storage device connected to the data processing server and adapted to obtain and store the changed data from the data processing server so as to update the data in the data storage device; as well as A business resource server corresponding to each business module is connected to the data storage device and is adapted to obtain corresponding change data from the data storage device; a comparison module, connected to the data storage device and the data acquisition device, respectively, adapted to compare the data in the data storage device and the data acquisition device at predetermined intervals to determine whether there is any missing data in the data storage device; If missing data exists, a compensation task is generated based on the missing data, and the missing data is restored to the data storage device by executing the compensation task.

8. The system of claim 7, wherein: Also includes: a resource server connected to the data storage device and adapted to obtain the changed data from the data storage device; The search engine is connected to the resource server, adapted to receive the change data sent by the resource server in real time, and adapted to generate a corresponding index based on the change data.

9. A computing device comprising: at least one processor; as well as A memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, and the program instructions include instructions for executing the data processing method according to any one of claims 1 to 6.

10. A readable storage medium storing program instructions, wherein when the program instructions are read and executed by a computing device, the computing device executes the data processing method according to any one of claims 1 to 6.