Business data processing method and device, equipment and storage medium

Abstract

The invention discloses a business data processing method and device, equipment and a storage medium. The method is applied to scenes such as big data, and comprises the steps that a first data storage area in a service source database is inquired to obtain first service data, and the service earliest operation time corresponding to each piece of service data in the first data storage area is included in a target service time period; the target business time period comprises the first business time period and a second business time period of a preset business duration corresponding to the target business type, and the end time of the second business time period is the starting time of the first business time period, querying a second data storage area in the total database to obtain second business data, the latest business operation time corresponding to each piece of business data in the second data storage area is contained in the first business time period, and data checking is performed on the first business data and the second business data to obtain a data checking result, so that the business data checking efficiency between the heterogeneous databases is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer information processing, and particularly relates to a business data processing method and device, equipment and a storage medium. BACKGROUND

[0002] With the development of Internet technology, a large amount of business data is generated in various business processes, and the business data needs to be transmitted to the corresponding place for data storage. At present, the business data is mainly stored in a business source database of a business system, and the incremental business data in the business source database is synchronized to a full database of the business system. The business source database usually adopts a relational database with small data storage capacity, and the full database usually adopts a non-relational database capable of storing massive data.

[0003] In order to ensure the consistency of business data storage, data checking needs to be performed on the business source database and the full database. In the prior art, the data checking between heterogeneous databases usually adopts a full data comparison method, which results in low data checking efficiency. SUMMARY

[0004] The present application provides a business data processing method, device, equipment and storage medium, which can reduce the inconsistency of data checking results caused by the difference in data storage structure of heterogeneous databases, improve the accuracy of data checking between heterogeneous databases, and effectively improve the data checking efficiency between heterogeneous databases. The technical solutions of the present application are as follows:

[0005] In one aspect, a business data processing method is provided, which comprises:

[0006] In response to a data checking task for a target business type in a first business time period, a first data storage area in a business source database of a business system is queried to obtain first business data. The business system generates at least one piece of business data, and the earliest operation time of each piece of business data stored in the first data storage area is contained in the target business time period. The target business time period includes the first business time period and a second business time period, the duration of the second business time period is a preset business duration corresponding to the target business type, and the end time of the second business time period is the start time of the first business time period.

[0007] A second data storage area in a full database of the business system is queried to obtain second business data. The latest operation time of each piece of business data stored in the second data storage area is contained in the first business time period.

[0008] The first business data and the second business data are subjected to data checking to obtain a data checking result.

[0009] In another aspect, a business data processing apparatus is provided, the apparatus comprising:

[0010] The first query module is configured to, in response to a data checking task for a target business type in a first business time period, query a first data storage area in a business source database of a business system to obtain first business data, the business system generating at least one piece of business data, and each piece of business data stored in the first data storage area corresponding to a business earliest operation time contained in the target business time period.

[0011] The second query module is configured to query a second data storage area in a full-amount database of the business system to obtain second business data, each piece of business data stored in the second data storage area corresponding to a business latest operation time contained in the first business time period.

[0012] The data checking module is configured to subject the first business data and the second business data to data checking to obtain a data checking result.

[0013] In another aspect, a business data processing device is provided, the device comprising a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the business data processing method of the first aspect.

[0014] In another aspect, a computer readable storage medium is provided, the storage medium storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by a processor to implement the business data processing method of the first aspect.

[0015] In another aspect, a computer program product or computer program is provided, the computer program product or computer program comprising computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the business data processing method of the first aspect.

[0016] The business data processing method, apparatus, device and storage medium provided by the present application have the following technical effects:

[0017] By means of the technical solution of the present application, due to the different data storage structures of the business source database and the full-amount database, the business source database stores according to the earliest operation time of each piece of business data, and the full-amount database stores according to the latest operation time of each piece of business data, in the case where the checking time period of the data checking task is set as the first business time period, the preset business duration corresponding to the target business type can be determined according to the target business type corresponding to the data checking task, the query time period of the business source database is set as the target business time period containing the first business time period and the second business time period, the duration corresponding to the second business time period is the preset business duration corresponding to the target business type, and the end time of the second business time period is the start time of the first business time period, so that the first business data in the business source database corresponding to the earliest operation time contained in the target business time period is checked with the second business data in the full-amount database corresponding to the latest operation time contained in the first business time period, the start point of the query time period of the business source database is extended forward, the data missing check of the business source database is reduced, the inconsistency of the data checking result caused by the difference in the data storage structure of the heterogeneous database is reduced, the accuracy of the data checking between the heterogeneous databases can be improved, and the data checking efficiency between the heterogeneous databases can be effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 is an architecture diagram of a business data processing system provided by an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of a communication network architecture of a business data processing system provided by an embodiment of the present application;

[0021] Figure 3 is a flowchart of a business data processing method provided by an embodiment of the present application;

[0022] Figure 4 is a flowchart of another business data processing method provided by an embodiment of the present application;

[0023] Figure 5 is a flowchart of another business data processing method provided by an embodiment of the present application;

[0024] Figure 6 is a constituent block diagram of a service data processing apparatus provided by an embodiment of the present application;

[0025] Figure 7 is a structural schematic diagram of a service data processing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0027] It should be noted that the terms “include” and “have” and any variations thereof in the specification and claims of the present application and the above-described drawings are intended to cover the non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or units need not be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0028] It can be understood that, in the specific embodiments of the present application, data related to user information is involved, and when the above embodiments of the present application are applied to specific products or technologies, user permission or consent needs to be obtained, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of countries and regions.

[0029] In order to facilitate the understanding of the embodiments of the present application, the following will first briefly introduce several concepts:

[0030] Artificial Intelligence (AI): using digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceiving the environment, acquiring knowledge and using knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0031] Artificial intelligence technology: a comprehensive discipline, involving a wide range of fields, both hardware and software level technology. Artificial intelligence basic technology generally includes, such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning and other several major directions.

[0032] Machine learning (Machine Learning, ML): a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. Specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rule-based learning.

[0033] MySQL: an open source relational database management system based on structured query language (SQL), using row storage mode, with small data storage capacity.

[0034] HBase: an open source non-relational distributed database suitable for storing unstructured or semi-structured data, using column storage mode, capable of storing massive data, mainly used to store massive business data and log data.

[0035] Please refer to Figure 1 , Figure 1 An architecture diagram of a business data processing system provided by the embodiment of the present application. The business data processing system can include a business system, a business source database 101, a full database 102 and a data checking component 103. In an embodiment, the business data processing system can also include a database routing component 104 and a load balancing component 105. In an embodiment, the business data processing system can also include a complement database 106 and a complement component. In an embodiment, the complement component can include a complement log component 107, a complement log collection component 108, a complement message queue component 109 and a complement business transmission component 110. In an embodiment, the business data processing system can also include a metadata database 111 and a business log backup component 112 of the business source database 101.

[0036] The various components mentioned above can be connected through a mobile communication network, a computer network, etc. It should be noted that the various components involved in the embodiments of the present application can be some middleware that performs corresponding processing, which is deployed in a corresponding intelligent device to perform corresponding steps to achieve corresponding functions. The various nodes involved in the embodiments of the present application can also be intelligent devices and the like. The intelligent device can be a terminal or a server. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc., but is not limited thereto. The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0037] At least one piece of business data is generated in the business system. Specifically, each piece of business data can refer to business record data corresponding to a preset business. Each preset business can correspond to a type of business. Illustratively, the type of business can include, but is not limited to, an order business type, a POS (Point of sale) business type, a financial business type, etc. The business system can send a data processing request corresponding to its own business operation to the business source database 101. The data processing request can be a business record data addition request, a business record data update request, etc.

[0038] The business source database 101 can store business data generated by the business system within a preset time period before the current time, i.e., the length of the data storage period of the business source database 101 is the preset time period. The preset time period can be set in advance in combination with the business log update period in actual application and the storage capacity of the business source database. The business source database 101 can be used to receive the data processing request sent by the business system and perform corresponding data processing operations based on the data processing request. It can be understood that the business system can perform at least one business operation for a preset business and send at least one data processing request corresponding to the at least one business operation to the business source database 101. Correspondingly, the business source database 101 can perform at least one data processing operation on the business data of the preset business in response to the at least one data processing request.

[0039] The full-amount database 102 can store full-amount business data generated by a business system. The full-amount database 102 can be used to store business data synchronized from the business source database 101. It can be understood that the business source database 101 can perform at least one data processing operation on the business data of a preset business in response to at least one data processing request for the preset business sent by the business system, and correspondingly synchronize the at least one data processing operation to the full-amount database 102, so that the full-amount database stores the business data corresponding to the at least one data processing operation.

[0040] In an embodiment, the business source database 101 and the full-amount database 102 can be heterogeneous databases, which adopt different data storage formats and structures. Specifically, the business source database 101 can be a relational database, and illustratively, the relational database can include but is not limited to MySQL, Microsoft SQL Server, Microsoft Access, etc. The full-amount database 102 can be a non-relational database, and illustratively, the non-relational database can include but is not limited to Hbase, Hive, etc.

[0041] The data checking component 103 can be used to check whether the data in the business source database 101 and the full-amount database 102 is consistent, and in the case of inconsistency, determine the target business data that needs to be supplemented to the full-amount database 102. Specifically, the data checking component 103 can query the first data storage area in the business source database 101 to obtain first business data in response to a data checking task for a target business type in a first business time period, each piece of business data stored in the first data storage area corresponds to a business earliest operation time contained in the target business time period. The target business time period can include a first business time period and a second business time period, the second business time period corresponds to a preset business duration of the target business type, and the end time of the second business time period is the start time of the first business time period. The data checking component 103 can query the second data storage area in the full-amount database 102 to obtain second business data, each piece of business data stored in the second data storage area corresponds to a business latest operation time contained in the first business time period. Then, the data checking component 103 can perform data checking on the first business data and the second business data to obtain a data checking result.

[0042] The database routing component 104 can be built-in with a sharding rule of the full-amount database 102. The database routing component 104 can parse a business data query request (which can refer to a business data query statement) for the full-amount database sent by the data checking component according to the sharding rule, and route the parsed business data query statement to the corresponding area (database table) in the full-amount database for execution, to obtain a business data layer corresponding to the business data query statement.

[0043] The load balancing component 105 can be configured to balance the load of the data query requests sent by the data reconciliation component to the business source database.

[0044] The complement database 106 can be configured to store the target business data that needs to be supplemented to the full database 102 in the case of inconsistency between the data in the business source database 101 and the full database 102.

[0045] The complement component can be configured to supplement the target business data in the complement database 106 to the full database 102. In an embodiment, the complement component can include a supplement log component 107, a supplement log collection component 108, a supplement message queue component 109, and a supplement business transmission component 110. Specifically, the supplement log component 107 can be a transaction log component of the complement database 106, which can include a binary log component (such as the binlog component of MySQL), and the supplement log component 107 is configured to store the target business data in the complement database 106 in the form of a transaction log. The supplement log collection component 108 can obtain the transaction log recorded by the supplement log component 107, and parse the transaction log to obtain a data processing operation statement, and send the data processing operation statement to the supplement message queue component 109. The supplement message queue component 109 can store the data processing operation statement in the form of a message queue, and the supplement message queue component 109 can be Kafka. The supplement business transmission component 110 can be configured to transmit the data processing operation statement stored in the message queue to the full database 102, wherein the supplement business transmission component 110 can be a DTS (Data Transmission Service) device, which can realize data migration between different databases.

[0046] In an embodiment, the business data processing system can further include a metadata database 111 and a business log backup component 112 of the business source database 101. Specifically, the metadata database 111 can be configured to store the metadata of each component in the business data processing system, and the business log backup component 112 can be configured to store a backup file of the transaction log of the complement database 106.

[0047] In an embodiment, the business source database 101 is located in a first local area network where the business system is located, and the full database 102, the data checking component 103 and the database routing component 104 are located in a second local area network. In related prior art, the data checking component is usually deployed in the same local area network as the business source database, and the data communication between different machine rooms or within the machine room in the same local area network will generate cross traffic. The data query operation of the data checking component on the data of the business source database will increase the cross traffic in the local area network where the business system is located, causing bandwidth overload and communication delay, affecting the related operation of the business system and reducing the processing timeliness of the business system. Therefore, in the embodiment of the present application, the business source database is deployed in the first local area network where the business system is located, and the data checking component is deployed in the second local area network where the full database is located, which can avoid the influence of the data checking process on the operation of the business system. In addition, the database proxy component for analyzing the storage structure rules of the full database is deployed between the data checking component and the full database, and the data checking component queries the second data to be checked from the full database through the database proxy component, which can improve the query efficiency of the data checking component on heterogeneous data and further improve the data checking efficiency.

[0048] In an optional embodiment, the first local area network and the second local area network can communicate data through a public network, and a firewall can be further arranged between the first local area network and the second local area network to ensure the security of data communication. Each local area network can communicate data through a private network.

[0049] In an embodiment, referring to Figure 2 The full database 102 can include a first full database 1021 and a second full database 1022, and the first full database 1021 and the second full database 1022 can be primary and backup storage devices. Correspondingly, the database routing component 104 can include a first routing component 1041 corresponding to the first full database 1021 and a second routing component 1042 corresponding to the second full database 1022, and the data checking component 103 can include a first checking component 1031 and a second checking component 1032. The first checking component 1031, the first routing component 1041 and the first full database 1021 can be deployed in a first data center in the second local area network, and the business source database 101, the first checking component 1031, the first routing component 1041 and the first full database 1021 can constitute a first checking link. The second checking component 1032, the second routing component 1042 and the second full database 1022 can be deployed in a second data center in the second local area network, and the business source database 101, the second checking component 1032, the second routing component 1042 and the second full database 1022 can constitute a second checking link. Here, the first data center and the second data center can refer to machine rooms in different cities in the second local area network.

[0050] Based on the above business data processing system, please refer to Figure 3 , Figure 3 A flowchart of a business data processing method provided by an embodiment of the present application. The business data processing method can be executed by the data checking component 103 in the above business data processing system. The present application provides method operation steps as described in the embodiments or flowcharts, but more or fewer operation steps can be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one of the many execution orders, and does not represent the only execution order. In actual system or product execution, the method order shown in the embodiments or the drawings can be executed in sequence or in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically, as shown in Figure 3 The method can include:

[0051] S301, in response to a data checking task for a target business type in a first business time period, querying a first data storage area in a business source database of a business system to obtain first business data, the business system generating at least one piece of business data, and the earliest operation time corresponding to each piece of business data stored in the first data storage area being contained in the target business time period; the target business time period including the first business time period and a second business time period, the duration of the second business time period being a preset business duration corresponding to the target business type, and the end time of the second business time period being the start time of the first business time period.

[0052] S302, querying a second data storage area in a full database of the business system to obtain second business data, each piece of business data stored in the second data storage area corresponding to a latest operation time of the business contained in the first business time period.

[0053] In the embodiments of the present application, the business system generates at least one piece of business data, specifically, each piece of business data can refer to business record data corresponding to a preset business, and each preset business can correspond to a business type. Illustratively, the business type can include but is not limited to: order business type, POS (Point of sale) business type, financial business type, etc. The business system can send a data processing request corresponding to its own business operation to the business source database, and the data processing request can be a business record data addition request, a business record data update request, etc.

[0054] In the embodiments of the present specification, the business source database can store the business data generated by the business system within a preset time period before the current time, that is, the data storage period of the business source database has a length of the preset time period. The preset time period can be set in advance in combination with the business log update period in actual application and the storage capacity of the business source database. The business source database can be used to receive the data processing request sent by the business system, and perform corresponding data processing operation based on the data processing request.

[0055] In the embodiments of the present specification, the full-amount database can store full-amount business data generated by the business system. The full-amount database can be used to store the business data synchronized from the business source database.

[0056] It can be understood that the business system can perform at least one business operation for a certain preset business, and send at least one data processing request corresponding to the at least one business operation to the business source database. The business source database can perform at least one data processing operation on the business data of the preset business in response to the at least one data processing request, and correspondingly synchronize the at least one data processing operation to the full-amount database, so that the full-amount database stores the business data corresponding to the at least one data processing operation.

[0057] In one embodiment, the business source database and the full-amount database can be heterogeneous databases, and the two adopt different data storage formats and structures. Specifically, the business source database can be a relational database, which can include but is not limited to MySQL, Microsoft SQL Server, Microsoft Access, etc. The full-amount database can be a non-relational database, which can include but is not limited to Hbase, Hive, etc.

[0058] In an embodiment, the business source database can store at least one piece of business data from the business system according to a business earliest operation time corresponding to each piece of business data, the business source database comprises a plurality of first type data storage areas, the area attribute information of each first type data storage area corresponds to a sub-storage period in the data storage period of the business source database, and the first type data storage area corresponding to a sub-storage period of the business source database can store business data whose corresponding business earliest operation time is contained in the sub-storage period; the full-amount database can store at least one piece of business data from the business source database according to a business latest operation time corresponding to each piece of business data, and the full-amount database comprises a plurality of second type data storage areas, the full-amount database comprises a plurality of second type data storage areas, and the area attribute information of each second type data storage area corresponds to a sub-storage period in the data storage period of the full-amount database, and the second type data storage area corresponding to a sub-storage period of the full-amount database can store business data whose corresponding business latest operation time is contained in the sub-storage period. It can be understood that the "first type data storage area" and the "second type data storage area" herein are only used to distinguish the data storage areas in the business source database and the data storage areas in the full-amount database, and in an optional embodiment, the order of magnitude of the data storage area herein can be a library or a table.

[0059] In an embodiment, the business source database can store at least one piece of business data from the business system according to a business earliest operation time corresponding to each piece of business data, the business source database comprises a plurality of first type data storage areas, the area attribute information of each first type data storage area corresponds to a sub-storage period in the data storage period of the business source database, and the first type data storage area corresponding to a sub-storage period of the business source database can store business data whose corresponding business earliest operation time is contained in the sub-storage period; the full-amount database can store at least one piece of business data from the business source database according to a business latest operation time corresponding to each piece of business data, and the full-amount database comprises a plurality of second type data storage areas, the full-amount database comprises a plurality of second type data storage areas, and the area attribute information of each second type data storage area corresponds to a sub-storage period in the data storage period of the full-amount database, and the second type data storage area corresponding to a sub-storage period of the full-amount database can store business data whose corresponding business latest operation time is contained in the sub-storage period. It can be understood that the "first type data storage area" and the "second type data storage area" herein are only used to distinguish the data storage areas in the business source database and the data storage areas in the full-amount database, and in an optional embodiment, the order of magnitude of the data storage area herein can be a library or a table.

[0060] In an embodiment, the division length of the sub-storage period of the database (the business source database or the full-amount database) can be set in combination with the business data traffic size and data management requirements in actual application. Illustratively, the division length of the sub-storage period can be a month, a day or an hour. Taking the division length of the sub-storage period as a day as an example, when the sub-storage period 1 is set as "April 3, 2024" and the sub-storage period 2 is set as "April 4, 2024", the data storage area corresponding to the sub-storage period 1 can store business data whose target operation time is within April 3, 2024, and the data storage area corresponding to the sub-storage period 2 can store business data whose target operation time is within April 4, 2024. It can be understood that when the data storage area is a first type data storage area, the target operation time is a business earliest operation time, and when the data storage area is a second type data storage area, the target operation time is a business latest operation time.

[0061] In the embodiments of the present disclosure, the region attribute information of the data storage region can include region identification text and region creation time. In an optional embodiment, the region attribute information of a certain data storage region corresponding to a certain sub storage period can include that the time representation text contained in the region identification text of the certain data storage region matches the period representation text of the sub storage period, or the region creation time of the certain data storage region matches the period representation text of the sub storage period. Illustratively, taking the table with the magnitude unit of the data storage region as an example, the period representation text of the foregoing sub storage period 1 is “2024 / 4 / 3”, the time representation text contained in the region identification text “db_202404tb_03” of the data storage region D is “20240403”, and the region creation time of the data storage region D is “2024 / 4 / 3 00:00:00”. The period representation text “2024 / 4 / 3” matches the time representation text “20240403”, and the period representation text “2024 / 4 / 3” matches the region creation time “2024 / 4 / 3 00:00:00”. It is considered that the region attribute information of the data storage region D corresponds to the sub storage period 1. In the optional embodiment, the time representation text, the region creation time and the period representation text can be converted into the same time representation format before the matching processing.

[0062] Illustratively, taking a commodity order as an example of a preset business, at least one business operation of the commodity order can include an ordering operation on March 1, 2024 and a refund operation on March 6, 2024, and at least one data processing request corresponding to the at least one business operation can include a business data addition request corresponding to the ordering operation and a business data update request corresponding to the refund operation. Correspondingly, the business source database determines that the data storage sub-period corresponding to the ordering operation is "March 1, 2024" in response to the business data addition request sent by the business system, adds a piece of business data a in the data storage area A (the area attribute information of the data storage area A corresponds to "March 1, 2024") of itself, and synchronizes the business data addition operation corresponding to the business data addition request to the full-amount database, so that the full-amount database adds a piece of business data a in the data storage area B (the area attribute information of the data storage area B corresponds to "March 1, 2024") of itself. Then, the business source database modifies the business data a to business data b in the data storage area A of itself in response to the business data update request sent by the business system, and synchronizes the business data update operation corresponding to the business data update request to the full-amount database, so that the full-amount database determines that the data storage sub-period corresponding to the refund operation is "March 6, 2024", deletes the business data a in the data storage area B of itself, and adds a piece of business data b in the data storage area C (the area attribute information of the data storage area C corresponds to "March 6, 2024") of itself.

[0063] In the embodiments of the present specification, the first data storage area can be a data storage area of the plurality of data storage areas of the business source database corresponding to the target business time period and the target business time period, and the second data storage area can be a data storage area of the plurality of data storage areas of the full-amount database corresponding to the first business time period and the first business time period.

[0064] In one specific embodiment, in response to a data checking task for a target business type in a first business time period, querying the first data storage area in the business source database of the business system to obtain first business data can include: determining a target business time period containing the first business time period and a second business time period, determining a first sub-storage period associated with the target business time period from the plurality of sub-storage periods of the business source database, thereby determining the first data storage area of the plurality of data storage areas of the business source database corresponding to the first sub-storage period, and taking the business data queried in the first data storage area as the first business data.

[0065] In an optional embodiment, determining the first sub-storage period in the plurality of sub-storage periods of the business source database associated with the target business time period can include: determining the first sub-storage period in the plurality of sub-storage periods of the business source database that has a time overlap with the target business time period.

[0066] In an optional embodiment, the first sub-storage period can be a sub-storage period with a corresponding period range that is the target business time period.

[0067] In an optional embodiment, determining the second sub-storage period in the plurality of sub-storage periods of the full database associated with the first business time period can include: determining the second sub-storage period in the plurality of sub-storage periods of the full database that has a time overlap with the first business time period.

[0068] In an optional embodiment, determining the second sub-storage period in the plurality of sub-storage periods of the full database associated with the first business time period can include: determining the second sub-storage period in the plurality of sub-storage periods of the full database that has a time overlap with the first business time period.

[0069] In an optional embodiment, the second sub-storage period can be a sub-storage period with a corresponding period range that is the first business time period.

[0070] In the embodiments of the present specification, the first business time period can be a check time period configured in the data check task, the second business time period can be an extended time period before the check time period, and the length of the second business time period can be a preset business duration corresponding to the target business type in the data check task.

[0071] In optional embodiments, the preset business duration corresponding to each business type in the plurality of business types can be set in advance. Generally, the preset business duration can be set in combination with the business operable duration of the corresponding business type. Illustratively, when the business type is an order business type, the business system sets that the order can be refunded within 7 days, and it can be considered that the business operable duration of the order business type is 7 days, and thus the preset business duration corresponding to the order business type is set to 7 days. When the business type is a deposit business type, the business system sets that the deposit period of the deposit is 3 months, and it can be considered that the business operable duration of the deposit business type is 3 months, and thus the preset business duration corresponding to the order business type is set to 3 months.

[0072] In the related prior art, the business source database and the full database are usually checked for data in the same checking time period. In the case that the length of the checking time period is short, the latest operation data of the corresponding data in the business source database in the checking time period but stored in the historical storage area (the sub-storage period corresponding to the historical storage area is the historical time period, and the end time of the historical time period is before the start time of the query time period) may be missed, resulting in inconsistent data checking results between the business source database and the full database. For example, the business data b of the order is taken as an example. In the case that the checking time period is set to 0 o'clock on March 6, 2024 to 24 o'clock on March 6, 2024, the business data b of the order cannot be queried in the data storage area of the business source database corresponding to the checking time period, but the business data b of the order can be queried in the data storage area C of the full database corresponding to the checking time period, resulting in inconsistent final data checking results.

[0073] Therefore, in the embodiment of the present application, in the case that the checking time period of the data checking task is set to the first business time period, the preset business duration corresponding to the target business type can be determined according to the target business type corresponding to the data checking task, and the query time period of the business source database is set to the target business time period containing the first business time period and the second business time period, wherein the duration corresponding to the second business time period is the preset business duration corresponding to the target business type, and the end time of the second business time period is the start time of the first business time period, so that the first business data in which the earliest operation time of the corresponding business is contained in the target business time period is checked for data with the second business data in which the latest operation time of the corresponding business is contained in the first business time period in the full database, the start point of the query time period of the business source database is extended forward, the data missing of the business source database is reduced, and the inconsistency of the data checking results caused by the difference in the data storage structure of the heterogeneous database is reduced. For example, the preset business duration corresponding to the order business type is taken as 7 days. In the case that the checking time period is set to 0 o'clock on March 6, 2024 to 24 o'clock on March 6, 2024, the query time period of the full database is 0 o'clock on March 6, 2024 to 24 o'clock on March 6, 2024, the query start point of the business source database is extended from 0 o'clock on March 6, 2024 to 0 o'clock on February 27, 2024, and the query time period of the business source database is 0 o'clock on February 27, 2024 to 24 o'clock on March 6, 2024, so that the business data b can be queried in the business source database and the full database, and the accuracy of the data checking is improved.

[0074] In a specific embodiment, before configuring the data checking task, task initialization processing can be performed. Specifically, the task initialization processing can include:

[0075] 1) Query the metadata information of the service source database and the metadata information of the full-amount database. Illustratively, the metadata information can include the mapping relationship information between the service type and the service database (database file).

[0076] 2) Query the database agent information, which can include the related information of the load balancing component and the related information of the database routing component.

[0077] 3) Build a first check link corresponding to the first full-amount database and a second check link corresponding to the second full-amount database in the full-amount database.

[0078] 4) Query the bridging information between the data query link and the specific instance information in the database, and associate the multiple instance information corresponding to the same service object.

[0079] 5) Cache the information queried in steps 1) to 4) in the memory of the data check component.

[0080] 6) If there is check task configuration information in the data check component, generate a data check task according to the check task configuration information, otherwise create the check task configuration information.

[0081] In a specific embodiment, after the data check task configuration is completed, the data check task can be scheduled. Specifically, the data check component can include multiple check task execution units, and the task scheduling process can include:

[0082] 1) Send the data check task to the message queue, and distribute the data check task to the check task execution unit by the message queue.

[0083] 2) The check task execution unit performs data query on the service source database according to the data check task to obtain first query data, and performs data query on the full-amount database to obtain second query data.

[0084] 3) Aggregate the first query data according to the table structure of the full-amount database to obtain first aggregated data, and aggregate the second query data according to the table structure of the full-amount database to obtain second aggregated data.

[0085] 4) Perform data check processing on the first aggregated data and the second aggregated data to obtain a data check result.

[0086] 5) If the data check result indicates that the data is consistent, the task execution is considered successful; if the data check result indicates that the data is inconsistent, the data check task is executed again, and a fuse mechanism is set, and when the number of retries reaches a preset number threshold, the task execution is considered failed.

[0087] In an optional embodiment, when the business data of the same business type is queried for the first time, the field metadata corresponding to the business type needs to be requested from the business source database and stored in the memory area. When the business data of the same business type is queried subsequently, the field metadata can be directly extracted from the memory area to generate a data query statement, thereby improving the data query efficiency.

[0088] In an optional embodiment, the full-amount database can adopt a data table structure of a bucket table combined with an index table. The index table can be used for the positioning query of the business data, and the bucket table can be used for the reconciliation query of a large amount of data. Illustratively, the primary key (RowKey) of the bucket table can be set as SUBSTR (an identification field for partitioning, -3) # the latest operation time field of the business # the globally unique identification field, wherein SUBSTR (an identification field for partitioning, -3) represents the last three digits of the identification field for partitioning. Optionally, the globally unique identification field can be used as the identification field for partitioning. All field values of the corresponding business data can be stored in the row value (RowValue) of the bucket table. The primary key (RowKey) of the index table can be set as REVERSE (an identification field for indexing) # the latest operation time field of the business # the globally unique identification field, wherein REVERSE (an identification field for indexing) represents the reverse operation of the value in the identification field for indexing. Optionally, the globally unique identification field can be used as the identification field for indexing. Generally, the last few digits of the globally unique identification field are used as the basis for partitioning. By reversing the last few digits of the globally unique identification field to the first few digits through the reverse operation, the index query efficiency is improved.

[0089] S303, data reconciliation is performed on the first business data and the second business data to obtain a data reconciliation result.

[0090] In the embodiments of the present specification, the data reconciliation result can be used to represent the data reconciliation between the first business data and the second business data.

[0091] In a specific embodiment, the data reconciliation result can include: the first business data and the second business data are consistent, and there is at least one piece of difference business data in the second business data that is inconsistent with the first business data. Specifically, the at least one piece of difference business data can include: extra business data, and / or missing business data, and / or error business data.

[0092] As can be seen from the above embodiments, due to the difference between the data storage structures of the business source database and the full-amount database, the business source database stores according to the earliest operation time of each piece of business data, and the full-amount database stores according to the latest operation time of each piece of business data. In the case where the checking time period of the data checking task is set as the first business time period, the preset business duration corresponding to the target business type can be determined according to the target business type corresponding to the data checking task, the query time period of the business source database is set as the target business time period containing the first business time period and the second business time period, the duration corresponding to the second business time period is the preset business duration corresponding to the target business type, and the end time of the second business time period is the start time of the first business time period. Thus, the first business data in the business source database corresponding to the earliest operation time contained in the target business time period is checked with the second business data in the full-amount database corresponding to the latest operation time contained in the first business time period, the start point of the query time period of the business source database is extended forward, the data missing check of the business source database is reduced, the inconsistency of the data checking result caused by the difference between the data storage structures of the heterogeneous databases is reduced, the accuracy of the data checking between the heterogeneous databases is improved, and the data checking efficiency between the heterogeneous databases is effectively improved.

[0093] In a specific embodiment, the above data checking of the first business data and the second business data to obtain the data checking result can include:

[0094] S3031, data encoding is performed on the non-Chinese field data in the first business data and the non-Chinese field data in the second business data to obtain the first encoded data corresponding to the first business data and the second encoded data corresponding to the second business data.

[0095] In an optional embodiment, in the case where the business data flow of the business system is large, the preset encoding algorithm can be used to perform data encoding processing on the non-Chinese field data to reduce the memory occupation and the data transmission flow. Illustratively, the preset encoding algorithm can include but is not limited to MD5 (Message-Digest Algorithm, information digest algorithm) and CRC32 algorithm.

[0096] S3032, data checking is performed on the first encoded data and the second encoded data to obtain the first checking result.

[0097] Specifically, the first encoded data and the second encoded data are compared to obtain a first comparison result, the first comparison result can include: the first encoded data and the second encoded data are consistent, or the first encoded data and the second encoded data are inconsistent; in the case that the first encoded data and the second encoded data are consistent, it is considered that the non-Chinese field data in the first service data and the non-Chinese field data in the second service data are consistent, and in the case that the first encoded data and the second encoded data are inconsistent, it is considered that the non-Chinese field data in the first service data and the non-Chinese field data in the second service data are inconsistent.

[0098] S3033, the Chinese field data in the first service data and the Chinese field data in the second service data are compared to obtain a second comparison result.

[0099] In actual application, because the format of the first service data and the format of the second service data are different, the encoded data of the Chinese field data is different, therefore, the Chinese field data in the first service data and the Chinese field data in the second service data are separately compared to obtain the second comparison result.

[0100] Specifically, the Chinese field data in the first service data and the Chinese field data in the second service data are compared to obtain the second comparison result, the second comparison result can include: the Chinese field data is consistent, or the Chinese field data is inconsistent.

[0101] S3034, the data comparison result is obtained according to the first comparison result and the second comparison result.

[0102] As can be seen from the above embodiment, in the case that the service data flow of the service system is large, the non-Chinese field data can be processed by using the preset encoding algorithm for data encoding, so as to reduce the memory occupation and the data transmission flow, and the Chinese field data is separately compared to avoid the influence of different data formats on the accuracy of the comparison result.

[0103] In an optional embodiment, as shown in Figure 4 in the case that the data comparison result indicates that there is at least one piece of difference service data inconsistent with the first service data in the second service data, the above method can further include:

[0104] S401, in the case that the number of the at least one piece of difference service data is less than a preset number threshold, the at least one piece of difference service data is traversed.

[0105] Specifically, the preset number threshold can be preset according to the data comparison efficiency requirement in actual application, in the case that the number of the difference service data is small, the specific positioning query can be performed on each piece of difference service data on the service source database side.

[0106] S402, determine a current query time, the current query time being a business earliest operation time corresponding to the difference business data currently traversed.

[0107] S403, query a third data storage area corresponding to the current query time in the business source database, to obtain third business data.

[0108] Specifically, a third sub-storage period containing the current query time is determined from a plurality of sub-storage periods of the business source database, so as to determine a third data storage area corresponding to the third sub-storage period according to the region attribute information, and the third data storage area can store business data whose corresponding business earliest operation time is contained in the third sub-storage period. The business data queried in the third data storage area is taken as the third business data.

[0109] S404, perform data matching processing on the third business data and the difference business data currently traversed, to obtain a first data matching result corresponding to the difference business data currently traversed.

[0110] Specifically, when the third business data contains business data consistent with the difference business data currently traversed, it is confirmed that the first data matching result is matching success, and when the third business data does not contain business data consistent with the difference business data currently traversed, it is confirmed that the first data matching result is matching failure.

[0111] S405, after the traversal of the at least one difference business data ends, according to the first data matching result corresponding to each of the at least one difference business data, review the at least one difference business data, to generate a data review result of the at least one difference business data.

[0112] As can be seen from the above embodiment, in the case that the number of difference business data is small, specific positioning query can be performed on each difference business data on the business source database side, so as to improve the accuracy of the positioning query of the difference business data, and further improve the accuracy of the data review result.

[0113] In an optional embodiment, as shown in Figure 5 the above method can further include:

[0114] S501, in the case that the number of at least one difference business data is greater than or equal to a preset number threshold, query storage area management information of the business source database, to determine a fourth data storage area in which data update is generated in the first business time period in the business source database.

[0115] Specifically, in the case that the number of the difference service data is large, the efficiency of the specific positioning query of each difference service data on the service source database side is low, and the storage area management information of the service source database records the data update time of each data storage area in the service source database, so that the storage area management information is queried to determine the fourth data storage area in the service source database in which data update is generated in the first service time period, thereby improving the positioning query efficiency of the difference service data.

[0116] S502, at least one difference service data is matched with the service data in the fourth data storage area to obtain a second data matching result corresponding to the at least one difference service data.

[0117] Specifically, when the fourth data storage area has service data consistent with the difference service data, it is confirmed that the second data matching result is a matching success, and when the fourth data storage area does not have service data consistent with the difference service data, it is confirmed that the second data matching result is a matching failure.

[0118] S503, according to the second data matching result, the at least one difference service data is reviewed to generate a data review result.

[0119] As can be seen from the above embodiment, in the case that the number of the difference service data is large, the storage area management information can be queried to determine the fourth data storage area in the service source database in which data update is generated in the first service time period, thereby improving the positioning query efficiency of the difference service data, and further improving the data review efficiency.

[0120] In one specific embodiment, in the case that the data comparison result indicates that the second service data lacks target service data compared with the first service data, the above method can further include:

[0121] S304, obtaining a data operation transaction corresponding to the target service data.

[0122] In one specific embodiment, the transaction log corresponding to the target service data can be extracted from the backup platform according to the data instance corresponding to the target service data, and the data operation transaction can be obtained by analyzing the transaction log.

[0123] S305, the data operation transaction is played back to the complement database, so that the corresponding complement component of the complement database numbers the data processing operation corresponding to the data operation transaction with a first preset numbering step, generates an execution timestamp corresponding to the data processing operation, and controls the full-amount database to execute the data processing operation according to the execution timestamp to supplement the target service data, wherein the first preset numbering step is greater than or equal to 2.

[0124] In one specific embodiment, in the case that the data operation transaction is a data update transaction, the replaying the data operation transaction to the complement database can include:

[0125] S3051, replaying the data update transaction to the complement database, so that the complement component splits the data update operation corresponding to the data update transaction into a data deletion operation and a data addition operation, numbers the data deletion operation and the data addition operation in sequence at a second preset numbering step, generates an execution timestamp, and controls the full-database to execute the data deletion operation and the data addition operation in sequence according to the execution timestamp, wherein the second preset numbering step is smaller than the first preset numbering step.

[0126] Specifically, in the case that the full-database is an Hbase database, since the Hbase database cannot execute data update operations, in the case that the data operation transaction is a data update transaction, the complement component needs to split the data update operation corresponding to the data update transaction into a data deletion operation and a data addition operation. Wherein the complement component numbers the data processing operation corresponding to the data operation transaction at a first preset numbering step greater than or equal to 2, so that when the data update transaction needs to be split, a numbering vacancy can be generated, and the data deletion operation and the data addition operation are numbered in sequence at a second preset numbering step smaller than the first preset numbering step to generate an execution timestamp, and the full-database is controlled to execute the data deletion operation and the data addition operation in sequence according to the execution timestamp, avoiding the influence of operation disorder on data consistency.

[0127] In one specific embodiment, the execution timestamp can be composed of a start timestamp and an incremental timestamp in the transaction log. Specifically, the incremental timestamp can be implemented to increment within a second and reset across seconds, so as to be used to distinguish different data processing operations with the same start timestamp. In an optional embodiment, the start timestamp can be 10 bits, and the incremental timestamp can be 9 bits.

[0128] As can be seen from the above embodiments, in the case where the data checking result indicates that the target business data is missing compared with the first business data, the target business data corresponding data operation transaction is obtained, the execution timestamp corresponding to the data processing operation is generated by the supplement component, and the full-database is controlled to execute the data processing operation according to the execution timestamp to supplement the target business data, so that the missing data can be accurately supplemented, and additional redundant data is avoided. In addition, the supplement component numbers the data processing operation corresponding to the data operation transaction by a first preset numbering step greater than or equal to 2, so that when the data update transaction needs to be split, a numbering gap can be generated, and the data deletion operation and the data addition operation are sequentially numbered to generate execution timestamps by a second preset numbering step smaller than the first preset numbering step, and the full-database is controlled to sequentially execute the data deletion operation and the data addition operation according to the execution timestamps, so as to avoid the influence of operation out-of-order on data consistency.

[0129] In an optional embodiment, the above method is applied to the data checking component, and the above obtaining the second business data from the second data storage area in the full-database of the business system can include:

[0130] S3021, the second business data is obtained by querying the second data storage area in the full-database through the database routing component.

[0131] Among them, the business source database is located in the first local area network where the business system is located, and the full-database, the database routing component and the data checking component are located in the second local area network.

[0132] In related prior art, the data checking component and the business source database are usually deployed in the same local area network, and the data communication between the rooms or in the room will produce cross traffic. The data query operation of the data checking component to the business source database will increase the cross traffic in the local area network where the business system is located, causing bandwidth overload and communication delay, affecting the related operation of the business system, and reducing the processing timeliness of the business system. Therefore, in the embodiment of the present application, the business source database is deployed in the first local area network where the business system is located, and the data checking component is deployed in the second local area network where the full-database is located, so that the operation influence of the data checking process on the business system can be avoided. In addition, the database proxy component for analyzing the storage structure rule of the full-database is deployed between the data checking component and the full-database, and the second to-be-checked data is queried from the full-database by the data checking component through the database proxy component, so as to improve the query efficiency of the data checking component to heterogeneous data, and further improve the data checking efficiency.

[0133] In an optional embodiment, the first local area network and the second local area network can communicate data through a public network, and a firewall can be further arranged between the first local area network and the second local area network to ensure the security of data communication. Each local area network can communicate data through a private network.

[0134] As can be seen from the above embodiments, deploying the service source database in the first local area network where the service system is located and deploying the data checking component to the second local area network where the full-amount database is located can avoid the operation influence of the data checking process on the service system. In addition, deploying the database proxy component for analyzing the storage structure rule of the full-amount database between the data checking component and the full-amount database, and querying the second data to be checked from the full-amount database by the data checking component through the database proxy component can improve the query efficiency of the data checking component on heterogeneous data, and further improve the data checking efficiency.

[0135] According to the technical scheme provided by the embodiment of the present application, because the data storage structures of the business source database and the full database are different, the business source database stores the earliest operation time of each piece of business data, and the full database stores the latest operation time of each piece of business data. In the case that the checking time period of the data checking task is set as the first business time period, the preset business duration corresponding to the target business type can be determined according to the target business type corresponding to the data checking task, the query time period of the business source database is set as the target business time period containing the first business time period and the second business time period, the duration corresponding to the second business time period is the preset business duration corresponding to the target business type, and the end time of the second business time period is the start time of the first business time period. Thus, the first business data in the business source database corresponding to the earliest operation time contained in the target business time period is checked with the second business data in the full database corresponding to the latest operation time contained in the first business time period. By extending the start point of the query time period of the business source database, the data missing check of the business source database is reduced, the inconsistency of the data checking result caused by the difference in the data storage structure of the heterogeneous database is reduced, the accuracy of the data checking between the heterogeneous databases is improved, and the data checking efficiency between the heterogeneous databases is effectively improved. In the case that the number of the difference business data corresponding to the data checking result is small, the specific positioning query of each piece of difference business data can be performed on the business source database side, so that the accuracy of the positioning query of the difference business data is improved, and the accuracy of the data rechecking result is improved. In the case that the number of the difference business data is large, the fourth data storage area in the business source database in which the data is updated in the first business time period can be determined by querying the storage area management information, so that the positioning query efficiency of the difference business data is improved, and the data rechecking efficiency is improved. In the case that the traffic of the business data of the business system is large, the preset encoding algorithm can be used to perform the data encoding processing on the non-Chinese field data, so that the memory occupation is reduced and the data transmission traffic is reduced, and the Chinese field data is checked separately, so that the influence of the different data formats on the accuracy of the checking result is avoided. In addition, the business source database can be deployed in the first local area network where the business system is located, and the data checking component can be deployed in the second local area network where the full database is located, so that the influence of the data checking process on the operation of the business system is avoided. In addition, the database proxy component for analyzing the storage structure rule of the full database is deployed between the data checking component and the full database, the second business data is queried from the full database by the data checking component through the database proxy component, the query efficiency of the heterogeneous data of the data checking component is improved, and the data checking efficiency is improved.

[0136] The embodiment of the present application also provides a business data processing device, which comprises a business source database, a full database, a data checking component and a database proxy component. Figure 6As shown, the business data processing apparatus can apply the data checking component 103 in the business data processing system described above, and the business data processing apparatus can include:

[0137] The first query module 610 is configured to, in response to a data checking task for a target business type in a first business time period, query a first data storage area in a business source database of a business system to obtain first business data, the business system generating at least one piece of business data, and each piece of business data stored in the first data storage area corresponding to a business earliest operation time contained in the target business time period; the target business time period includes the first business time period and a second business time period, the second business time period corresponding to a preset business duration corresponding to the target business type, and the end time of the second business time period being the start time of the first business time period.

[0138] The second query module 620 is configured to query a second data storage area in a full database of the business system to obtain second business data, each piece of business data stored in the second data storage area corresponding to a business latest operation time contained in the first business time period.

[0139] The data checking module 630 is configured to perform data checking on the first business data and the second business data to obtain a data checking result.

[0140] In one specific embodiment, the data checking module 630 described above can include:

[0141] The data encoding unit is configured to perform data encoding on non-Chinese field data in the first business data and non-Chinese field data in the second business data to obtain first encoded data corresponding to the first business data and second encoded data corresponding to the second business data.

[0142] The encoded data checking unit is configured to perform data checking on the first encoded data and the second encoded data to obtain a first checking result.

[0143] The Chinese field data checking unit is configured to perform data checking on Chinese field data in the first business data and Chinese field data in the second business data to obtain a second checking result.

[0144] The data checking result generation unit is configured to obtain the data checking result according to the first checking result and the second checking result.

[0145] In an optional embodiment, in the case where the data checking result indicates that there is at least one piece of difference business data inconsistent with the first business data in the second business data, the apparatus described above can further include:

[0146] The data traversal module is configured to traverse the at least one piece of difference service data when the number of the at least one piece of difference service data is less than the preset number threshold.

[0147] The current query time determination module is configured to determine a current query time, the current query time being an earliest operation time of a service corresponding to the difference service data currently traversed.

[0148] The third query module is configured to query a third data storage area corresponding to the current query time in the service source database to obtain third service data.

[0149] The first data matching result module is configured to perform data matching processing on the third service data and the difference service data currently traversed to obtain a first data matching result corresponding to the difference service data currently traversed.

[0150] The first review module is configured to review the at least one piece of difference service data according to the first data matching result corresponding to each of the at least one piece of difference service data after the traversal of the at least one piece of difference service data is completed, and generate a data review result of the at least one piece of difference service data.

[0151] In an optional embodiment, the apparatus described above can further include:

[0152] The fourth query module is configured to query storage area management information of the service source database to determine a fourth data storage area in which data is updated in the first service time period when the number of the at least one piece of difference service data is greater than or equal to the preset number threshold.

[0153] The second data matching result module is configured to perform data matching processing on the at least one piece of difference service data and service data in the fourth data storage area to obtain a second data matching result corresponding to the at least one piece of difference service data.

[0154] The second review module is configured to review the at least one piece of difference service data according to the second data matching result to generate the data review result.

[0155] In a specific embodiment, when the data review result indicates that the second service data is missing target service data compared with the first service data, the apparatus described above can further include:

[0156] The data operation transaction acquisition module is configured to acquire a data operation transaction corresponding to the target service data.

[0157] The transaction playback module is configured to playback the data operation transaction to the complement database, so that the complement component corresponding to the complement database numbers the data processing operation corresponding to the data operation transaction at a first preset numbering step, generates an execution timestamp corresponding to the data processing operation, and controls the full-database to execute the data processing operation according to the execution timestamp, so as to supplement the target business data, wherein the first preset numbering step is greater than or equal to 2.

[0158] In one specific embodiment, in the case where the data operation transaction is a data update transaction, the transaction playback module can include:

[0159] The data update transaction playback unit is configured to playback the data update transaction to the complement database, so that the complement component splits the data update operation corresponding to the data update transaction into a data deletion operation and a data addition operation, numbers the data deletion operation and the data addition operation in sequence at a second preset numbering step, generates an execution timestamp, and controls the full-database to execute the data deletion operation and the data addition operation in sequence according to the execution timestamp, wherein the second preset numbering step is less than the first preset numbering step.

[0160] In one optional embodiment, the second query module 620 can include:

[0161] The routing query unit is configured to query the second data storage area in the full-database through the database routing component, and obtain the second business data.

[0162] The business source database is located in a first local area network where the business system is located, and the full-database, the database routing component and the data checking component are located in a second local area network.

[0163] It should be noted that the device in the device embodiment and the method embodiment are based on the same inventive concept.

[0164] The embodiments of the present application provide a business data processing device, which includes a processor and a memory. The memory stores at least one instruction or at least one program. The at least one instruction or the at least one program is loaded and executed by the processor to implement the business data processing method provided in the above method embodiments.

[0165] Further, Figure 7 A hardware structure schematic diagram of a business data processing device for implementing the business data processing method provided in the embodiments of the present application is shown, and the business data processing device can participate in constituting or containing the business data processing device provided in the embodiments of the present application. As shown in Figure 7 The business data processing device 70 can include one or more Figure 7The computer system includes a processor 702 (shown as 702a, 702b, ..., 702n) (the processor 702 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 704 for storing data, and a transmission device 706 for communication functions. In addition, the computer system may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply, and / or a camera. It will be understood by those skilled in the art that Figure 7 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 7 More or fewer components than shown, or with Figure 7 Different configurations shown.

[0166] It should be noted that the one or more processors 702 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the business data processing device 70 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0167] The memory 704 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the business data processing method described in the embodiments of the present application. The processor 702 executes various functional applications and data processing by running the software programs and modules stored in the memory 704, that is, implementing the above-mentioned business data processing method. The memory 704 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 704 may further include a memory remotely located relative to the processor 702, and these remote memories may be connected to the business data processing device 70 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0168] The transmission device 706 is configured to receive or send data via a network. The network can include a wireless network provided by a communication provider of the business data processing device 70. In one example, the transmission device 706 includes a network interface controller (NIC) that can connect to other network devices through a base station to communicate with the Internet. In one embodiment, the transmission device 706 can be a radio frequency (RF) module that is configured to communicate with the Internet via a wireless manner.

[0169] The display can be a liquid crystal display (LCD) that is touch screen, for example, which can enable a user to interact with a user interface of the business data processing device 70 (or the mobile device).

[0170] The embodiments of the present application further provide a computer readable storage medium, which can be arranged in the business data processing device to store at least one instruction or at least one program for implementing the business data processing method in the method embodiments. The at least one instruction or the at least one program is loaded and executed by the processor to implement the business data processing method provided in the above method embodiments.

[0171] Optionally, in the present embodiment, the storage medium can be located in at least one of a plurality of network servers of a computer network. Optionally, in the present embodiment, the storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0172] The embodiments of the present application further provide a computer program product or a computer program, which includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the business data processing method provided in the method embodiments.

[0173] It should be noted that the above-mentioned order of the embodiments of the present application is only for description, and does not represent the advantages and disadvantages of the embodiments. And the above describes the specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different than the order in which the steps are recited, and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0174] Each of the embodiments in the present application is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments. Especially, for the device and equipment embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0175] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0176] A person of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by a program instructing related hardware, and the program can be stored in a computer readable storage medium, and the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0177] The above only describes the preferred embodiments of the present application, and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A service data processing method characterized by, The method comprises: In response to a data reconciliation task for a target business type in a first business time period, querying a first data storage area in a business source database of a business system to obtain first business data, the business system generates at least one piece of business data, and the earliest operation time of each piece of business data stored in the first data storage area is contained in the target business time period; the target business time period contains the first business time period and a second business time period, the duration corresponding to the second business time period is a preset business duration corresponding to the target business type, and the end time of the second business time period is the start time of the first business time period; Querying a second data storage area in a full database of the business system to obtain second business data, each piece of business data stored in the second data storage area corresponds to the latest operation time of the business contained in the first business time period; The first business data and the second business data are subjected to data reconciliation to obtain a data reconciliation result.

2. The method of claim 1, wherein, In the case that the data reconciliation result indicates that there is at least one piece of difference business data inconsistent with the first business data in the second business data, the method further comprises: In the case that the number of the at least one piece of difference business data is less than a preset number threshold, traversing the at least one piece of difference business data; Determine the current query time, the current query time is the earliest operation time of the business corresponding to the currently traversed difference business data; Querying a third data storage area corresponding to the current query time in the business source database to obtain third business data; The third business data and the currently traversed difference business data are subjected to data matching processing to obtain a first data matching result corresponding to the currently traversed difference business data; After traversing the at least one piece of difference business data, according to the first data matching result corresponding to each of the at least one piece of difference business data, the at least one piece of difference business data is rechecked to generate a data rechecking result of the at least one piece of difference business data.

3. The method of claim 2, wherein, The method further comprises: In the case that the number of the at least one piece of difference business data is greater than or equal to the preset number threshold, querying the storage area management information of the business source database to determine a fourth data storage area in the business source database that generates data update in the first business time period; The at least one piece of difference business data and the business data in the fourth data storage area are subjected to data matching processing to obtain a second data matching result corresponding to the at least one piece of difference business data; According to the second data matching result, the at least one piece of difference business data is rechecked to generate the data rechecking result.

4. The method of claim 1, wherein, In the case that the data reconciliation result indicates that the second business data is missing target business data compared with the first business data, the method further comprises: Obtaining a data operation transaction corresponding to the target business data; The data operation transaction is played back to the complement database, so that the complement component corresponding to the complement database numbers the data processing operation corresponding to the data operation transaction at a first preset numbering step, generates an execution timestamp corresponding to the data processing operation, and controls the full-amount database to execute the data processing operation according to the execution timestamp, to supplement the target business data, wherein the first preset numbering step is greater than or equal to 2.

5. The method of claim 4, wherein, In the case where the data operation transaction is a data update transaction, the playing back of the data operation transaction to the complement database comprises: The data update transaction is played back to the complement database, so that the complement component splits the data update operation corresponding to the data update transaction into a data deletion operation and a data addition operation, numbers the data deletion operation and the data addition operation in turn at a second preset numbering step, generates the execution timestamp, and controls the full-amount database to execute the data deletion operation and the data addition operation in turn according to the execution timestamp, wherein the second preset numbering step is smaller than the first preset numbering step.

6. The method of claim 1, wherein, The data checking of the first business data and the second business data to obtain a data checking result comprises: data encoding of non-Chinese field data in the first business data and non-Chinese field data in the second business data to obtain first encoded data corresponding to the first business data and second encoded data corresponding to the second business data; data checking of the first encoded data and the second encoded data to obtain a first checking result; data checking of Chinese field data in the first business data and Chinese field data in the second business data to obtain a second checking result; obtaining the data checking result according to the first checking result and the second checking result.

7. The method according to any one of claims 1 to 6, characterized in that, The method is applied to a data checking component, and the querying of a second data storage area in a full-amount database of a business system to obtain second business data comprises: querying, through a database routing component, the second data storage area in the full-amount database to obtain the second business data; wherein the business source database is located in a first local area network where the business system is located, and the full-amount database, the database routing component, and the data checking component are located in a second local area network.

8. A service data processing apparatus characterized by comprising: The device comprises: a first query module configured to, in response to a data checking task for a target business type in a first business time period, query a first data storage area in a business source database of a business system to obtain first business data, the business system generating at least one piece of business data, and a business earliest operation time corresponding to each piece of business data stored in the first data storage area being contained in the target business time period; the target business time period containing the first business time period and a second business time period, a preset business duration corresponding to the target business type corresponding to a duration of the second business time period, and an end time of the second business time period being a start time of the first business time period. The second query module is configured to query a second data storage area in a full database of the business system to obtain second business data, and each piece of business data stored in the second data storage area corresponds to a business latest operation time included in the first business time period. The data checking module is configured to perform data checking on the first business data and the second business data to obtain a data checking result.

9. A service data processing apparatus characterized by comprising: The device comprises a processor and a memory, and the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the business data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the business data processing method according to any one of claims 1 to 7.

11. A computer program product, characterised in that, The computer program product comprises at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the business data processing method according to any one of claims 1 to 7.

Images