A data detection method, device, equipment and readable storage medium

Abstract

The application discloses a data detection method, device and equipment and a readable storage medium. The method comprises the following steps: determining to-be-detected data in a target system; obtaining target detection data corresponding to the to-be-detected data in other systems according to the to-be-detected data; generating a detection result according to the to-be-detected data and the target detection data; determining a service type of a target service corresponding to the to-be-detected data, the target service being a service that has called the to-be-detected data; and determining a solution according to the service type and the detection result. It can be seen that the application can automatically determine data that needs to be detected in the target system, and automatically detect the data, without manual detection. Furthermore, a corresponding solution can be determined according to different service types, so that the detection of data related to various service types can be realized, and the efficiency of data detection is improved.

Description

Technical Field

[0001] This application relates to the field of data verification, and more specifically, to a data detection method, apparatus, device, and readable storage medium. Background Technology

[0002] To facilitate data retrieval, data is typically stored across multiple systems, and the data in each system changes as various business processes unfold. If data in one system changes, the corresponding data in other systems should also be updated accordingly to maintain consistency across the systems. Inconsistency between multiple systems can lead to data corruption.

[0003] Existing data consistency detection methods rely on manual inspection or writing detection logic tailored to each type of business. This is not only time-consuming and labor-intensive, but also cannot simultaneously support data detection for multiple business types, resulting in low data detection efficiency. Summary of the Invention

[0004] This application provides a data detection method, apparatus, device, and readable storage medium that eliminates the need for manual detection and allows for the determination of corresponding solutions based on different business types. This enables the detection of data related to various business types, thereby improving the efficiency of data detection.

[0005] In view of this, embodiments of this application provide a data detection method, which includes:

[0006] Determine the data to be detected in the target system;

[0007] Based on the data to be detected, obtain target detection data corresponding to the data to be detected in other systems;

[0008] Detection results are generated based on the data to be detected and the target detection data;

[0009] Determine the business type of the target service corresponding to the data to be detected, wherein the target service is the service that has called the data to be detected;

[0010] A solution is determined based on the business type and the test results.

[0011] Optionally, determining the data to be detected in the target system includes:

[0012] Real-time monitoring of the target system;

[0013] Data that has undergone data change events in the target system is identified as data to be detected.

[0014] Optionally, obtaining target detection data corresponding to the data to be detected in other systems based on the data to be detected includes:

[0015] Based on the data to be detected, obtain the target raw data corresponding to the data to be detected in other systems;

[0016] The target original data is converted into a target format to obtain the target detection data, wherein the target format is the format of the data to be detected.

[0017] Optionally, generating a detection result based on the data to be detected and the target detection data includes:

[0018] Calculate the hash value of the data to be detected and the hash value of the target detection data respectively;

[0019] If the hash value of the data to be detected is the same as the hash value of the target data to be detected, then the detection result is that the data is consistent.

[0020] If the hash value of the data to be detected is different from the hash value of the target data to be detected, then the detection result is that the data is inconsistent.

[0021] Optionally, determining the solution based on the business type and the detection results includes:

[0022] If the detection result indicates data inconsistency, a solution table is obtained, which includes multiple business types and corresponding solutions for those business types.

[0023] The solution is determined based on the business type and the solution table.

[0024] This application embodiment also provides a data detection device, the device comprising:

[0025] The first determining unit is used to determine the data to be detected in the target system;

[0026] The acquisition unit is used to acquire target detection data corresponding to the data to be detected in other systems based on the data to be detected;

[0027] A generation unit is used to generate a detection result based on the data to be detected and the target detection data;

[0028] The second determining unit determines the service type of the target service corresponding to the data to be detected, wherein the target service is a service that has called the data to be detected.

[0029] The third determining unit determines a solution based on the business type and the detection results.

[0030] Optionally, the first determining unit is specifically used for:

[0031] Real-time monitoring of the target system;

[0032] Data that has undergone data change events in the target system is identified as data to be detected.

[0033] The acquisition unit is specifically used for:

[0034] Based on the data to be detected, obtain the target raw data corresponding to the data to be detected in other systems;

[0035] The target original data is converted into a target format to obtain the target detection data, where the target format is the format of the data to be detected.

[0036] This application also provides a computer device, including: a memory, a processor, and a bus system;

[0037] The memory is used to store programs;

[0038] The processor is used to execute the program in the memory to implement any of the data detection methods described above;

[0039] The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

[0040] This application also provides a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to perform any of the data detection methods described above.

[0041] This application provides a data detection method, comprising: determining data to be detected in a target system; obtaining target detection data corresponding to the data to be detected in other systems based on the data to be detected; generating a detection result based on the data to be detected and the target detection data; determining the business type of a target service corresponding to the data to be detected, wherein the target service is a service that has called the data to be detected; and determining a solution based on the business type and the detection result. As can be seen, this application can automatically determine the data to be detected in the target system and automatically detect the data based on the business type of the target service corresponding to the data to be detected and the detection result, eliminating the need for manual detection. Furthermore, it can further determine corresponding solutions based on different business types, thereby enabling the detection of data related to multiple business types and improving the efficiency of data detection. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0043] Figure 1 A schematic diagram of a data detection process provided in an embodiment of this application;

[0044] Figure 2 This is a schematic diagram of the structure of a data detection system provided in an embodiment of this application;

[0045] Figure 3 This is a schematic diagram of the structure of a data detection device provided in an embodiment of this application. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0048] To facilitate data retrieval, data is typically stored across multiple systems, and the data in each system changes as various business processes unfold. If data in one system changes, the corresponding data in other systems should also be updated accordingly to maintain consistency across the systems. Inconsistency between multiple systems can lead to data corruption.

[0049] Existing data consistency detection methods rely on manual inspection or writing detection logic tailored to each type of business. This is not only time-consuming and labor-intensive, but also cannot simultaneously support data detection for multiple business types, resulting in low data detection efficiency.

[0050] Therefore, to address the aforementioned issues, this application provides a data detection method, apparatus, device, and readable storage medium that eliminates the need for manual detection and allows for the determination of corresponding solutions based on different business types. This enables the detection of data related to various business types, thereby improving the efficiency of data detection.

[0051] Please see Figure 1 The data detection method provided in this application includes the following steps.

[0052] S101. Determine the data to be detected in the target system.

[0053] In this embodiment, it can be first determined which data in the target system needs to be checked for data consistency. It is understood that, to improve data detection efficiency, the data most likely to have inconsistency issues can be identified from the target system first; for example, recently accessed or modified data in the target system can be used as the data to be checked.

[0054] In one possible implementation, the target system can be monitored in real time; data in the target system that has undergone data change events is identified as data to be monitored. It is understood that when data changes occur in the target system, data in other systems may not be updated in a timely manner, meaning that data that has undergone data change events is likely to have inconsistencies, and this data can be identified as data to be monitored. Specifically, data change events can be collected through database binlog monitoring, message queues, HTTP interfaces, etc. Furthermore, existing technologies typically analyze offline data, making it difficult to detect data anomalies in a timely manner during business operations. This embodiment, by monitoring the target system in real time, can promptly detect data with inconsistencies.

[0055] S102. Obtain target detection data corresponding to the data to be detected from other systems based on the data to be detected.

[0056] In this embodiment, to determine whether there is a data inconsistency issue in the data to be detected, after acquiring the data to be detected in the target system, target detection data corresponding to the data to be detected in other systems can be obtained based on the data to be detected. It is understood that an identifier can be set for the data in each system. Data that needs to be synchronized and consistent across different systems can have the same identifier; that is, other systems can be systems that store data with the same identifier as the data to be detected. After acquiring the data to be detected in the target system, the identifier information of the data to be detected can be obtained, and data that should be synchronized and consistent with the data to be detected can be obtained from other systems based on this identifier information. This data obtained from other systems is then identified as the target detection data. It should be noted that different systems provide data in different ways; data collection can be performed through configuration-enabled support for HTTP interfaces, database queries, and other methods.

[0057] In one possible implementation, target raw data corresponding to the data to be detected can be obtained from other systems based on the data to be detected. The format of the target raw data is then converted to a target format to obtain the target detection data, where the target format is the same as the format of the data to be detected. It is understandable that the data stored in different systems may have different formats; that is, the format of the target raw data corresponding to the data to be detected obtained from other systems may vary, for example, some may be in XML format, and others may be in JSON format. After obtaining the target raw data corresponding to the data to be detected from other systems, the target raw data can be converted to the same format as the data to be detected to obtain the target detection data, facilitating subsequent determination of the data consistency of the data to be detected. Specifically, this can be achieved by introducing Groovy scripts, configuring different scripts for each system, and uniformly formatting the obtained target raw data into the same data structure.

[0058] S103. Generate detection results based on the data to be detected and the target detection data.

[0059] In this embodiment, after obtaining the data to be detected and the target detection data, an equal-value matching can be performed on the data to be detected and the target detection data to determine whether the data to be detected and the target detection data are the same. If the data to be detected and the target detection data are the same, a detection result with consistent data can be generated; if the data to be detected and the target detection data are different, a detection result with inconsistent data can be generated and the detection result can be saved.

[0060] In one possible implementation, the hash values ​​of the data to be detected and the target data to be detected can be calculated separately. If the hash values ​​of the data to be detected and the target data are the same, the detection result is that the data is consistent; if the hash values ​​of the data to be detected and the target data are different, the detection result is that the data is inconsistent. Understandably, if the data to be detected and the target data to be detected are long or contain a lot of information, equivalent matching may take a considerable amount of time. To further improve the efficiency of data detection, the hash values ​​of the data to be detected and the target data to be detected can be calculated directly. The similarity between the two data is determined by comparing their hash values, thereby shortening the data detection time and further improving the efficiency of data detection.

[0061] S104. Determine the service type of the target service corresponding to the data to be detected.

[0062] In this embodiment, the business type of the target service corresponding to the data to be detected can be determined, where the target service is the service that has called the data to be detected. It is understood that, to avoid greater adverse effects caused by data inconsistency, it is possible to determine which services have called or modified the data to be detected, and to identify the services that have called or modified the data as target services. Furthermore, the business type corresponding to the target service can be determined, which may include video upload, video take-off, video call, video information modification, etc.

[0063] S105. Determine a solution based on the business type and the detection results.

[0064] In this embodiment, since different types of services have different processing logics, after determining the target service type and the detection results of the data to be detected, a corresponding solution can be determined based on the service type and the detection results. It is understood that if the detection results show data consistency, the solution is basically the same for most target services, and the detection of the data to be detected can be stopped. If the detection results show data inconsistency, different solutions can be formulated for different service types.

[0065] In one possible implementation, if the detection result indicates data inconsistency, a solution table is retrieved. This table includes multiple business types and their corresponding solutions. The solution is then determined based on the business type and the solution table. It is understood that solutions for each business type can be pre-defined, such as alarm handling and automatic data repair, and the correspondence between each business type and its corresponding solution, as well as the relationship between them, can be stored in the solution table. If the detection result indicates data inconsistency, retrieving the solution table allows for the determination of the solution corresponding to the target business type, thereby enabling the detection of data related to multiple business types and improving the efficiency of data detection.

[0066] Therefore, this application provides a data detection method that can determine the data to be detected in the target system, and determine the solution based on the business type of the target business corresponding to the data to be detected and the detection result after determining the detection result of the data to be detected. It can automatically determine the data that needs to be detected in the target system and automatically detect the data without manual detection. Furthermore, it can further determine the corresponding solution based on different business types, thereby enabling the detection of data related to multiple business types and improving the efficiency of data detection.

[0067] The following section will introduce specific scenarios; please refer to [link / reference]. Figure 2 The data detection system provided in this application includes:

[0068] 1. Event Collector

[0069] The root cause of data inconsistency lies in the mutable nature of data. When data changes, other system data also needs to change accordingly to maintain consistency. This change is an event; multiple data changes constitute multiple events. Event collection aims to handle these events and can provide various collection methods such as database binlog listening, message queues, and HTTP interfaces.

[0070] 2. Data Collector

[0071] The system's goal is data consistency testing, so data collection is necessary before testing can begin. The data collector handles two tasks: first, collecting the raw target data; and second, formatting the raw target data into a uniform format to prepare for subsequent data testing.

[0072] Different systems provide data in different ways. This embodiment provides a configurable solution to address the differences between systems. It can support data collection through various methods such as HTTP interfaces and database queries.

[0073] Besides the different data delivery methods and data structures of each system, Groovy scripts are introduced to uniformly format the acquired raw data into a common data structure. The formatted data structure is not fixed and can be formatted according to event type, providing sufficient flexibility to ensure the feasibility of subsequent data detection.

[0074] 3. Data detector

[0075] The data detector provides its detection logic in the form of Groovy scripts. It supports detection of data with specified target reference values, as well as data detection between different systems. After execution, the detector generates detection results and saves them to a database, providing a data source for subsequent result processing.

[0076] 4. Result Processor

[0077] The result processor is responsible for processing the detection results. For consistent data, the processing is generally the same: stop the scheduled task. For inconsistent data, different logic is applied depending on the specific business logic. To meet this requirement, the ResultHandler interface is abstracted. This interface has only one handle method, and different business logics can implement this interface to handle different tasks. For example, it can be used for alarm processing and automatic data repair.

[0078] The specific process is as follows:

[0079] The event collector can collect events from different channels and generate corresponding scheduled tasks for each event;

[0080] The scheduled task periodically collects the raw target data from each system through the data collector, and then cleans it to obtain the target detection data.

[0081] The data detector compares the data to be detected with the target detection data to obtain the detection results.

[0082] The result processor handles different logic based on different detection results and business types. For example, it may stop scheduled tasks when data is consistent, handle alarms for data inconsistency after a certain period of time, and attempt automatic repair of data inconsistency after a certain period of time.

[0083] The data detection system provided in this application embodiment can provide a universal data consistency detection method storage, highly flexible data collection, data detection and result processing, can detect data inconsistencies in real time, and can provide real-time alarms or even automatic repairs for problematic data, effectively ensuring data reliability.

[0084] Please see Figure 3This application also provides a data detection device, including:

[0085] The first determining unit 301 is used to determine the data to be detected in the target system;

[0086] The acquisition unit 302 is used to acquire target detection data corresponding to the data to be detected in other systems based on the data to be detected;

[0087] The generation unit 303 is used to generate a detection result based on the data to be detected and the target detection data;

[0088] The second determining unit 304 determines the service type of the target service corresponding to the data to be detected, wherein the target service is a service that has called the data to be detected.

[0089] The third determining unit 305 determines a solution based on the business type and the detection results.

[0090] Optionally, the first determining unit 301 is specifically used for:

[0091] Real-time monitoring of the target system;

[0092] Data that has undergone data change events in the target system is identified as data to be detected.

[0093] Optionally, the acquisition unit 302 is specifically used for:

[0094] Based on the data to be detected, obtain the target raw data corresponding to the data to be detected in other systems;

[0095] The target original data is converted into a target format to obtain the target detection data, wherein the target format is the format of the data to be detected.

[0096] Optionally, the generation unit 303 is specifically used for:

[0097] Calculate the hash value of the data to be detected and the hash value of the target detection data respectively;

[0098] If the hash value of the data to be detected is the same as the hash value of the target data to be detected, then the detection result is that the data is consistent.

[0099] If the hash value of the data to be detected is different from the hash value of the target data to be detected, then the detection result is that the data is inconsistent.

[0100] Optionally, the third determining unit 305 is specifically used for:

[0101] If the detection result indicates data inconsistency, a solution table is obtained, which includes multiple business types and corresponding solutions for those business types.

[0102] The solution is determined based on the business type and the solution table.

[0103] Therefore, this application provides a data detection device that can determine the data to be detected in the target system, and determine a solution based on the business type of the target business corresponding to the data to be detected and the detection result after determining the detection result of the data to be detected. It can automatically determine the data that needs to be detected in the target system and automatically detect the data without manual detection. Furthermore, it can further determine the corresponding solution based on different business types, thereby enabling the detection of data related to multiple business types and improving the efficiency of data detection.

[0104] This application also provides a computer device, including: a memory, a processor, and a bus system;

[0105] The memory is used to store programs;

[0106] The processor is used to execute the program in the memory to implement any of the data detection methods described above;

[0107] The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

[0108] This application also provides a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to perform any of the data detection methods described above.

[0109] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0110] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data detection method, characterized in that, The method includes: Identify the data to be detected in the target system; Based on the data to be detected, obtain target detection data corresponding to the data to be detected in other systems; Detection results are generated based on the data to be detected and the target detection data; Determine the business type of the target service corresponding to the data to be detected, wherein the target service is the service that has called the data to be detected; A solution will be determined based on the business type and the test results; The step of determining a solution based on the business type and the detection results includes: If the detection result indicates data inconsistency, a solution table is obtained, which includes various business types and corresponding solutions. The result processing interface corresponding to the business type is called, and the corresponding solution is executed through the result processing interface. The result processing interface is a predefined unified interface, and different business types provide their own different processing logic by implementing the result processing interface. The solution includes alarm processing and automatic data repair processing. The solution is determined based on the business type and the solution table; If the test results show that the data is consistent, then the solution is to stop testing the data to be tested.

2. The method according to claim 1, characterized in that, The process of determining the data to be detected in the target system includes: Real-time monitoring of the target system; The data in the target system that has undergone data change events is identified as the data to be detected.

3. The method according to claim 1, characterized in that, The step of obtaining target detection data corresponding to the data to be detected in other systems based on the data to be detected includes: Based on the data to be detected, obtain the target raw data corresponding to the data to be detected in other systems; The target original data is converted into a target format to obtain the target detection data, wherein the target format is the format of the data to be detected.

4. The method according to claim 1, characterized in that, The step of generating a detection result based on the data to be detected and the target detection data includes: Calculate the hash value of the data to be detected and the hash value of the target detection data respectively; If the hash value of the data to be detected is the same as the hash value of the target data to be detected, then the detection result is that the data is consistent. If the hash value of the data to be detected is different from the hash value of the target data to be detected, then the detection result is that the data is inconsistent.

5. A data detection device, characterized in that, The device includes: The first determining unit is used to determine the data to be detected in the target system; The acquisition unit is used to acquire target detection data corresponding to the data to be detected in other systems based on the data to be detected; A generation unit is used to generate a detection result based on the data to be detected and the target detection data; The second determining unit determines the service type of the target service corresponding to the data to be detected, wherein the target service is a service that has called the data to be detected. The third determining unit determines a solution based on the business type and the detection result. Specifically, determining the solution based on the business type and the detection result involves: if the detection result indicates data inconsistency, obtaining a solution table, which includes multiple business types and corresponding solutions; calling the result processing interface corresponding to the business type and executing the corresponding solution through the result processing interface; the result processing interface is a predefined unified interface, and different business types provide their own different processing logic by implementing the result processing interface; the solution includes alarm processing and automatic data repair processing; determining the solution based on the business type and the solution table; if the detection result indicates data consistency, determining the solution is to stop the detection of the data to be detected.

6. The apparatus according to claim 5, characterized in that, The first determining unit is specifically used for: Real-time monitoring of the target system; The data in the target system that has undergone data change events is identified as the data to be detected.

7. The apparatus according to claim 5, characterized in that, The acquisition unit is specifically used for: Based on the data to be detected, obtain the target raw data corresponding to the data to be detected in other systems; The target original data is converted into a target format to obtain the target detection data, wherein the target format is the format of the data to be detected.

8. A computer device, characterized in that, include: Memory, processor, and bus system; The memory is used to store programs; The processor is used to execute the program in the memory to implement the method of any one of claims 1 to 4; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

9. A computer-readable storage medium, characterized in that, It stores instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1 to 4.

Images