Data verification method and device, electronic equipment and computer readable storage medium

Abstract

The invention relates to the technical field of data processing, and provides a data verification method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining to-be-verified data and verification metadata; performing environment verification processing on a preset environment verification script according to the verification metadata to obtain environment verification information; under the condition that the environment verification information represents that the test environment is normal, selecting a target test task from a preset test task pool according to the data attribute of the to-be-verified data; performing deconstruction processing on the target test task to obtain a plurality of task rules; and performing data verification processing on the to-be-verified data according to the plurality of task rules to obtain a data verification result. Through the technical scheme, the data verification process can be more efficient and reliable.

Description

Technical Field

[0001] The embodiments of the present application relate to but are not limited to the field of data processing technology, and in particular, to a data verification method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the continuous development of social economy and the continuous advancement of science and technology, people's living standards are also constantly improving. In the financial industry, various servers are used to store, analyze and process relevant financial data. In order to achieve decentralized processing of quantitative traffic, improve data processing efficiency and ensure the reliability of data transmission, cluster servers are often used to process financial data. However, in the process of processing financial business, cluster relocation may be involved. In the process of cluster relocation, data verification is required, but the current data verification method is relatively cumbersome and cannot achieve efficient data verification processing. Summary of the invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] In order to solve the problems mentioned in the above background technology, the embodiments of the present application provide a data verification method, device, electronic device and computer-readable storage medium, so that the data verification process can be more efficient and reliable.

[0005] In a first aspect, an embodiment of the present application provides a data verification method, the method comprising:

[0006] Obtain the data to be verified and the verification metadata;

[0007] Performing environment verification processing on a preset environment verification script according to the verification metadata to obtain environment verification information;

[0008] When the environment verification information indicates that the test environment is normal, selecting a target test task from a preset test task pool according to the data attributes of the data to be verified;

[0009] Deconstructing the target test task to obtain multiple task rules;

[0010] The data to be verified is subjected to data verification processing according to the plurality of task rules to obtain a data verification result.

[0011] According to some embodiments of the present application, performing environment verification processing on a preset environment verification script according to the verification metadata to obtain environment verification information includes:

[0012] Analyzing and processing the environment verification script to obtain multiple verification sub-scripts;

[0013] The plurality of verification sub-scripts are executed based on the verification metadata to obtain the environment verification information.

[0014] According to some embodiments of the present application, there are multiple test tasks in the test task pool, each of the test tasks carries data type information, and the target test task is selected from the preset test task pool according to the data attribute of the data to be verified, including:

[0015] Matching the data attributes of the to-be-verified data with the data type information of each of the test tasks in the test task pool;

[0016] The test task corresponding to the data type information matching the data attribute of the data to be verified is used as the target test task.

[0017] According to some embodiments of the present application, the target test task is deconstructed to obtain a plurality of task rules, including:

[0018] Decomposing the target test task according to preset functional division information to obtain a plurality of test functional subtasks, wherein each of the test functional subtasks carries priority information;

[0019] Sorting the plurality of test function subtasks according to the priority information to obtain a task test chain;

[0020] Port configuration processing is performed on the task test chain to obtain a plurality of task rules.

[0021] According to some embodiments of the present application, performing data verification processing on the data to be verified according to the plurality of task rules to obtain a data verification result includes:

[0022] Extracting multiple groups of target verification data from the data to be verified;

[0023] Execute the plurality of task rules according to the target verification data to obtain data verification information;

[0024] The multiple data verification information are fused and processed to obtain the data verification result.

[0025] According to some embodiments of the present application, extracting multiple groups of target verification data from the data to be verified includes:

[0026] Preprocessing the data to be verified to obtain preprocessed verification data;

[0027] A plurality of groups of target verification data are obtained by screening the preprocessed verification data according to preset verification target information.

[0028] According to some embodiments of the present application, after performing data verification processing on the data to be verified according to the plurality of task rules and obtaining the data verification result, the method further includes:

[0029] Converting the data verification result into verification portrait data;

[0030] The verification image data is sent to a preset monitoring terminal.

[0031] In a second aspect, an embodiment of the present application further provides a data verification device, including:

[0032] A first processing module is used to obtain the data to be verified and the verification metadata;

[0033] A second processing module is used to perform environment verification processing on a preset environment verification script according to the verification metadata to obtain environment verification information;

[0034] A third processing module is used to select a target test task from a preset test task pool according to data attributes of the data to be verified when the environment verification information indicates that the test environment is normal;

[0035] A fourth processing module, used for deconstructing the target test task to obtain a plurality of task rules;

[0036] The fifth processing module is used to perform data verification processing on the data to be verified according to the multiple task rules to obtain a data verification result.

[0037] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data verification method as described in the first aspect above is implemented.

[0038] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the data verification method described in the first aspect above.

[0039] The data verification method according to the embodiment provided by the present application has at least the following beneficial effects: in the process of data verification, first obtain the data to be verified and the verification metadata; then perform environmental verification processing on the preset environment verification script according to the verification metadata to obtain environmental verification information; then, when the environmental verification information indicates that the test environment is normal, select the target test task from the preset test task pool according to the data attributes of the data to be verified; then deconstruct the target test task to obtain multiple task rules; finally, perform data verification processing on the data to be verified according to the multiple task rules to obtain the data verification result. In the above manner, the data verification process can be more efficient and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0041] Figure 1 is a flow chart of a data verification method provided by an embodiment of the present application;

[0042] Figure 2 is a specific flow chart of step S200 provided in one embodiment of the present application;

[0043] Figure 3 is a specific flow chart of step S300 provided in one embodiment of the present application;

[0044] Figure 4 is a specific flow chart of step S400 provided in one embodiment of the present application;

[0045] Figure 5 is a specific flow chart of step S500 provided in one embodiment of the present application;

[0046] Figure 6 is a specific flow chart of step S510 provided by an embodiment of the present application;

[0047] Figure 7 is a flow chart of a data verification method provided by another embodiment of the present application;

[0048] Figure 8 is a schematic diagram of a data verification device provided by an embodiment of the present application;

[0049] Fig. 9 It is a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0052] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0053] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0054] AI is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Artificial intelligence is a branch of computer science. Artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0055] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0056] Artificial intelligence is AI. AI is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0057] The servers involved in artificial intelligence technology can be independent servers or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0058] The present application provides a data verification method, device, electronic device and computer-readable storage medium. In the process of data verification, firstly, the data to be verified and the verification metadata are obtained; then, the preset environment verification script is subjected to environment verification processing according to the verification metadata to obtain environment verification information; then, when the environment verification information indicates that the test environment is normal, the target test task is selected from the preset test task pool according to the data attributes of the data to be verified; then, the target test task is deconstructed to obtain multiple task rules; finally, the data to be verified is subjected to data verification processing according to the multiple task rules to obtain the data verification result. Through the above method, the data verification process can be more efficient and reliable.

[0059] The data verification method provided in the embodiment of the present application relates to the field of data processing technology. The data verification method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the data verification method, etc., but is not limited to the above forms.

[0060] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0061] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0062] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0063] like Figure 1 As shown, Figure 1 It is a flowchart of a data verification method provided by an embodiment of the present application, and the data verification method includes but is not limited to steps S100 to S500.

[0064] Step S100, obtaining the data to be verified and the verification metadata;

[0065] Step S200, performing environment verification processing on a preset environment verification script according to the verification metadata to obtain environment verification information;

[0066] Step S300, when the environment verification information indicates that the test environment is normal, selecting a target test task from a preset test task pool according to the data attributes of the data to be verified;

[0067] Step S400, deconstructing the target test task to obtain multiple task rules;

[0068] Step S500, performing data verification processing on the data to be verified according to multiple task rules to obtain a data verification result.

[0069] It should be noted that in the process of data verification, first obtain the data to be verified and the verification metadata; then perform environmental verification processing on the preset environmental verification script according to the verification metadata to obtain environmental verification information; then, when the environmental verification information indicates that the test environment is normal, select the target test task from the preset test task pool according to the data attributes of the data to be verified; then deconstruct the target test task to obtain multiple task rules; finally, perform data verification processing on the data to be verified according to the multiple task rules to obtain the data verification result. In the above manner, the data verification process can be more efficient and reliable.

[0070] It is worth noting that the user's permission or consent will be obtained before obtaining the data to be verified and the verification metadata, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or jump to a confirmation page. After the user's separate permission or consent is clearly obtained, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0071] It is worth noting that metadata is data about data, which describes the attributes, characteristics, sources, structure, quality, usage and other information of the data. In different fields and applications, metadata has different meanings and functions. The following are some basic concepts and application scenarios about metadata. Metadata is data that describes data. It provides contextual information about data to help users understand the content, source, structure and purpose of data. Descriptive metadata: describes the content, subject, author, publisher, copyright and other information of data. Structural metadata: describes the organizational structure of data, such as the table structure in the database, the directory structure of the file, etc. Administrative metadata: describes the management information of data, such as the creation time, modification time, access rights, etc. of data. Technical metadata: describes the technical details of data, such as data format, encoding method, storage location, etc. Function: Help users quickly find the required data resources; provide contextual information of data to help users understand the meaning and purpose of data; support the organization, storage, maintenance and update of data; promote the sharing and exchange of data, and improve the availability and value of data. In an embodiment of the present application, after obtaining the data to be verified and the verification metadata, the preset environment verification script can be subjected to environment verification processing according to the verification metadata to obtain environment verification information; and subsequently, when the environment verification information indicates that the test environment is normal, the target test task will be selected from the preset test task pool according to the data attributes of the data to be verified; subsequently, the target test task can be deconstructed to obtain multiple task rules; subsequently, the data to be verified can be subjected to data verification processing according to the multiple task rules to obtain the data verification result.

[0072] It is worth noting that the environment verification information is obtained by performing environment verification processing on the preset environment verification script according to the verification metadata. Subsequently, when the environment verification information indicates that the test environment is normal, the target test task will be selected from the preset test task pool according to the data attributes of the data to be verified. The entire data verification process is stable and reliable.

[0073] It is worth noting that when the environmental verification information indicates that the test environment is normal, the target test task can be selected from the preset test task pool according to the data attributes of the data to be verified; wherein, the test task pool includes multiple test tasks, and based on the corresponding target test tasks, the data to be verified can be well verified to obtain the corresponding data verification results.

[0074] In some embodiments of the present application, multiple task rules can be obtained by deconstructing the target test task. Based on the multiple task rules obtained by the deconstruction, the data to be verified can be subsequently verified according to the multiple task rules to obtain data verification results; through the above technical solution, the data verification process can be simpler and faster.

[0075] It is worth noting that under the complex distributed system architecture, ensuring the consistency of business data is an important issue. Common business application scenarios include offline or online verification, pre- or post-verification. Offline verification usually involves configuring scheduled tasks to compare data at fixed time points, while online verification detects data inconsistencies in real time. Pre-verification is performed before business operations to block operations that do not comply with the rules; post-verification is performed after business operations to detect and fix data inconsistencies. Data verification methods include integrity verification, rule verification, boundary value verification, comparison verification, and data cleaning. In automated testing, the test task pool can be used to manage and execute test cases. Existing tools provide visualization and automation functions to help testers cope with the challenges brought about by changes in requirements. Through automated testing strategies, test cases can be quickly generated, test sequences can be arranged, and fully automated test execution and regression can be achieved, thereby improving test efficiency and quality.

[0076] In some embodiments, Figure 2 As shown, the above step S200 may include but is not limited to steps S210 to S220.

[0077] Step S210, analyzing and processing the environment verification script to obtain multiple verification sub-scripts;

[0078] Step S220: Execute multiple verification sub-scripts based on the verification metadata to obtain environment verification information.

[0079] It should be noted that in the process of performing environment verification processing on the preset environment verification script according to the verification metadata to obtain the environment verification information, the environment verification script is first analyzed and processed to obtain multiple verification sub-scripts; then, the multiple verification sub-scripts are executed based on the verification metadata to obtain the corresponding environment verification information; through the above technical solution, the environment verification is completed to make the prerequisite preparation for the subsequent data verification.

[0080] It is worth noting that by analyzing and processing the environment verification script, multiple verification sub-scripts can be obtained; by executing and processing multiple verification sub-scripts based on the verification metadata, the corresponding environment verification information can be obtained. Only when the environment verification is normal will the subsequent data verification processing continue.

[0081] It should be noted that an environment verification script is an automated script used to verify whether the software or system environment meets expectations. They are usually used to ensure that the environment configuration, dependencies, and system status meet the test requirements at different test stages. Environment verification scripts are automated scripts used to check and verify whether the configuration of the software or system environment is correct; these scripts can automatically perform a series of checks to ensure that all necessary components, libraries, and services are correctly installed and configured.

[0082] In some embodiments, Figure 3 As shown, there are multiple test tasks in the test task pool, each test task carries data type information, and the above step S300 may include but is not limited to steps S310 to S320.

[0083] Step S310, matching the data attributes of the data to be verified with the data type information of each test task in the test task pool;

[0084] Step S320: taking the test task corresponding to the data type information matching the data attribute of the data to be verified as the target test task.

[0085] It should be noted that in the process of selecting the target test task from the preset test task pool according to the data attributes of the data to be verified, the data attributes of the data to be verified are first matched with the data type information of each test task in the test task pool; then the test task corresponding to the data type information matching the data attributes of the data to be verified is taken as the target test task, in order to make the prerequisite preparation for subsequent data verification.

[0086] It is worth noting that the test task pool includes multiple test tasks, each of which can be used to verify the data. Different test tasks can be applied to different application scenarios to adapt to the data verification requirements in different scenarios.

[0087] It is worth noting that in the process of selecting the target test task, the data attributes of the data to be verified are first matched with the data type information of each test task in the test task pool, and finally the test task corresponding to the data type information that matches the data attributes of the data to be verified is used as the target test task to prepare for subsequent data verification processing.

[0088] In some embodiments, Figure 4 As shown, the above step S400 may include but is not limited to step S410 to step S430.

[0089] Step S410, decomposing the target test task according to preset function division information to obtain a plurality of test function subtasks, wherein each test function subtask carries priority information;

[0090] Step S420, sorting the multiple test function subtasks according to the priority information to obtain a task test chain;

[0091] Step S430, performing port configuration processing on the task test chain to obtain multiple task rules.

[0092] It should be noted that in the process of deconstructing the target test task to obtain multiple task rules, the target test task is first decomposed according to the preset function division information to obtain multiple test function sub-tasks, wherein each test function sub-task carries priority information; then the multiple test function sub-tasks are sorted according to the priority information to obtain a task test chain; finally, the task test chain is subjected to port configuration processing to obtain multiple task rules, which makes the prerequisite preparation for subsequent data verification processing.

[0093] It is worth noting that in the process of deconstructing the target test task to obtain multiple task rules, the target test task is first decomposed according to the preset functional division information to obtain multiple test function sub-tasks, and each test function sub-task carries priority information; subsequently, the multiple test function sub-tasks can be sorted according to the priority information to obtain a task test chain; finally, the task test chain is port configured to obtain multiple task rules to realize subsequent data verification processing.

[0094] In some embodiments, Figure 5 As shown, the above step S500 may include but is not limited to step S510 to step S530.

[0095] Step S510, extracting multiple groups of target verification data from the data to be verified;

[0096] Step S520, executing and processing multiple task rules according to the target verification data to obtain data verification information;

[0097] Step S530, multiple data verification information are merged to obtain a data verification result.

[0098] It should be noted that in the process of performing data verification processing on the data to be verified according to multiple task rules to obtain data verification results, multiple groups of target verification data are first extracted from the data to be verified; then, multiple task rules are executed according to the target verification data to obtain data verification information; then, multiple data verification information are fused to obtain data verification results; through the above technical solution, the determination of data verification results can be more accurate and reliable.

[0099] It is worth noting that in the process of performing data verification processing on the data to be verified according to multiple task rules to obtain data verification results, multiple groups of target verification data are first extracted from the data to be verified; then, multiple task rules are executed according to the target verification data to obtain data verification information; finally, multiple data verification information are fused to obtain the corresponding data verification results.

[0100] It is worth noting that handling multiple task rules usually involves aspects such as task scheduling, prioritization, concurrent execution, and resource management. Here are some general steps and strategies for handling and executing multiple task rules: Task identification and classification, identify all tasks that need to be executed. Categorize tasks according to their nature and requirements. Task prioritization, prioritize tasks based on their urgency, importance, or deadline. Use priority queues to manage tasks. Resource allocation, determine the resources required for each task. Assign tasks based on resource availability. Task scheduling, use scheduling algorithms (such as polling, shortest job first, priority scheduling, etc.) to determine the order in which tasks should be executed. Consider task dependencies and ensure that dependent tasks are executed only after the predecessor tasks are completed. Concurrent execution, determine which tasks can be executed in parallel to improve efficiency. Use multithreading or multiprocessing to achieve concurrency. Error handling and logging: Set up error handling mechanisms for each task so that they can be recovered or rescheduled in case of execution failure. Record task execution logs for easy monitoring and debugging. Monitoring and performance optimization, monitor the execution status and performance indicators of tasks. Adjust the execution strategy of tasks based on monitoring results to optimize performance. Task completion and feedback: Ensure that each task has a corresponding feedback mechanism after completion. For long-running tasks, provide progress updates. User interface and interaction: If task execution involves user interaction, design an intuitive user interface. Provide real-time updates of task status. Security and permission management: Ensure data security and privacy protection during task execution. Control access and execution of tasks based on user permissions. Automation and scripting: For repetitive tasks, write automation scripts to reduce manual intervention. Use workflow management systems to automate task scheduling and execution. Testing and verification: Perform sufficient testing before task execution to ensure the correctness of task rules. Verify whether the results of task execution meet expectations. These steps and strategies can be adjusted and optimized according to specific application scenarios and requirements. In actual operations, specific business logic, technical limitations, and organizational structure may also need to be considered.

[0101] In some embodiments, Figure 6 As shown, the above step S510 may include but is not limited to steps S511 to S512.

[0102] Step S511, preprocessing the data to be verified to obtain preprocessed verification data;

[0103] Step S512, multiple groups of target verification data are obtained by screening the pre-processed verification data according to preset verification target information.

[0104] It should be noted that, in the process of extracting multiple sets of target verification data from the data to be verified, the data to be verified is first preprocessed to obtain preprocessed verification data; then, multiple sets of target verification data are screened from the preprocessed verification data according to the preset verification target information, so as to make prerequisite preparations for subsequent data verification processing.

[0105] It is worth noting that in the process of screening multiple sets of target verification data, the verification data can be preprocessed to obtain preprocessed verification data first; then, multiple sets of target verification data can be screened from the preprocessed verification data according to the preset verification target information, in order to prepare for subsequent data verification and meet the requirements of data verification well.

[0106] In some embodiments, Figure 7 As shown, after executing step S500, it may also include but not limited to step S610 and step S620.

[0107] Step S610, converting the data verification result into verification portrait data;

[0108] Step S620, sending the verification portrait data to a preset monitoring terminal.

[0109] It should be noted that after the data to be verified is processed according to multiple task rules to obtain the data verification results, the data verification results are first converted into verification portrait data, and then the verification portrait data is sent to a pre-set monitoring terminal; based on the pre-set monitoring terminal, maintenance personnel can easily and quickly understand the data verification status.

[0110] It is worth noting that the verification of user portrait data is a multi-dimensional and multi-stage process, involving the accuracy, completeness, consistency, uniqueness, standardization and timeliness of data. Six major standards of data quality: Completeness: Check whether the data is missing, such as null values ​​or invalid values. Consistency: Ensure the consistency of data in different systems or applications, such as the consistency of data format and logic. Accuracy: Ensure that the data truly and accurately reflects the entity it models. Uniqueness: Ensure the uniqueness of data in the database and avoid duplicate data. Standardization: The value, format and presentation of data should meet the requirements of data definition and business definition. Timeliness: The real-time and validity of data, ensuring that the update frequency of data meets the requirements. Data quality verification rules: non-empty verification, value range verification, size verification, length verification, uniqueness verification, timeliness verification, etc. Verification in data governance: In the process of data governance, data quality is both a premise and a goal. The closed-loop management of data quality issues is achieved through the data quality problem handling process, which promotes the continuous improvement of data quality. Standardization of regulatory data for banking financial institutions: There are strict regulations for the collection, verification, and submission of banking regulatory data, including the physical storage, transmission form, and collection technology interface of data, as well as clear provisions for the responsible parties for data quality. Through these methods and steps, the accuracy and quality of user profile data can be effectively verified to ensure the availability and reliability of data.

[0111] In addition, if Figure 8 As shown, an embodiment of the present application further provides a data verification device 10, comprising:

[0112] The first processing module 100 is used to obtain the data to be verified and the verification metadata;

[0113] The second processing module 200 is used to perform environment verification processing on a preset environment verification script according to the verification metadata to obtain environment verification information;

[0114] The third processing module 300 is used to select a target test task from a preset test task pool according to data attributes of the data to be verified when the environment verification information indicates that the test environment is normal;

[0115] The fourth processing module 400 is used to deconstruct the target test task to obtain multiple task rules;

[0116] The fifth processing module 500 is used to perform data verification processing on the data to be verified according to multiple task rules to obtain a data verification result.

[0117] It should be noted that in the process of data verification, first obtain the data to be verified and the verification metadata; then perform environmental verification processing on the preset environmental verification script according to the verification metadata to obtain environmental verification information; then, when the environmental verification information indicates that the test environment is normal, select the target test task from the preset test task pool according to the data attributes of the data to be verified; then deconstruct the target test task to obtain multiple task rules; finally, perform data verification processing on the data to be verified according to the multiple task rules to obtain the data verification result. In the above manner, the data verification process can be more efficient and reliable.

[0118] It is worth noting that the user's permission or consent will be obtained before obtaining the data to be verified and the verification metadata, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or jump to a confirmation page. After the user's separate permission or consent is clearly obtained, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0119] It is worth noting that metadata is data about data, which describes the attributes, characteristics, sources, structure, quality, usage and other information of the data. In different fields and applications, metadata has different meanings and functions. The following are some basic concepts and application scenarios about metadata. Metadata is data that describes data. It provides contextual information about data to help users understand the content, source, structure and purpose of data. Descriptive metadata: describes the content, subject, author, publisher, copyright and other information of data. Structural metadata: describes the organizational structure of data, such as the table structure in the database, the directory structure of the file, etc. Administrative metadata: describes the management information of data, such as the creation time, modification time, access rights, etc. of data. Technical metadata: describes the technical details of data, such as data format, encoding method, storage location, etc. Function: Help users quickly find the required data resources; provide contextual information of data to help users understand the meaning and purpose of data; support the organization, storage, maintenance and update of data; promote the sharing and exchange of data, and improve the availability and value of data. In an embodiment of the present application, after obtaining the data to be verified and the verification metadata, the preset environment verification script can be subjected to environment verification processing according to the verification metadata to obtain environment verification information; and subsequently, when the environment verification information indicates that the test environment is normal, the target test task will be selected from the preset test task pool according to the data attributes of the data to be verified; subsequently, the target test task can be deconstructed to obtain multiple task rules; subsequently, the data to be verified can be subjected to data verification processing according to the multiple task rules to obtain the data verification result.

[0120] The specific implementation of the data verification device 10 is substantially the same as the specific implementation of the above-mentioned data verification method, and will not be described in detail herein.

[0121] In addition, if Fig. 9 As shown, an embodiment of the present application further provides an electronic device 700 , which includes: a memory 720 , a processor 710 , and a computer program stored in the memory 720 and executable on the processor 710 .

[0122] The processor 710 and the memory 720 may be connected via a bus or in other ways.

[0123] The non-transitory software programs and instructions required to implement the data verification methods of the above embodiments are stored in the memory 720 , and when executed by the processor 710 , the data verification methods of the above embodiments are executed.

[0124] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0125] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are executed by a processor 710 or a controller, for example, by a processor 710 in the above-mentioned device embodiment, so that the above-mentioned processor 710 can execute the data verification method in the above-mentioned embodiment.

[0126] The above embodiments may be used in combination, and modules with the same name in different embodiments may be the same or different.

[0127] The above describes specific embodiments of the present application, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0128] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0129] The apparatus, device, computer-readable storage medium and method provided in the embodiments of the present application correspond to each other. Therefore, the apparatus, device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the corresponding apparatus, device and computer storage medium will not be repeated here.

[0130] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0131] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0132] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0133] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0134] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0135] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0136] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0138] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0139] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory (Flash RAM). The memory is an example of a computer-readable medium.

[0140] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0141] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0142] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0143] Embodiments of the present application may be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Embodiments of the present application may also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0144] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0145] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A data verification method, characterized in that: The method comprises: Obtain the data to be verified and the verification metadata; Performing environment verification processing on a preset environment verification script according to the verification metadata to obtain environment verification information; When the environment verification information indicates that the test environment is normal, selecting a target test task from a preset test task pool according to the data attributes of the data to be verified; Deconstructing the target test task to obtain multiple task rules; The data to be verified is subjected to data verification processing according to the plurality of task rules to obtain a data verification result.

2. The data verification method according to claim 1, characterized in that: The step of performing environment verification processing on a preset environment verification script according to the verification metadata to obtain environment verification information includes: Analyzing and processing the environment verification script to obtain multiple verification sub-scripts; The plurality of verification sub-scripts are executed based on the verification metadata to obtain the environment verification information.

3. The data verification method according to claim 1, characterized in that: There are multiple test tasks in the test task pool, each of which carries data type information. The step of selecting a target test task from the preset test task pool according to the data attributes of the data to be verified includes: Matching the data attributes of the to-be-verified data with the data type information of each of the test tasks in the test task pool; The test task corresponding to the data type information matching the data attribute of the data to be verified is used as the target test task.

4. The data verification method according to claim 1, characterized in that: The target test task is deconstructed to obtain multiple task rules, including: Decomposing the target test task according to preset functional division information to obtain a plurality of test functional subtasks, wherein each of the test functional subtasks carries priority information; Sorting the plurality of test function subtasks according to the priority information to obtain a task test chain; Port configuration processing is performed on the task test chain to obtain a plurality of task rules.

5. The data verification method according to claim 1, characterized in that: The step of performing data verification processing on the data to be verified according to the plurality of task rules to obtain a data verification result includes: Extracting multiple groups of target verification data from the data to be verified; Execute the plurality of task rules according to the target verification data to obtain data verification information; The multiple data verification information are fused and processed to obtain the data verification result.

6. The data verification method according to claim 5, characterized in that: The step of extracting multiple groups of target verification data from the data to be verified includes: Preprocessing the data to be verified to obtain preprocessed verification data; A plurality of groups of target verification data are obtained by screening the preprocessed verification data according to preset verification target information.

7. The data verification method according to claim 1, characterized in that: After performing data verification processing on the data to be verified according to the plurality of task rules and obtaining the data verification result, the method further includes: Converting the data verification result into verification portrait data; The verification image data is sent to a preset monitoring terminal.

8. A data verification device, characterized in that: include: A first processing module is used to obtain the data to be verified and the verification metadata; A second processing module is used to perform environment verification processing on a preset environment verification script according to the verification metadata to obtain environment verification information; A third processing module is used to select a target test task from a preset test task pool according to data attributes of the data to be verified when the environment verification information indicates that the test environment is normal; A fourth processing module, used for deconstructing the target test task to obtain a plurality of task rules; The fifth processing module is used to perform data verification processing on the data to be verified according to the multiple task rules to obtain a data verification result.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data verification method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing computer-executable instructions, characterized in that: The computer executable instructions are used to execute the data verification method described in any one of claims 1 to 7.

Images