Data dynamic verification method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and discloses a data dynamic verification method and device, computer equipment and a storage medium, the method comprises the following steps: configuring a connection pool corresponding to a multi-source database, and establishing a connection with the multi-source database based on the connection pool; setting a data verification rule, executing the data verification rule based on the specified object set and the corresponding database connection in the connection pool, and dynamically generating a structured query statement as an execution result; and outputting a verification conclusion based on the execution result and a preset expected value. According to the method, from database connection establishment, rule execution, query result comparison and conclusion output, full-process automation is achieved, connection with a multi-source database can be established, cross-database data verification is achieved, and the problems that in the prior art, multi-rule configuration is not supported, and cross-database query is difficult are solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data dynamic verification method, device, computer equipment and storage medium. Background Art

[0002] In existing technologies, verification of user behavior data usually relies on single rules or static conditions, leading to the following problems:

[0003] 1. Inefficiency: Query statements need to be written manually, and batch processing of user data is impossible.

[0004] 2. Poor scalability: Adding new rules requires modifying the code, making dynamic configuration difficult and not supporting multi-rule configuration.

[0005] 3. Difficulty in coordinating multiple data sources: Joint queries across databases cannot be performed, resulting in data silos.

[0006] 4. Complex time window calculation: Dynamic time conditions such as "nearly 90 days" need to be hard-coded, resulting in high maintenance costs.

[0007] Therefore, there is an urgent need for a user data verification method that supports multiple rule configurations and can achieve collaborative work among multiple source databases. Summary of the Invention

[0008] In view of this, the present invention provides a data dynamic verification method, apparatus, computer equipment and storage medium to solve the problems in the prior art of not supporting multi-rule configuration and difficulty in cross-database query.

[0009] In a first aspect, the present invention provides a method for dynamic data verification, the method comprising:

[0010] Configure the connection pool corresponding to the multi-source database and establish connections with the multi-source database based on the connection pool;

[0011] Set data validation rules and execute them based on the specified object collection and the corresponding database connection in the connection pool, and dynamically generate structured query statements as the execution results;

[0012] Output verification conclusions based on execution results and preset expected values.

[0013] The present invention provides a data dynamic verification method, which pre-establishes and maintains a certain number of database connections by configuring connection pools corresponding to multiple source databases, thereby avoiding the resource consumption and time overhead caused by frequently creating and destroying connections. When the system needs to access the database, it can directly obtain a connection from the connection pool, significantly shortening the data query response time. At the same time, the connection pool reuses and manages the connections, rationally allocates system resources, reduces the pressure on the database server, and improves overall resource utilization. It allows users to freely set data verification rules, has good readability and scalability, and can easily add, modify or delete verification conditions to meet the complex and changeable data verification needs in different business scenarios. According to the data verification rules and the specified object set, structured query statements are dynamically generated, which can accurately match the characteristics and query requirements of different databases and support complex multi-table association queries and logical operations. Through dynamic parameter replacement, the risk of SQL injection is effectively avoided and data security is guaranteed. From database connection establishment and rule execution to query result comparison and conclusion output, the whole process is automated, and it is possible to establish connections with multiple source databases and realize cross-database data verification, solving the problems of the prior art that does not support multiple rule configurations and is difficult to query across databases.

[0014] In an optional embodiment, the multi-source database includes a business database and a tag database;

[0015] Configure the connection pool corresponding to the multi-source database and establish connections with the multi-source database based on the connection pool, including:

[0016] Configure the connection pools and corresponding connection pool parameters corresponding to the business database and label database respectively, and establish connections with the business database and label database respectively through the connection pool parameters; the connection pool parameters include the maximum number of connections, the minimum number of idle connections, and the connection timeout.

[0017] The present invention provides a method for dynamic data verification, which configures independent connection pools for the business database and the label database respectively, and can set the connection pool parameters in a targeted manner according to the business characteristics and load conditions of the two. The business database usually handles high-frequency read and write operations, sets a larger maximum number of connections and a shorter connection timeout, and can quickly respond to business requests; while the label database is mainly used for query, and by adjusting the minimum number of idle connections, it ensures the connection availability during complex queries and avoids the overhead of frequent connection creation. The independent connection pool isolates the connection operations of the business database and the label database. When the business database has high concurrent requests, resulting in tight connection pool resources, it will not affect the normal connection and query operations of the label database. The independent connection pool and precise parameter configuration help to ensure the accuracy and consistency of data interaction between the business database and the label database.

[0018] In an optional embodiment, setting data validation rules includes:

[0019] Set data validation rules using preset data formats, which include table names, conditions, and logical operators.

[0020] The present invention provides a data dynamic verification method, which adopts a preset data format to set verification rules, and provides users with a clear rule writing framework. Users only need to fill in the table name, conditions and logical operators in accordance with the regulations, without having to have an in-depth understanding of the complex underlying syntax or data structure, which reduces the technical difficulty of rule setting. The preset data format standardizes the structure and content of the rules, ensuring that each verification rule has a unified format and specification, avoiding problems such as grammatical errors and logical confusion caused by arbitrary rule writing, and reducing the possibility of human error. By combining different table names, conditions and logical operators, a variety of verification rules can be constructed to flexibly meet the data verification needs of various complex business scenarios. Clear table name, condition and logical operator settings ensure that data verification rules can accurately filter and verify target data.

[0021] In an optional embodiment, executing data validation rules based on a specified object set and a corresponding database connection in a connection pool, and dynamically generating a structured query statement as an execution result, includes:

[0022] Get the unique identifier of each object in the specified object collection;

[0023] Determine whether the unique identifier of each object meets the preset time window;

[0024] When the unique identifier of each object meets the preset time window, the data validation rules are executed for the unique identifier of each object and the corresponding database connection in the connection pool, and a structured query statement is dynamically generated as the execution result.

[0025] The present invention provides a data dynamic verification method, which can accurately screen out objects that meet specific time conditions by obtaining a unique identifier for each object in a specified object set and making judgments based on a preset time window, thereby avoiding indiscriminate verification of the entire amount of data and significantly reducing the amount of data processing. Data verification rules are only executed and query statements are generated for objects that meet the time window, reducing invalid queries and resource usage in the database. Combined with the reuse mechanism of the connection pool, database connection resources are reasonably allocated when executing verification tasks to avoid database overload due to a large number of concurrent requests. From object identifier acquisition, time window judgment to rule execution and query generation, the entire process can be automated. At the same time, by presetting time windows and verification rules, the accuracy of data processing is guaranteed, and the links of manual screening and intervention are reduced.

[0026] In an optional embodiment, the data validation rules include single-table query rules and multi-table joint query rules;

[0027] Execute data validation rules for each object's unique identifier and the corresponding database connection in the connection pool, and dynamically generate structured query statements as execution results, including:

[0028] When the data validation rule is a single-table query rule, the query is performed based on the database connection corresponding to the table name, condition, and logical operator in the connection pool, and a structured query statement is directly generated;

[0029] When the data validation rule is a multi-table joint query rule, use the operator tool to query the database connection corresponding to each table in the connection pool to obtain the multi-table query results, merge the query results of multiple tables, and add connection clauses to connect the tables according to the association relationship between the conditions in different tables to dynamically generate structured query statements.

[0030] The present invention provides a data dynamic verification method, which generates query statements directly based on table names, conditions and logical operators for single-table query rules, without the need for complex connection operations, reducing query processing steps and resource consumption, and can quickly retrieve qualified data from a single data table. During the execution of multi-table joint query rules, by adding connection clauses to clarify the relationship between tables, it is ensured that data obtained from different data tables can be accurately integrated, and multi-dimensional data related to verification can be fully obtained, avoiding data omissions or incorrect associations, and ensuring the integrity and accuracy of data verification. The combination of single-table query rules and multi-table joint query rules enables the system to meet diverse business needs. The unified rule execution and query statement generation framework reduces the coding complexity of developers.

[0031] In an optional embodiment, the preset data format further includes an expected result type, and the method further includes:

[0032] When dynamically generating a structured query statement, the conditional value in the data validation rule is dynamically replaced in the structured query statement, and the corresponding aggregate function is added to the structured query statement based on the expected result type.

[0033] The present invention provides a method for dynamic data verification, which clarifies the goal of data verification through a preset expected result type, and can make the query results closely fit the verification requirements. Combined with the dynamic replacement of conditional values, it ensures that the query statement is executed for specific business conditions, avoiding verification deviations caused by hard-coded conditions. Associating the expected result type with the aggregate function makes the system more flexible when dealing with different business scenarios. Developers do not need to manually write complex aggregate function logic and conditional splicing code. They only need to configure the expected result type and conditional value according to the preset format, and the system can automatically generate accurate structured query statements, which not only reduces the difficulty of development, reduces syntax errors and logical loopholes introduced by manually written code, but also saves a lot of development time. Dynamic replacement of conditional values ​​and adding aggregate functions based on expected result types ensure the consistency and standardization of the data query and verification process.

[0034] In an optional embodiment, outputting a verification conclusion based on the execution result and a preset expected value includes:

[0035] When the execution result is consistent with the preset expected value, the output is the conclusion of successful verification; otherwise, the output is the conclusion of failed verification.

[0036] This invention provides a dynamic data verification method that directly compares execution results with preset expectations, outputting "verification successful" or "verification failed" as the conclusion. This method presents simple, clear, and transparent results. Whether a technician is reviewing system logs or a business person is obtaining verification feedback, they can quickly understand the data verification status without complex interpretation, significantly reducing the cost of receiving and understanding information. A clear verification conclusion output mechanism ensures the reliability and consistency of data verification results.

[0037] In a second aspect, the present invention provides a data dynamic verification device, the device comprising:

[0038] The connection pool configuration and database connection establishment module is used to configure the connection pool corresponding to the multi-source database and establish connections with the multi-source database based on the connection pool;

[0039] The data validation rule execution module is used to set data validation rules and execute them based on the specified object set and the corresponding database connection in the connection pool, and dynamically generate structured query statements as execution results;

[0040] The verification conclusion output module is used to output the verification conclusion based on the execution results and preset expected values.

[0041] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, computer instructions being stored in the memory, and the processor executing the data dynamic verification method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0042] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the data dynamic verification method of the first aspect or any corresponding embodiment thereof.

[0043] In a fifth aspect, the present invention provides a computer program product comprising computer instructions for causing a computer to execute the data dynamic verification method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 is a flowchart of a method for dynamic data verification according to an embodiment of the present invention;

[0046] Figure 2 is a flow chart of another method for dynamic data verification according to an embodiment of the present invention;

[0047] Figure 3 is a flowchart of another method for dynamic data verification according to an embodiment of the present invention;

[0048] Figure 4 is a structural block diagram of a data dynamic verification device according to an embodiment of the present invention;

[0049] Figure 5 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0051] Dynamic data validation refers to the process of dynamically verifying the correctness and validity of current data during data entry or processing based on its real-time status or other relevant data. This verification method ensures data accuracy and consistency and is applicable to various scenarios, such as user input and external data sources. Data validation rules and multi-source databases are used in this process. In related technologies, adding new rules requires code modifications, making dynamic configuration difficult and not supporting multi-rule configuration. Furthermore, cross-database query operations are difficult when querying multi-source databases.

[0052] The embodiment of the present invention provides a data dynamic verification method, which achieves the effect of supporting multi-rule verification and cross-database query by connecting multiple source databases and setting dynamic data verification rules.

[0053] According to an embodiment of the present invention, an embodiment of a method for dynamic data verification is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0054] In this embodiment, a data dynamic verification method is provided, which can be used in the above-mentioned computer device. Figure 1 Flowchart of a data dynamic verification method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0055] Step S101 : configuring a connection pool corresponding to a multi-source database, and establishing a connection with the multi-source database based on the connection pool.

[0056] Specifically, data connection pooling is a technology used to manage and optimize database connections. It creates a certain number of database connections when the program starts, and forms a connection pool with these connections for the program to dynamically apply for, use, and release.

[0057] Connection pool initialization: For multiple source database types, create independent connection pools for each database. You can use a database connection pool framework (such as HikariCP, C3P0) to initialize the connection pool according to pre-configured connection pool parameters during the data verification system startup phase.

[0058] Connection Establishment and Management: After the connection pool is initialized, a physical connection is established with the database through a driver (such as the MySQL JDBC driver or the StarRocks JDBC driver). The connection pool maintains connection status. When a database operation request arrives, it prioritizes obtaining an idle connection from the connection pool. If the connection pool has no idle connections and the maximum number of connections has not been reached, a new connection is created. If the connection pool is full, the request enters a waiting queue until a connection is released or times out. The connection pool also regularly performs health checks on connections, removing failed connections and adding new ones to ensure connection availability.

[0059] Exception handling mechanism: During the connection establishment and acquisition process, if a network failure or database service unavailability occurs, the connection pool will record the exception information and attempt to reconnect. If the connection fails multiple times, a database connection exception is thrown, notifying the upper-layer application to perform downgrade processing, such as using cached data or suspending related business operations.

[0060] Step S102: setting data validation rules, executing the data validation rules based on the specified object set and the corresponding database connection in the connection pool, and dynamically generating a structured query statement as an execution result.

[0061] Specifically, users define data validation rules in JSON format through the management interface or API. The rules are stored in the configuration database or distributed configuration center (such as Apollo, Nacos) to facilitate unified management and dynamic updates.

[0062] A specified object collection (such as a list of user UnionIDs) can be uploaded to the system through file upload, API (Application Programming Interface) access, or database connection filtering. Based on the table name, conditions, and logical operators in the rule, a SQL (Structured Query Language) template engine (such as MyBatis's dynamic SQL) is used to dynamically generate query statements. Condition values ​​are parameterized to prevent SQL injection attacks. The corresponding database connection is obtained from the connection pool, and the query statement is executed to obtain the result set.

[0063] Step S103: output a verification conclusion based on the execution result and the preset expected value.

[0064] Specifically, a verification conclusion is output based on the consistency comparison result between the execution result and the preset expected value.

[0065] The data dynamic verification method provided in this embodiment configures the connection pool corresponding to the multi-source database, pre-establishes and maintains a certain number of database connections, and avoids the resource consumption and time overhead caused by the frequent creation and destruction of connections. When the system needs to access the database, it can directly obtain the connection from the connection pool, significantly shortening the data query response time. At the same time, the connection pool reuses and manages the connection, rationally allocates system resources, reduces the pressure on the database server, and improves the overall resource utilization. It allows users to freely set data validation rules, has good readability and scalability, and can easily add, modify or delete validation conditions to meet the complex and changeable data validation needs in different business scenarios. According to the data validation rules and the specified object set, structured query statements are dynamically generated, which can accurately match the characteristics and query requirements of different databases and support complex multi-table association queries and logical operations. Through dynamic parameter replacement, the risk of SQL injection is effectively avoided and data security is guaranteed. From database connection establishment, rule execution to query result comparison and conclusion output, the whole process is automated, and it can establish connections with multi-source databases to realize cross-database data validation, solving the problems of the existing technology that does not support multi-rule configuration and is difficult to query across databases.

[0066] In this embodiment, a data dynamic verification method is provided, which can be used in the above-mentioned computer device. Figure 2 Flowchart of a data dynamic verification method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0067] Step S201 : configuring a connection pool corresponding to a multi-source database, and establishing a connection with the multi-source database based on the connection pool.

[0068] Specifically, the multi-source database includes a business database and a tag database. The business database is connected to read behavioral data such as orders, live broadcasts, and store check-ins; the tag database is connected to obtain basic attributes such as user registration information.

[0069] The above step S201 includes:

[0070] Step a: configure the connection pools and corresponding connection pool parameters corresponding to the business database and label database respectively, and establish connections with the business database and label database respectively through the connection pool parameters; the connection pool parameters include the maximum number of connections, the minimum number of idle connections and the connection timeout.

[0071] The business database connection pool configuration and the tag database connection pool configuration are described separately, where:

[0072] 1. Business database connection pool configuration and establishment:

[0073] Parameter configuration: In the system configuration file (e.g., YAML format), create a separate connection pool configuration section for the business database. Set reasonable connection pool parameters based on the read and write characteristics of the business database. For example, considering that the business database handles high-frequency operations such as order processing, live data recording, and store check-ins, set maximumPoolSize (maximum number of connections) to 50 to cope with high concurrency requests; set minimumIdle (minimum number of idle connections) to 10 to ensure sufficient connections are available even during low-peak business periods; and set connectionTimeout (connection timeout) to 30,000 milliseconds to avoid blocking business processes due to long waits for connections.

[0074] Connection pool initialization: When the system starts, a database connection pool framework (such as HikariCP) reads the above configuration and creates a business database connection pool instance. HikariCP pre-establishes a minimum number of idle database connections based on the configuration parameters. These connections remain idle, ready to respond to business requests. Simultaneously, the connection pool starts a background thread that regularly checks the health of the connections to ensure connection availability.

[0075] Connection Establishment and Acquisition: When a business system needs to access a business database, it obtains a connection from the business database connection pool. If there are idle connections in the connection pool, they are directly acquired and used. If there are insufficient idle connections and the current number of connections has not reached the maximum number of connections, the connection pool will create a new database connection. For example, when processing a large number of order data queries, the connection pool will quickly allocate connection resources to ensure timely execution of query operations. If the connection pool is full, new connection requests will be queued until a connection is released or the connection times out.

[0076] 2. Tag database connection pool configuration and establishment:

[0077] Parameter Configuration: Given that the tag database primarily queries basic attributes such as user registration information, separate connection pool parameters are configured. Set maximumPoolSize to 30, as the tag database has relatively few concurrent requests; set minimumIdle to 5 to balance resource usage and connection response speed; and set connectionTimeout to 20,000 milliseconds to accommodate the response requirements of query operations.

[0078] Connection pool initialization and management: Also during system startup, the tag database connection pool is initialized using a database connection pool framework (such as C3P0). After initialization, the connection pool maintains connection status, regularly checks connection validity, removes invalid connections, and adds new ones. For example, if the tag database undergoes structural adjustments or service restarts, the connection pool can automatically detect and reestablish invalid connections, ensuring that subsequent queries are not affected.

[0079] Connection Acquisition and Usage: When querying the tag database to execute user data validation rules, a connection is obtained from the tag database connection pool. The connection pool allocates connection resources based on the priority and status of the connection request. For example, when verifying user registration information, an available connection is quickly provided to ensure smooth execution of the validation rules. After the query is completed, the connection is promptly returned to the connection pool for reuse by other requests.

[0080] For example, the database connection configuration can also use the mysql.connector library to connect to MySQL databases and StarRocks (extremely fast, full-scenario MPP (Massively Parallel Processing)) databases, isolating sensitive information through configuration files. The mysql.connector library is an official library for connecting to MySQL databases, allowing Python programs to interact with MySQL databases. This library provides functions such as database connection, cursor management, SQL execution, result set processing, parameterized queries, and transaction processing, simplifying the process of managing MySQL databases in Python. The sample code for database connection is as follows:

[0081]

[0082]

[0083] Step S202 : setting data validation rules, executing the data validation rules based on the specified object set and the corresponding database connection in the connection pool, and dynamically generating a structured query statement as an execution result.

[0084] Specifically, the above step S202 includes:

[0085] Step S2021: Setting data validation rules in a preset data format, where the preset data format includes a table name, a condition, and a logical operator.

[0086] Specifically, the preset data format in this embodiment is JSON, meaning that data validation rules are stored and transmitted in JSON. Table names are represented by "table," and conditions refer to data validation conditions, such as user types with registration dates after 2024 or users with registration dates 181 days or more. Logical operators include AND, OR, and combinations of AND and OR.

[0087] The system provides users with a visual rule creation interface, allowing them to set rules through graphical interaction. The interface includes a table name selection drop-down box, which automatically retrieves all table names from database metadata and supports fuzzy search to quickly locate the target table. The condition setting area adopts a modular design. Clicking the "Add Condition" button will bring up a condition editing window, where users can select a field name (a list of fields is automatically generated based on the table structure), an operator (such as >, <, =, IN, etc.), and enter the corresponding value. Logical operators are available for selection via radio buttons or drop-down menus, allowing users to combine multiple conditions using AND or OR.

[0088] The sample code for data validation rules is as follows:

[0089]

[0090] Field Description: table: target data table name; conditions: condition array, each element includes field name, operator and value; operator: logical operator between conditions; expected_result: expected result. Data validation rules support multi-conditional logical combination, multi-table joint query, and multiple expected result type definitions. The name field in the above fields is used to describe the name or title of the rule, which is a brief description of the rule and is usually used to display in the user interface or log. In this example, the value of name is "No punch-in record in the past 90 days", which means that this rule checks whether there are no punch-in records in the past 90 days.

[0091] The message field is used to describe error messages or prompts when a rule is violated. When a data validation rule's conditions are not met, this message notifies the user of the problem. In this example, the message value is "There are clock-in records in the last 90 days," indicating that the data validation rule is violated if there are clock-in records in the last 90 days.

[0092] It's important to note that before saving a rule, the system validates the JSON format, checking field integrity, data type correctness (e.g., whether the operator is AND or OR), and compatibility between the conditional operator and the value (e.g., whether a date field uses a date format value). If validation fails, an error message will be displayed on the interface, guiding the user to correct the error.

[0093] Step S2022: Obtain a unique identifier for each object in the specified object set.

[0094] Specifically, multiple methods are supported for obtaining a specified object collection (e.g., a list of user UnionIDs). For file import, the system supports common formats such as CSV and Excel. File parsing libraries (e.g., Apache POI for Excel files and OpenCSV for CSV files) are used to read the file contents and parse the unique identifiers line by line. For API access, HTTP client libraries (e.g., OkHttp or Apache HttpClient) are used to send requests, receive and parse the response data in JSON or XML format, and extract the unique identifiers.

[0095] Step S2023: Determine whether the unique identifier of each object conforms to a preset time window.

[0096] Specifically, the preset time window is stored in a configuration file or database in JSON format. The preset time window is set based on actual circumstances and is not specifically limited here. For example, the preset time window can be dynamically calculated based on the registration date, such as triggering a check after 181 days of registration. The time window calculation is based on a dynamic offset calculation based on the registration date and a rolling time window generation mechanism.

[0097] Based on the unique identifier, query the corresponding registration time from the business database or tag database (assuming it is stored in the register_date field). Use a date and time processing library (such as the java.time package in Java 8) to perform time calculations and determine whether it fits within the time window.

[0098] Step S2024 , when the unique identifier of each object meets the preset time window, the data validation rule is executed for the unique identifier of each object and the corresponding database connection in the connection pool, and a structured query statement is dynamically generated as the execution result.

[0099] For unique identifiers that match the time window, the connection is obtained from the business database or the tag database connection pool based on the table name in the data validation rule. For example, if the data validation rule targets the user_info table (assuming it is in the tag database), the connection is obtained from the tag database connection pool.

[0100] In some optional implementations, the data validation rules include single-table query rules and multi-table joint query rules;

[0101] The above step S2024 includes:

[0102] Step b1: When the data validation rule is a single-table query rule, query the database connection corresponding to the table name, condition, and logical operator in the connection pool to directly generate a structured query statement.

[0103] Specifically, if the data validation rule is a single-table query rule, use a SQL template engine (such as MyBatis's dynamic SQL or Apache Commons Text's StringSubstitutor) to dynamically generate SQL query statements based on the table name, conditions, and logical operators in the data validation rule. The following is a code example for executing data validation rules on the user_info table:

[0104]

[0105] The generated SQL statement is: SELECT * FROM user_info WHERE user_type = ? .

[0106] Step b2, when the data validation rule is a multi-table joint query rule, use the operator tool to query the database connection corresponding to each table in the connection pool to obtain the multi-table query results, and merge the query results of the multiple tables, and according to the relationship between the conditions in different tables, add a connection clause to connect the tables to dynamically generate a structured query statement. Specifically, the connection clause in this embodiment uses a JOIN clause. If it is a multi-table joint query rule, first use an SQL parser (such as ANTLR) to independently generate the query statement for each table, and then use operations such as UNION ALL to merge the query results. According to the relationship between the tables, add a JOIN clause to connect. For example, the code example of the data validation rule involving the joint query of the user_info table and the order_info table is as follows:

[0107]

[0108] The generated SQL statement is:

[0109]

[0110] Step S203: Output the verification conclusion based on the execution result and the preset expected value. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0111] The dynamic data validation method provided in this embodiment configures independent connection pools for the business database and the tag database, respectively. Based on the business characteristics and load of each database, independent connection pools and precise parameter configuration help ensure the accuracy and consistency of data exchange between the business and tag databases. Using a preset data format to set validation rules provides users with a clear framework for rule writing. Users only need to fill in the table name, conditions, and logical operators according to the regulations, without having to deeply understand the complex underlying syntax or data structure, which reduces the technical difficulty of rule setting. The preset data format standardizes the structure and content of the rules, ensuring that each validation rule has a unified format and specifications, avoiding syntax errors, logical confusion, and other problems caused by haphazard rule writing, and reducing the possibility of human error. By combining different table names, conditions, and logical operators, a variety of validation rules can be constructed to flexibly meet the data validation needs of various complex business scenarios. Clear table name, condition, and logical operator settings ensure that data validation rules can accurately filter and verify target data. By obtaining the unique identifier of each object in a specified object set and judging based on a preset time window, objects that meet specific time conditions can be accurately filtered, avoiding indiscriminate validation of all data and significantly reducing data processing volume. Data validation rules and query statements are only executed for objects that meet the time window, reducing invalid database queries and resource usage. Combined with the connection pool's reuse mechanism, database connection resources are rationally allocated when executing validation tasks, avoiding database overload caused by a large number of concurrent requests. The entire process, from object identifier acquisition and time window determination to rule execution and query generation, is automated. Furthermore, pre-set time windows and validation rules ensure accurate data processing and reduce the need for manual screening and intervention.

[0112] In this embodiment, a data dynamic verification method is provided, which can be used in the above-mentioned computer device. Figure 3 Flowchart of a data dynamic verification method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0113] Step S301: Configure the connection pool corresponding to the multi-source database and establish a connection with the multi-source database based on the connection pool. Figure 2 Step S201 of the illustrated embodiment will not be described in detail here.

[0114] Step S302: Set data validation rules and execute them based on the specified object set and the corresponding database connection in the connection pool, and dynamically generate structured query statements as the execution results. Figure 2 Step S202 of the illustrated embodiment will not be described in detail here.

[0115] The preset data format in the data validation rule also includes the expected result type.

[0116] Step S303 , when dynamically generating a structured query statement, dynamically replace the condition value in the data validation rule into the structured query statement, and add a corresponding aggregation function into the structured query statement based on the expected result type.

[0117] Specifically, when generating SQL query statements, parameterized queries are used to dynamically replace conditional values. Taking Java's JDBC as an example, this is achieved through the PreparedStatement object (an interface for executing precompiled SQL statements). The conditional values ​​in the SQL statement are replaced with placeholders (such as "?"). Before executing the query, the actual conditional values ​​are set to the corresponding placeholders, avoiding direct splicing of user input into the SQL statement and effectively preventing SQL injection attacks. The specific implementation steps include:

[0118] Parse rules to obtain conditional values: Parse the conditional values ​​from the JSON format of the data validation rules.

[0119] Generate SQL with placeholders: Generates SQL query statements containing placeholders based on the rules.

[0120] Set the condition value: Create a PreparedStatement object and set the parsed condition value to the placeholder. The same process applies to rules containing multiple conditions or complex logical operators.

[0121] Expected result type analysis: Data validation rules define expected result types. Common types include count, sum, average, and existence (determining whether the result set is empty). Before generating a query statement, the system analyzes the expected result type from the rules.

[0122] Aggregate function addition logic: Add the corresponding aggregate function to the SQL query statement based on the expected result type. For example:

[0123] Count value (count): Add COUNT(*) or COUNT(specified field) to the query statement.

[0124] Sum value (sum): Add the SUM function.

[0125] Average (avg): Add AVG function.

[0126] Existence: Use the EXISTS keyword to construct queries.

[0127] When adding the aggregation function, the condition value is dynamically replaced.

[0128] Step S304: output a verification conclusion based on the execution result and the preset expected value.

[0129] Specifically, the above step S304 includes:

[0130] Step c: When the execution result is consistent with the preset expected value, the conclusion of successful verification is output; otherwise, the conclusion of failed verification is output.

[0131] Specifically, when the expected result type in the execution result is a single value (such as a count value, a sum value, etc.), the execution result is directly compared with the preset expected value to see if they are equal. For example, if the number of users is queried and the execution result is 50 and the preset expected value is 50, then the verification is determined to be successful.

[0132] In some scenarios, the preset expected value may be a range (such as the order amount being within a certain interval). In this case, you need to write corresponding range judgment logic. For example, if the preset expected value is that the order amount is between 100 and 1000, and the execution result is 500, you can use if (result>=100&&result<=1000) to determine whether it meets the requirements.

[0133] If the execution result is a complex data structure (such as a result set containing multiple records), and the preset expected value is a specific requirement for the data structure (such as a combination of values ​​in a specific field), it is necessary to traverse the result set and compare it according to the rules. For example, if the execution result is a list of user information, and the preset expected value requires that all users' registration dates must be after 2024, the list will be traversed to check whether each user's registration date field meets the condition. Based on the comparison results, the corresponding verification conclusion is generated. If the execution result is consistent with the preset expected value, the conclusion "Verification Successful" is generated; otherwise, the conclusion "Verification Failed" is generated. To facilitate troubleshooting, the specific execution result and preset expected value information can be attached to the conclusion. For example, "Verification Failed: Execution Result is 50, Preset Expected Value is 100."

[0134] The dynamic data verification method provided in this embodiment directly compares execution results with preset expectations, outputting "verification successful" or "verification failed" as the conclusion. This provides a simple, clear, and transparent presentation of the results. Whether a technician is reviewing system logs or a business person is obtaining verification feedback, they can quickly understand the data verification status without complex interpretation, significantly reducing the cost of receiving and understanding information. A clear verification conclusion output mechanism ensures the reliability and consistency of data verification results.

[0135] As one or more specific application examples of the embodiments of the present invention, the data dynamic verification method provided by the present invention is further described in detail. The specific process is as follows:

[0136] 1. Database connection configuration, using dual connection pool configuration:

[0137] For business database connections: read order, live broadcast, store check-in and other behavioral data.

[0138] For tag database connection: obtain basic attributes such as user registration information.

[0139] 2. Data validation rule definition and analysis:

[0140] Data validation rules are defined in JSON format and include table name, condition, logical operator, and expected result.

[0141] 3. Dynamic SQL statement generation:

[0142] Generate SQL statements based on table names, conditions, and logical operators in data validation rules, supporting multi-table joint queries.

[0143] 4. Automated verification process:

[0144] Specify the user and store the user's related information in a txt text, traverse the user UnionID list (unique identifier list), and perform rule verification in sequence.

[0145] The time window is dynamically calculated based on the registration date (e.g., a check is triggered after 181 days of registration).

[0146] The verification process is as follows:

[0147] 1) Read the UnionID list.

[0148] 2) Verify registration duration (181-day threshold).

[0149] 3) Valid users enter the data validation rules for verification.

[0150] 4) Multiple verification rules are executed in parallel to check the rules.

[0151] 5) Execution results are summarized and output.

[0152] The data dynamic verification method provided in this embodiment realizes a dynamic, intelligent, and efficient user behavior verification system by configuring a rule engine, solving the problems of low data verification efficiency and poor scalability in the existing technology, and can be widely used in scenarios such as user grouping and risk control.

[0153] In this embodiment, a data dynamic verification device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments. The details that have been described will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation by hardware, or a combination of software and hardware, is also possible and contemplated.

[0154] This embodiment provides a data dynamic verification device, such as Figure 4 Shown, including:

[0155] The connection pool configuration and database connection establishment module 401 is used to configure connection pools corresponding to multiple source databases and establish connections with the multiple source databases based on the connection pools.

[0156] The data validation rule execution module 402 is used to set data validation rules, execute the data validation rules based on the specified object set and the corresponding database connection in the connection pool, and dynamically generate a structured query statement as an execution result.

[0157] The verification conclusion output module 403 is used to output the verification conclusion based on the execution result and the preset expected value.

[0158] In some optional implementations, the multi-source database includes a business database and a tag database; the connection pool configuration and database connection establishment module 401 includes:

[0159] The connection pool parameter configuration and database connection establishment unit is used to configure the connection pools and corresponding connection pool parameters corresponding to the business database and label database respectively, and establish connections with the business database and label database respectively through the connection pool parameters; the connection pool parameters include the maximum number of connections, the minimum number of idle connections and the connection timeout.

[0160] In some optional implementations, the data validation rule execution module 402 includes:

[0161] The data validation rule setting unit is used to set data validation rules in a preset data format, where the preset data format includes a table name, a condition, and a logical operator.

[0162] The unique identifier obtaining unit is used to obtain the unique identifier of each object in the specified object set.

[0163] The judging unit is configured to judge whether the unique identifier of each object conforms to a preset time window.

[0164] The data validation rule execution unit is used to execute the data validation rule for the unique identifier of each object and the corresponding database connection in the connection pool when the unique identifier of each object meets the preset time window, and dynamically generate a structured query statement as the execution result.

[0165] In some optional implementations, the data validation rules include single-table query rules and multi-table joint query rules; and the data validation rule execution unit includes:

[0166] The single-table rule execution subunit is used to query the database connection corresponding to the table name, condition, and logical operator in the connection pool when the data validation rule is a single-table query rule, and directly generate a structured query statement.

[0167] The multi-table rule execution sub-unit is used to use the operator tool to query the database connection corresponding to each table in the connection pool when the data validation rule is a multi-table joint query rule, obtain the multi-table query results, merge the query results of multiple tables, and add connection clauses to connect tables according to the association relationship between conditions in different tables to dynamically generate structured query statements.

[0168] In some optional implementations, the preset data format further includes an expected result type, and the data dynamic verification device further includes:

[0169] The conditional value replacement and aggregate function addition module is used to dynamically replace the conditional values ​​in the data validation rules into the structured query statements when dynamically generating structured query statements, and to add corresponding aggregate functions to the structured query statements based on the expected result type.

[0170] In some optional implementations, the verification conclusion output module 403 includes:

[0171] The verification conclusion output unit is used to output the conclusion of successful verification when the execution result is consistent with the preset expected value; otherwise, it outputs the conclusion of failed verification.

[0172] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0173] The data dynamic verification device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0174] The embodiment of the present invention also provides a computer device having the above Figure 4 The data dynamic verification device shown.

[0175] See also Figure 5 , Figure 5 is a structural diagram of a computer device provided by an optional embodiment of the present invention, such as Figure 5 As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0176] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0177] The memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0178] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0179] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0180] The computer device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 5 The bus connection is taken as an example.

[0181] The input device 30 can receive input digital or character information and generate key signal input related to user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touch pad, an indicator stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 can include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display, and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0182] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0183] A portion of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the form in which the computer program instruction exists in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc. Accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium that can be accessed by the computer.

[0184] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A data dynamic verification method, characterized in that: The method comprises: Configuring connection pools corresponding to multiple source databases, and establishing connections with the multiple source databases based on the connection pools; Set data validation rules and execute them based on the specified object collection and the corresponding database connection in the connection pool, and dynamically generate structured query statements as the execution results; Output a verification conclusion based on the execution result and the preset expected value.

2. The method according to claim 1, characterized in that The multi-source database includes a business database and a label database; The configuring a connection pool corresponding to the multi-source database and establishing a connection with the multi-source database based on the connection pool includes: The connection pools and corresponding connection pool parameters corresponding to the business database and the tag database are configured respectively, and connections with the business database and the tag database are established respectively through the connection pool parameters; the connection pool parameters include the maximum number of connections, the minimum number of idle connections and the connection timeout.

3. The method according to claim 1, characterized in that The setting of data validation rules includes: Data validation rules are set in a preset data format, where the preset data format includes a table name, a condition, and a logical operator.

4. The method according to claim 3, characterized in that Executes data validation rules based on the specified object collection and the corresponding database connection in the connection pool, and dynamically generates structured query statements as execution results, including: Get the unique identifier of each object in the specified object collection; Determine whether the unique identifier of each object meets the preset time window; When the unique identifier of each object meets the preset time window, the data validation rules are executed for the unique identifier of each object and the corresponding database connection in the connection pool, and a structured query statement is dynamically generated as the execution result.

5. The method according to claim 4, characterized in that The data validation rules include single table query rules and multi-table joint query rules; Execute data validation rules for each object's unique identifier and the corresponding database connection in the connection pool, and dynamically generate structured query statements as execution results, including: When the data validation rule is a single-table query rule, the query is performed based on the database connection corresponding to the table name, condition, and logical operator in the connection pool, and a structured query statement is directly generated; When the data validation rule is a multi-table joint query rule, use the operator tool to query the database connection corresponding to each table in the connection pool to obtain the multi-table query results, merge the query results of multiple tables, and add connection clauses to connect the tables according to the association relationship between the conditions in different tables to dynamically generate structured query statements.

6. The method according to claim 3, characterized in that The preset data format also includes an expected result type, and the method further includes: When dynamically generating a structured query statement, the conditional value in the data validation rule is dynamically replaced in the structured query statement, and the corresponding aggregate function is added to the structured query statement based on the expected result type.

7. The method according to claim 1, characterized in that Outputting a verification conclusion based on the execution result and a preset expected value includes: When the execution result is consistent with the preset expected value, a conclusion of successful verification is output; otherwise, a conclusion of failed verification is output.

8. A data dynamic verification device, characterized in that: The device comprises: The connection pool configuration and database connection establishment module is used to configure the connection pool corresponding to the multi-source database and establish connections with the multi-source database based on the connection pool; The data validation rule execution module is used to set data validation rules and execute them based on the specified object set and the corresponding database connection in the connection pool, and dynamically generate structured query statements as execution results; The verification conclusion output module is used to output the verification conclusion based on the execution result and the preset expected value.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the data dynamic verification method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data dynamic verification method according to any one of claims 1 to 7.

Images