A data processing method, apparatus, device, and medium

Abstract

The application discloses a data processing method, device and equipment and medium. The application relates to the technical field of big data. The method comprises the following steps: when obtaining to-be-processed data from a source system, determining a target docking type corresponding to the to-be-processed data; determining a target running environment corresponding to the target docking type; based on a pre-checking rule, performing preliminary checking on the to-be-processed data in the target running environment; under the condition that the checking result is consistent with a preset checking result, storing the to-be-processed data into a temporary data table; performing post-checking processing according to a post-checking rule corresponding to the to-be-processed data to obtain a target checking result; and when the target checking result is consistent with the preset checking result, storing the to-be-processed data into a target cache position and removing the to-be-processed data from the temporary data table. The technical scheme of the application can reduce the data processing cost, support data processing of multiple docking types, and improve the universality of data docking.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of big data technology, and in particular to a data processing method, apparatus, device and medium. Background Technology

[0002] With the development of big data technology, the forms of data connection between systems are also becoming more diverse, such as message connection, data connection between databases, data file connection, and so on.

[0003] Currently, the data exchange method between systems typically involves deploying a separate processing device for each type of data exchange, such as message processing or file processing. This results in high data processing costs and poor versatility. Summary of the Invention

[0004] This invention provides a data processing method, apparatus, device, and medium to reduce data processing costs while supporting data processing for various interface types and improving the versatility of data interface.

[0005] In a first aspect, embodiments of the present invention provide a data processing method, the method comprising:

[0006] When obtaining data to be processed from the source system, determine the target docking type corresponding to the data to be processed;

[0007] Determine the target runtime environment corresponding to the target docking type, and load the target runtime environment;

[0008] In the target operating environment, the data to be processed is initially verified based on the pre-verification rules, and the data to be processed is stored in a temporary data table if the verification result is consistent with the preset verification result.

[0009] When the conditions for post-data verification are met, the data to be processed is retrieved from the temporary data table, and post-data verification is performed according to the post-data verification rules corresponding to the data to be processed to obtain the target verification result.

[0010] When the target verification result matches the preset verification result, the data to be processed is stored in the target cache location, and the data to be processed is removed from the temporary data table.

[0011] Secondly, embodiments of the present invention also provide a data processing apparatus, the apparatus comprising:

[0012] The target docking type determination module is used to determine the target docking type corresponding to the data to be processed when the data to be processed is obtained from the source system.

[0013] The target runtime environment determination module is used to determine the target runtime environment corresponding to the target docking type and load the target runtime environment;

[0014] The pre-verification module is used to perform preliminary verification on the data to be processed based on pre-verification rules under the target operating environment, and store the data to be processed in a temporary data table if the verification result is consistent with the preset verification result.

[0015] The post-verification module is used to retrieve the data to be processed from the temporary data table when the conditions for post-verification of data are met, and to perform post-verification processing according to the post-verification rules corresponding to the data to be processed, so as to obtain the target verification result.

[0016] The data processing module is used to store the data to be processed in the target cache location and remove the data to be processed from the temporary data table when the target verification result is consistent with the preset verification result.

[0017] Thirdly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data processing method as described in any of the embodiments of the present invention.

[0018] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described in any of the embodiments of the present invention.

[0019] Fifthly, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the data processing method as described in any of the embodiments of the present invention.

[0020] In this embodiment of the invention, when obtaining data to be processed from the source system, the target interface type corresponding to the data to be processed is determined; the target operating environment corresponding to the target interface type is determined and loaded; in the target operating environment, the data to be processed is initially verified based on the pre-verification rules, and if the verification result is consistent with the preset verification result, the data to be processed is stored in a temporary data table; when the conditions for post-verification are met, the data to be processed is retrieved from the temporary data table, and post-verification processing is performed according to the post-verification rules corresponding to the data to be processed to obtain the target verification result; when the target verification result is consistent with the preset verification result, the data to be processed is stored in the target cache location, and the data to be processed is removed from the temporary data table. This solves the problem in the prior art where a separate processing device is deployed for each data interface type, resulting in high data processing costs and poor versatility. It realizes that by automatically adapting the target operating environment corresponding to the target interface type of the data to be processed obtained from the source system, the data to be processed is distributed to the target operating environment, and a single system can support data processing for multiple interface types, improving the versatility of data processing while reducing data processing costs. Furthermore, in the target operating environment, pre-verification rules are used to perform preliminary verification of the data to be processed, ensuring the integrity and accuracy of the data before processing. After the pre-verification passes, the data to be processed is temporarily stored. Then, post-verification is performed on the data to be processed according to the post-verification rules corresponding to the data to be processed, ensuring the integrity and accuracy of the data after processing. After the post-verification passes, the data to be processed is stored in the target cache location, ensuring data quality and removing the data to be processed from the temporary data table, reducing system resource pressure. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a data processing method provided according to an embodiment of the present invention;

[0023] Figure 2 This is a flowchart of a data processing method provided according to an embodiment of the present invention;

[0024] Figure 3 This is a flowchart of a data processing method provided according to an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of a data processing method provided according to an embodiment of the present invention;

[0026] Figure 5 This is a schematic diagram of a data processing method provided according to an embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of the structure of a data processing device according to an embodiment of the present invention;

[0028] Figure 7 This is a schematic diagram of the structure of an electronic device that implements the data processing method of the present invention. Detailed Implementation

[0029] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0030] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance. The acquisition, storage, use, and processing of data in the technical solutions of this application all comply with the relevant provisions of national laws and regulations.

[0031] Figure 1 This is a flowchart of a data processing method according to an embodiment of the present invention. This embodiment is applicable to situations involving data interoperation. The method can be executed by a data processing device, which can be implemented in hardware and / or software. Figure 1 As shown, the method includes:

[0032] S110. When obtaining data to be processed from the source system, determine the target docking type corresponding to the data to be processed.

[0033] The data to be processed can be files, messages, images, audio, video, etc., or data from a relational database. The source system refers to the system that provides the data. For example, a source system can be a database or a data warehouse. Data from these source systems can be extracted or loaded into the target system as data to be processed. For example, transferring a file from the source system to the target system makes that file the data to be processed; or sending a message from the source system to the target system makes that message the data to be processed. The interface type refers to the type of data communication between different systems or applications. For example, interface types include, but are not limited to, database interface, file interface, and message interface. Database interface can be implemented in various ways, such as ODBC (Open Database Connectivity), JDBC (Java Database Connectivity), and ETL (Extract Transform Load). File interface can be implemented in various ways, such as FTP (File Transfer Protocol) transfer, file sharing, and cloud storage. Message interface can also be implemented in various ways, such as using message queues, publish / subscribe patterns, and API (Application Programming Interface) calls.

[0034] In practical applications, data to be processed can be obtained from the source system via an interface or network. Furthermore, the target interface type for the data to be processed can be determined based on its data attributes. Optional data attributes include, but are not limited to, data type, data structure, data volume, access frequency, and security. For example, the target interface type for structured data (such as data in a relational database) is database interface; the target interface type for unstructured data (such as files, images, etc.) is file interface. Alternatively, the target interface type can also be determined based on the data transfer method between systems. For example, if the data transfer method is FTP, the target interface type for the data to be processed can be file interface; if the data transfer method is ODBC, the target interface type for the data to be processed can be database interface.

[0035] Considering that different source systems may use different data formats, data structures, and encoding methods, the processing requirements for the data to be processed from different source systems may also differ. To improve the effectiveness of data processing, during the process of determining the target interface type corresponding to the data to be processed, the target data source to which the data to be processed belongs can be identified, and the target association information corresponding to the target data source can be determined. Based on the target data source, the target interface type can be determined, and based on the target association information, the interface parameters corresponding to the target interface type can be determined, so that the data to be processed can be processed based on the interface parameters.

[0036] Among them, docking parameters refer to the parameters that need to be set during the docking process, which are used to specify the docking target, method, data format, etc.

[0037] Specifically, the source of the data to be processed can be designated as the target data source, and the target association information corresponding to the target data source can be determined. Optionally, the target association information may include, but is not limited to, the data source's address or domain name, port number, transmission protocol, encryption method, data format, data structure, encoding method, etc. Furthermore, the target connection type corresponding to the data to be processed can be determined based on the target data source. For example, connection types corresponding to different data sources can be pre-configured. If the data to be processed comes from a database, then its target connection type is database connection; if the data to be processed comes from a message queue, then its target connection type is message connection. For example, relevant source system information can be recorded according to different connection types, including the data source's system name, corresponding IP address, connection type, etc.; the corresponding connection type can be found as the target connection type based on the system name, IP address, etc. Further, connection parameters corresponding to the target connection type can be determined from the target association information, so that the data to be processed can be processed using the connection parameters, thereby improving data processing efficiency while ensuring effective data processing.

[0038] S120. Determine the target runtime environment corresponding to the target docking type, and load the target runtime environment.

[0039] The target runtime environment refers to the operating system used during data processing. Different data formats require different runtime environments; for example, these can be big data systems, database systems, file systems, distributed systems, etc. Different runtime environments employ different processing methods and tools to handle the processing of different data formats.

[0040] In practical applications, a suitable target runtime environment can be selected based on the target interface type of the data to be processed. This target runtime environment is then loaded, and the data to be processed is distributed to it, ensuring the correctness of the data processing. For example, if the target interface type is database interface, a relational database management system (RDBMS) can be used as the target runtime environment; if the target interface type is file interface, a text analysis system can be used as the target runtime environment.

[0041] To improve the compatibility between the data to be processed and the target runtime environment, during the process of determining the target runtime environment corresponding to the target interface type and loading the target runtime environment, the target runtime environment can be obtained from the source system based on the target interface type, the target association information to which the data to be processed belongs, and the configuration information, so that data processing can be performed in the target runtime environment.

[0042] The configuration information includes, but is not limited to, the connection type corresponding to different data sources, and the runtime environment identifier corresponding to different connection types. The runtime environment identifier can be used to indicate the uniqueness of the runtime environment.

[0043] In this embodiment, an identifier (e.g., the address or name of the data source) that represents the uniqueness of the data source can be obtained from the target association information to which the data to be processed belongs. Then, based on the data source identifier and the target docking type, a runtime environment identifier corresponding to the data source identifier and the target docking type can be selected from the configuration information. Then, the data to be processed can be distributed to the target runtime environment corresponding to the runtime environment identifier for data processing in the target runtime environment.

[0044] For example, the target association information to which the data to be processed belongs can be called Table A, the configuration information of the data source can be called Table B, and the environment information of the target system can be called Table C. By filtering the target association information of the source end of "Table A" for the current docking date, matching it with "Table B" to obtain the corresponding environment identifier, and matching "Table C" with this identifier to obtain the environment information of the target system as the target running environment, the data to be processed can then be distributed to the target running environment.

[0045] S130. Under the target operating environment, perform preliminary verification of the data to be processed based on the pre-verification rules, and store the data to be processed in a temporary data table if the verification result is consistent with the preset verification result.

[0046] Pre-validation rules refer to pre-configured rules that validate data before processing it. Validation results include success and failure; the default validation result is success.

[0047] In this embodiment, after the data to be processed is distributed to the target runtime environment, pre-configured pre-validation rules can be used to perform preliminary validation on the data. For example, the data format can be validated: checking whether the format of dates, times, and values ​​meets the requirements; the data length can be validated: checking whether the data exceeds the specified length; the data type can be validated: checking whether the data conforms to the specified data type; encoding type validation, data dictionary type validation, etc., to obtain the corresponding validation results. The validation results are compared with the preset validation results. If they match, all data to be processed can be stored in a temporary data table, or only the portion of data to be processed whose validation results match the preset validation results can be stored in the temporary data table. The advantage of pre-validation is that it ensures the integrity and accuracy of the data, avoiding errors or anomalies in subsequent processing.

[0048] Optionally, when configuring pre-validation rules, at least one validation field, validation format, and validation function corresponding to the pre-validation rule can be configured.

[0049] The validation field can be a field used to verify the integrity and accuracy of data. For example, when data passes through the validation field, it is compared with the value of the validation field to determine if the data is correct. If the data does not match the value of the validation field, it indicates that the data may contain errors or anomalies, meaning the validation result for that field is a failure. If the data matches the value of the validation field, it indicates that the data is normal. The validation format can be a defined data format. The format of the data to be processed is compared with the validation format to determine if the data format is correct. The validation function can be a function used to verify the integrity and accuracy of data. For example, a validation function performs specific calculations or processing on the data to obtain a validation value or result, which is then compared with the original data to determine if the data contains errors or anomalies. Validation functions include, but are not limited to, total summation check, parity check, and CRC (Cyclic Redundancy Check).

[0050] For example, for data file-based interfaces, the pre-verification stage can check data format, length, file format, common encoding types, and data dictionary definitions (by comparing files with control files). The verification results can be recorded in a data table so users can process the data accordingly. Verified data can be copied (between databases), parsed (messages), or loaded (data files) into a temporary data table. For large amounts of data, such as exceeding 1GB, the data can be split into equivalent parts and stored in a temporary data table.

[0051] S140. When the conditions for post-data verification are met, retrieve the data to be processed from the temporary data table and perform post-verification processing according to the post-verification rules corresponding to the data to be processed to obtain the target verification result.

[0052] Among them, post-validation rules can refer to rules for validating data after data processing.

[0053] In this embodiment, the conditions for post-verification of data are considered met when the data to be processed that has passed verification is detected to be stored in a temporary data table; alternatively, the conditions for post-verification of data are considered met when a preset post-verification period is reached. When the conditions for post-verification of data are met, the data to be processed is retrieved from the temporary data table, and then post-verification processing is performed on the data according to the post-verification rules corresponding to the data to be processed to obtain the target verification result.

[0054] Optionally, when configuring post-validation rules, rule data corresponding to the post-validation rules can be configured. This rule data can contain validation rules for at least one validation dimension, allowing data to be validated based on these rules. For example, validation dimensions include, but are not limited to, technical, business, and security dimensions. The rule data corresponds to pre-developed business information.

[0055] In this embodiment, when the conditions for post-verification of data are met, the data to be processed is retrieved from the temporary data table, and post-verification processing is performed according to the post-verification rules corresponding to the data to be processed to obtain the target verification result. This includes: retrieving the data to be processed from the temporary data table when the amount of data in the temporary data table that is of the target interface type to which the data to be processed belongs reaches a preset data amount threshold, or when a timed task is received; retrieving the post-verification rules corresponding to the data to be processed, and performing verification processing on the data to be processed according to at least one verification dimension in the post-verification rules to obtain the target verification result.

[0056] Scheduled tasks can be pre-configured and automatically execute specified tasks according to preset time intervals or conditions.

[0057] In practical applications, the system can automatically retrieve the data to be processed from the temporary data table when the amount of data in the temporary data table that corresponds to the target interface type of the data to be processed reaches a preset data volume threshold, or when a scheduled task is received. Furthermore, it can retrieve the post-validation rules corresponding to the data to be processed, and then use the validation rules under at least one validation dimension of the post-validation rules to validate the data to be processed, obtaining the corresponding target validation result.

[0058] For example, in the post-validation stage, post-validation rules are defined under two dimensions: technical and business. In the technical dimension, the data to be processed in the temporary data table is validated based on field attributes (whether it is non-nullable) and uniqueness. In the business dimension, a business validation rule table (referred to as Table E) is defined using pre-configured business validation rules. The data to be processed in the temporary data table is matched against Table E to obtain relevant validation rules, and these rules are then used to validate the data to be processed. The advantage of this setup is that post-validation rules are used to verify the integrity and accuracy of the data, preventing data corruption or loss during transmission and errors during storage or processing.

[0059] When the verification result of the data to be processed is determined to be inconsistent with the preset verification result based on the pre-verification rules and / or post-verification rules, the data to be processed and the error information are stored in the first data table for later retrieval.

[0060] Specifically, when the verification result of the data to be processed is inconsistent with the preset verification result based on the pre-verification rules and / or post-verification rules, the data to be processed and the error information of the verification result failure can be stored in the first data table for later retrieval, so as to facilitate targeted processing of the data with the verification result failure.

[0061] S150. When the target verification result matches the preset verification result, store the data to be processed in the target cache location and remove the data to be processed from the temporary data table.

[0062] In this embodiment, the target verification result obtained after post-verification can be compared with the preset verification result. If the target verification result matches the preset verification result, the data to be processed can be stored in the target cache location, and the data to be processed can be removed from the temporary data table. For example, the data to be processed that meets the post-verification rules can be updated to the target table in a predetermined manner, and this target table is the target cache location.

[0063] This embodiment solves the problem of high data processing costs and poor versatility caused by deploying a separate processing device for each data interface type in the prior art. It achieves automatic adaptation to the target interface type based on the target interface type obtained from the source system, distributing the data to the target interface to the target operating environment. This allows a single system to support data processing for multiple interface types, improving data processing versatility while reducing data processing costs. The system automatically adapts to the target operating environment based on the target interface type obtained from the source system, distributing the data to the target operating environment. This enables a single system to support data processing for multiple interface types, improving data processing versatility while reducing data processing costs. When the data post-verification conditions are met, the data is retrieved from the temporary data table and post-verified according to the corresponding post-verification rules to obtain the target verification result. Finally, when the target verification result matches the preset verification result, the data is stored in the target cache location and removed from the temporary data table. Furthermore, in the target operating environment, pre-verification rules are used to perform preliminary verification of the data to be processed, ensuring the integrity and accuracy of the data before processing. After the pre-verification passes, the data to be processed is temporarily stored. Then, post-verification is performed on the data to be processed according to the post-verification rules corresponding to the data to be processed, ensuring the integrity and accuracy of the data after processing. After the post-verification passes, the data to be processed is stored in the target cache location, ensuring data quality and removing the data to be processed from the temporary data table, reducing system resource pressure.

[0064] Based on the above embodiments, at least one data source and its corresponding association information can be predetermined, and a corresponding invocation interface type can be configured for at least one data source. Based on the configuration information and association information, the target interface type corresponding to the received data to be processed can be determined. Accordingly, the present invention proposes the following embodiments: Figure 2 This is a flowchart of a data processing method provided according to an embodiment of the present invention. Figure 2 As shown, the data processing method includes the following steps:

[0065] S210. Determine at least one data source and the associated information corresponding to the data source.

[0066] In this embodiment, the associated information corresponding to each data source can be determined based on the relevant information of different data sources, and the correspondence between the data source and its corresponding associated information can be established. This allows the target associated information corresponding to the target data source to be determined from these associated information, and the docking parameters corresponding to the target docking type can be determined based on the target associated information to process the data to be processed.

[0067] Optionally, at least one data source and its associated information are determined, including: when the data source is a database type, the associated information includes the database name, IP address, port, and database instance name; when the data source is a message type, the associated information includes the source IP, transaction code, and message message; when the data source is a data file type, the associated information includes the source IP, file path, and file name.

[0068] For example, data source association information can be recorded in the connection log table. Non-empty fields in the table include "data source system identifier, connection date, association information, and connection type." When the data source is a database, the corresponding association information includes the database name, IP address, port, and database instance name. When the data source is a message, the corresponding association information includes the source IP, transaction code, and message message. When the data source is a data file, the corresponding association information includes the source IP, file path, and file name, with the "connection type" recorded as "FILE".

[0069] S220. Configure the corresponding docking type to be invoked for at least one data source, so as to determine the target docking type corresponding to the received data to be processed based on the configuration information and association information.

[0070] In this embodiment, a corresponding docking type can be configured for at least one data source so that when data to be processed is received, the target docking type corresponding to the data to be processed is determined based on the target data source to which the data to be processed belongs. For example, when the data source is a database type, the docking type to be called is database docking, and the "Docking type" is recorded as "DB"; when the data source is a message type, the docking type to be called is message docking, and the "Docking type" is recorded as "MSG"; when the data source is a data file type, the docking type to be called is file docking, and the "Docking type" is recorded as "FILE". A corresponding runtime environment can also be configured for the docking type to be called so that the target runtime environment is determined when the target docking type is determined.

[0071] This embodiment determines the association information corresponding to at least one data source, configures the corresponding call interface type for at least one data source, and configures the corresponding runtime environment for the call interface type. This allows the target interface type corresponding to the received data to be processed to be determined based on the configuration information and association information. Then, after determining the target interface type, the target runtime environment required for data processing is automatically adapted so that the data to be processed is processed in the target runtime environment, ensuring the effectiveness and accuracy of data processing.

[0072] Based on the above embodiments, when performing preliminary verification of the data to be processed based on the pre-verification rules, the data to be processed can be preliminarily verified according to at least one level of verification sub-rules in the pre-verification rules. Then, when the verification results of at least one verification sub-rules are consistent with the preset verification results, the data to be processed is stored in a temporary data table. Accordingly, the present invention proposes the following embodiments: Figure 3 This is a flowchart of a data processing method provided according to an embodiment of the present invention. Figure 3 As shown, the data processing method includes the following steps:

[0073] S310. Perform preliminary verification on the data to be processed according to at least one level of verification sub-rule in the pre-verification rules.

[0074] In this embodiment, the level can characterize the rigor or precision requirement of data validation, and different levels correspond to different validation sub-rules. For example, data format validation is level A, and data length validation is level B. The higher the level, the higher the rigor and precision requirement of data validation; the lower the level, the lower the rigor and precision requirement of data validation. In practical applications, the validation sub-rules of each level in the pre-validation rules can be used to perform preliminary validation of the data to be processed.

[0075] S320. When the verification results of at least one verification sub-rule are consistent with the preset verification results, the data to be processed is stored in a temporary data table.

[0076] In this embodiment, after verifying the data to be processed based on each verification sub-rule, the verification result of each verification sub-rule is obtained. The verification result of each verification sub-rule can be compared with the preset verification result. If each verification result is consistent with the preset verification result, the data to be processed can be stored in a temporary data table.

[0077] In this embodiment, when the verification result of a verification sub-rule is inconsistent with the preset verification result, the target verification sub-rule corresponding to the inconsistency and the target level of the target verification sub-rule are determined; the feedback method corresponding to the target level is determined, and the data to be processed is processed according to the feedback method.

[0078] Feedback methods may include, but are not limited to, feedback channels, feedback formats, and feedback descriptions. For example, feedback channels may include SMS, email, and apps; feedback formats may include charts, text descriptions, visualizations, and voice messages.

[0079] Specifically, if the verification result of a validation sub-rule is inconsistent with the preset verification result, the level of the validation sub-rule used when the inconsistency occurred can be used as the target level. Furthermore, a feedback method suitable for the target level can be selected, and then feedback can be provided to the data to be processed according to this feedback method; alternatively, feedback can be provided to the data to be processed verified by the validation sub-rule used when the inconsistency occurred, based on the feedback method.

[0080] For example, different levels can correspond to different feedback methods. A high level might send inconsistent verification results to the user via SMS and email; a medium level might send inconsistent results via the app; and a low level might simply record the results without providing feedback. For instance, based on the target level, verification results can be selectively recorded in the feedback form, and then sent to the target user through the appropriate feedback channel. The advantage of this setup is that by focusing on strongly inconsistent verification results, targeted verification prompts can be provided, improving the effectiveness of data monitoring.

[0081] This embodiment performs preliminary verification of the data to be processed based on at least one level of verification sub-rules in the pre-verification rules. When the verification results of at least one verification sub-rule are consistent with the preset verification results, the data to be processed is stored in a temporary data table, ensuring data integrity before processing. When the verification results of a verification sub-rule are inconsistent with the preset verification results, a feedback method corresponding to the target level of the target verification sub-rule corresponding to the inconsistency is determined, enabling targeted feedback of error information and facilitating the user to adopt corresponding processing strategies based on the feedback information.

[0082] As an optional embodiment of the above embodiments, Figure 4 This is a schematic diagram of a data processing method provided according to an embodiment of the present invention. For details, please refer to the following specific content.

[0083] See Figure 4The technical solution provided in this embodiment can be implemented based on a data docking device, which can consist of a data docking unit, an environment allocation unit, a data processing unit, and an information feedback unit. The data docking unit can record docking information (including data source association information and corresponding docking types) of relevant data sources according to different docking types. The docking information includes the system name, corresponding IP address, file transfer node, and current docking type. Through the data docking unit, the docking information is recorded in docking log table A. Non-empty fields in table A include "docking date," "source system name," and "docking type." For example, for the database type, "database name, IP address, port, and database instance name" are recorded in table A, and the "docking type" is recorded as "DB"; for the message type, "source IP, transaction code, and message content (corresponding message)" are recorded in table A, and the "docking type" is recorded as "MSG"; for the file type, "source IP, file path, and file name" are recorded in table A, and the "docking type" is recorded as "FILE." In practical applications, the data docking unit can be used as a unified data docking interface. When obtaining data to be processed (including data, messages, data files, etc. from the source system), the data docking unit identifies the docking type based on the target data source of the data to be processed. The docking types include direct docking between databases (i.e., database type), message docking type, file type, etc.

[0084] Furthermore, the environment allocation unit, based on the current source system information (i.e., the data source association information) recorded by the data docking unit, and according to the target docking type, obtains the environment identifier corresponding to the target docking type from the data source environment information using information such as system name and IP address. Then, it retrieves the corresponding target runtime environment from the target system environment information database, and finally distributes the data to be processed to the target runtime environment. That is, the environment allocation unit defines a data source environment information table B and a target environment information table C, see [link to relevant documentation]. Figure 5The process involves filtering the data source information of the current connection date in the connection flow table A, matching it with table B, obtaining the environment identifier corresponding to the data source, and then using this identifier to match table C to obtain the target system's environment information as the target operating environment. Finally, the data is distributed to the target operating environment. Further, within the target operating environment, the data processing unit can process the data to be processed, performing multi-faceted verifications such as technical verification and business rule verification. Different processing methods are used for different data connection types, such as direct database writing, transaction message parsing, and data file parsing and importing. For example, for data file type connection methods, in the pre-verification stage, the data to be processed is verified based on pre-verification rules regarding data format, length, file format, common encoding types, and data dictionary definitions, and the verification results are recorded in a table. Simultaneously, based on the level of the verification sub-rule corresponding to the verification result, the verification results are selectively recorded in the information feedback table, along with the feedback channel (e.g., SMS, email, APP), the system to be fed back, and the personnel involved. Under the condition that the verification passes, the data processing unit processes the data to be processed, such as by using processing methods corresponding to the data to be processed, such as copying (between databases), message parsing (messages), and file loading (data files), to store the data to be processed into a temporary data table T. In the post-verification stage, post-verification rules are defined under two verification dimensions: technical and business. In the technical dimension, the data to be processed in the temporary data table is verified from aspects such as field attributes (whether it is not null) and uniqueness. In the business dimension, a business verification rule table E is defined based on pre-configured business verification rules. The data to be processed in the temporary data table is matched against table E to obtain relevant verification rules, and these rules are used to verify the data to be processed. The verification results are then selectively recorded in the information feedback table D (e.g., verification results that are inconsistent with the preset verification results are recorded in table D). The data to be processed that meets the verification rules is updated to the target table according to a predetermined method. Through the information feedback unit, the information recorded in the information feedback table D in the data processing unit is used to provide targeted reminders through pre-written feedback channels, systems, personnel, and other information.

[0085] The technical solution provided in this embodiment automatically adapts to the target operating environment corresponding to the target interface type based on the data to be processed obtained from the source system. It then distributes the data to the target operating environment, enabling a single system to support data processing for multiple interface types. This improves the versatility of data processing while reducing processing costs. Furthermore, in the target operating environment, pre-verification rules are used to initially verify the data to be processed, ensuring its integrity and accuracy before processing. Post-verification rules corresponding to the data are also used for post-verification, ensuring its integrity and accuracy after processing. After successful post-verification, the data to be processed is stored in the target cache location, guaranteeing data quality. The data is then removed from the temporary data table, reducing system resource pressure.

[0086] Figure 6 This is a schematic diagram of the structure of a data processing apparatus according to an embodiment of the present invention. Figure 6 As shown, the device includes: a target docking type determination module 410, a target operating environment determination module 420, a pre-verification module 430, a post-verification module 440, and a data processing module 450.

[0087] The system includes the following modules: a target docking type determination module 410, used to determine the target docking type corresponding to the data to be processed when obtaining the data to be processed from the source system; a target operating environment determination module 420, used to determine the target operating environment corresponding to the target docking type and load the target operating environment; a pre-verification module 430, used to perform preliminary verification on the data to be processed based on pre-verification rules in the target operating environment, and store the data to be processed in a temporary data table if the verification result is consistent with the preset verification result; a post-verification module 440, used to retrieve the data to be processed from the temporary data table when the conditions for post-verification are met, and perform post-verification processing according to the post-verification rules corresponding to the data to be processed to obtain the target verification result; and a data processing module 450, used to store the data to be processed in the target cache location and remove the data to be processed from the temporary data table when the target verification result is consistent with the preset verification result.

[0088] The technical solution of this embodiment, when obtaining data to be processed from the source system, determines the target interface type corresponding to the data to be processed; determines the target operating environment corresponding to the target interface type and loads the target operating environment; in the target operating environment, performs preliminary verification of the data to be processed based on pre-verification rules, and stores the data to be processed in a temporary data table if the verification result is consistent with the preset verification result; when the conditions for post-verification are met, retrieves the data to be processed from the temporary data table, and performs post-verification processing according to the post-verification rules corresponding to the data to be processed to obtain the target verification result; when the target verification result is consistent with the preset verification result, stores the data to be processed in the target cache location and removes the data to be processed from the temporary data table. This solves the problem of high data processing cost and poor versatility caused by deploying a separate processing device for each data interface type in the prior art. It realizes that by automatically adapting to the target operating environment corresponding to the target interface type based on the target interface type of the data to be processed obtained from the source system, the data to be processed is distributed to the target operating environment, enabling a single system to support data processing for multiple interface types, improving the versatility of data processing while reducing data processing costs. Furthermore, in the target operating environment, pre-verification rules are used to perform preliminary verification of the data to be processed, ensuring the integrity and accuracy of the data before processing. After the pre-verification passes, the data to be processed is temporarily stored. Then, post-verification is performed on the data to be processed according to the post-verification rules corresponding to the data to be processed, ensuring the integrity and accuracy of the data after processing. After the post-verification passes, the data to be processed is stored in the target cache location, ensuring data quality and removing the data to be processed from the temporary data table, reducing system resource pressure.

[0089] Optionally, based on the above-described device, the device may further include an association information configuration unit and an association information configuration unit.

[0090] The association information configuration unit is used to determine at least one data source and the association information corresponding to the data source; the docking type configuration unit is used to configure the corresponding docking type to be invoked for the at least one data source, so as to determine the target docking type corresponding to the received data to be processed based on the configuration information and the association information.

[0091] Based on the above-mentioned device, optionally, the association information configuration unit is configured to, when the data source is a database type, include the database name, IP address, port, and database instance name in the association information; when the data source is a message type, include the source IP, transaction code, and message message in the association information; and when the data source is a data file type, include the source IP, file path, and file name in the association information.

[0092] Optionally, based on the above-described device, the device may also include an operating environment configuration unit.

[0093] The runtime environment configuration unit is used to configure the corresponding runtime environment for the type of interface to be called, so as to determine the target runtime environment when the target interface type is determined.

[0094] Based on the above-mentioned device, optionally, the target docking type determination module 410 includes a target association information determination unit and a docking parameter determination unit.

[0095] The target association information determination unit is used to determine the target association information corresponding to the target data source based on the target data source to which the data to be processed belongs; the docking parameter determination unit is used to determine the target docking type based on the target data source, and determine the docking parameters corresponding to the target docking type based on the target association information, so as to process the data to be processed based on the docking parameters.

[0096] Based on the above-mentioned device, optionally, a target operating environment determination module 420 is used to obtain the target operating environment from the source system according to the target docking type, the target association information to which the data to be processed belongs, and the configuration information, so as to perform data processing in the target operating environment.

[0097] Based on the above-mentioned device, optionally, the pre-verification module 430 includes a pre-verification unit and a storage unit.

[0098] The pre-verification unit is used to perform preliminary verification on the data to be processed according to at least one level of verification sub-rules in the pre-verification rules; the storage unit is used to store the data to be processed in the temporary data table when the verification results of the at least one verification sub-rules are consistent with the preset verification results.

[0099] Optionally, based on the above-mentioned device, the device may further include: a target level determination unit and a feedback method determination unit.

[0100] The target level determination unit is used to determine the target verification sub-rule corresponding to the inconsistency and the target level of the target verification sub-rule when the verification result of the existing verification sub-rule is inconsistent with the preset verification result; the feedback method determination unit is used to determine the feedback method corresponding to the target level and process the data to be processed according to the feedback method.

[0101] Based on the above-mentioned device, optionally, the post-verification module 440 includes a data retrieval unit to be processed and a post-verification unit.

[0102] The pending data retrieval unit is used to retrieve the pending data from the temporary data table when the amount of data of the target interface type to which the pending data belongs in the temporary data table reaches a preset data amount threshold, or when a timed task is received; the post-verification unit is used to retrieve the post-verification rules corresponding to the pending data, and to verify the pending data according to at least one verification dimension in the post-verification rules to obtain the target verification result.

[0103] Optionally, based on the above-described device, the device may further include an error information storage unit.

[0104] An error information storage unit is used to store the data to be processed and the error information in a first data table for later retrieval when the verification result of the data to be processed is determined to be inconsistent with the preset verification result based on the pre-verification rules and / or post-verification rules.

[0105] Optionally, based on the above-described device, the device may further include: a verification rule configuration unit.

[0106] The verification rule configuration unit is used to configure at least one verification field, verification format, and verification function corresponding to the pre-verification rule, and to configure rule data corresponding to the post-verification rule, wherein the rule data corresponds to pre-developed business information.

[0107] The data processing apparatus provided in the embodiments of the present invention can execute the data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0108] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 7 A block diagram is shown of an exemplary electronic device 50 suitable for implementing embodiments of the present invention. Figure 7 The electronic device 50 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0109] like Figure 7 As shown, the electronic device 50 is represented in the form of a general-purpose computing device. The components of the electronic device 50 may include, but are not limited to: one or more processors or processing units 501, system memory 502, and bus 503 connecting different system components (including system memory 502 and processing unit 501).

[0110] Bus 503 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0111] Electronic device 50 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 50, including volatile and non-volatile media, removable and non-removable media.

[0112] System memory 502 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 504 and / or cache memory 505. Electronic device 50 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 506 may be used to read and write non-removable, non-volatile magnetic media (… Figure 7 Not shown; usually referred to as a "hard drive"). Although Figure 7 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 503 via one or more data media interfaces. Memory 502 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0113] A program / utility 508 having a set (at least one) of program modules 507 may be stored, for example, in memory 502. Such program modules 507 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 507 typically perform the functions and / or methods described in the embodiments of the present invention.

[0114] Electronic device 50 can also communicate with one or more external devices 509 (e.g., keyboard, pointing device, display 510, etc.), and with one or more devices that enable a user to interact with the electronic device 50, and / or with any device that enables the electronic device 50 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 511. Furthermore, electronic device 50 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 512. As shown, network adapter 512 communicates with other modules of electronic device 50 via bus 503. It should be understood that, although... Figure 7 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 50, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0115] The processing unit 501 executes various functional applications and data processing by running programs stored in the system memory 502, such as implementing the data processing method provided in the embodiments of the present invention.

[0116] This invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a data processing method, the method comprising:

[0117] When obtaining data to be processed from the source system, determine the target docking type corresponding to the data to be processed;

[0118] Determine the target runtime environment corresponding to the target docking type, and load the target runtime environment;

[0119] In the target operating environment, the data to be processed is initially verified based on the pre-verification rules, and the data to be processed is stored in a temporary data table if the verification result is consistent with the preset verification result.

[0120] When the conditions for post-data verification are met, the data to be processed is retrieved from the temporary data table and post-verified according to the post-verification rules corresponding to the data to be processed to obtain the target verification result.

[0121] When the target verification result matches the preset verification result, the data to be processed is stored in the target cache location, and the data to be processed is removed from the temporary data table.

[0122] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0123] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0124] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0125] Computer program code for performing the operations of embodiments of the present invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0126] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the data processing method provided in any embodiment of this application.

[0127] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0128] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A data processing method, characterized in that, include: When obtaining data to be processed from the source system, determine the target docking type corresponding to the data to be processed; Determine the target runtime environment corresponding to the target docking type, and load the target runtime environment; In the target operating environment, the data to be processed is initially verified based on the pre-verification rules, and the data to be processed is stored in a temporary data table if the verification result is consistent with the preset verification result. When the conditions for post-data verification are met, the data to be processed is retrieved from the temporary data table, and post-data verification is performed according to the post-data verification rules corresponding to the data to be processed to obtain the target verification result. When the target verification result matches the preset verification result, the data to be processed is stored in the target cache location and removed from the temporary data table; The step of determining the target docking type corresponding to the data to be processed includes: Based on the target data source to which the data to be processed belongs, the target association information corresponding to the target data source is determined; The target docking type is determined based on the target data source, and the docking parameters corresponding to the target docking type are determined based on the target association information, so as to process the data to be processed based on the docking parameters; the target docking type includes file docking and database docking, and the docking parameters refer to the parameters that need to be set during the docking process to specify the docking target, method and data format; The method further includes: Identify at least one data source and the associated information corresponding to the data source; Configure a corresponding docking type to be invoked for the at least one data source, so as to determine the target docking type corresponding to the received data to be processed based on the configuration information and the association information; The step of determining the target runtime environment corresponding to the target docking type and loading the target runtime environment includes: Based on the target docking type, the target association information to which the data to be processed belongs, and the configuration information, the target operating environment is obtained from the source system so as to perform data processing in the target operating environment; The step of performing preliminary verification on the data to be processed based on pre-verification rules, and storing the data to be processed in a temporary data table if the verification result is consistent with the preset verification result, includes: The data to be processed is initially verified according to at least one level of the verification sub-rules in the pre-verification rules; When the verification results of at least one verification sub-rule are consistent with the preset verification results, the data to be processed is stored in the temporary data table; The step of retrieving the data to be processed from the temporary data table when the conditions for post-validation are met, and performing post-validation processing according to the post-validation rules corresponding to the data to be processed to obtain the target validation result includes: When the amount of data in the temporary data table that corresponds to the target interface type of the data to be processed reaches a preset data volume threshold, or when a scheduled task is received, the data to be processed is retrieved from the temporary data table. Retrieve the post-validation rules corresponding to the data to be processed, and validate the data to be processed according to at least one validation dimension in the post-validation rules to obtain the target validation result; The method further includes: When the verification result of the data to be processed is determined to be inconsistent with the preset verification result based on the pre-verification rules and / or post-verification rules, the data to be processed and the error information are stored in the first data table for later retrieval.

2. The method according to claim 1, characterized in that, The determination of at least one data source and the associated information corresponding to the data source includes: When the data source is a database, the associated information includes the database name, IP address, port, and database instance name; When the data source is a message type, the associated information includes the source IP, transaction code, and message message. When the data source is a data file type, the associated information includes the source IP address, file path, and file name.

3. The method according to claim 1, characterized in that, Also includes: Configure the corresponding runtime environment for the interface type to be invoked, so that the target runtime environment can be determined when the target interface type is determined.

4. The method according to claim 1, characterized in that, Also includes: When the verification result of a verification sub-rule is inconsistent with the preset verification result, the target verification sub-rule corresponding to the inconsistency and the target level of the target verification sub-rule are determined. Determine the feedback method corresponding to the target level, and process the data to be processed according to the feedback method.

5. The method according to claim 1, characterized in that, Also includes: Configure at least one verification field, verification format, and verification function corresponding to the pre-verification rule, and configure rule data corresponding to the post-verification rule, wherein the rule data corresponds to pre-developed business information.

6. A data processing apparatus, characterized in that, include: The target docking type determination module is used to determine the target docking type corresponding to the data to be processed when the data to be processed is obtained from the source system. The target runtime environment determination module is used to determine the target runtime environment corresponding to the target docking type and load the target runtime environment; The pre-verification module is used to perform preliminary verification on the data to be processed based on pre-verification rules under the target operating environment, and store the data to be processed in a temporary data table if the verification result is consistent with the preset verification result. The post-verification module is used to retrieve the data to be processed from the temporary data table when the conditions for post-verification of data are met, and to perform post-verification processing according to the post-verification rules corresponding to the data to be processed, so as to obtain the target verification result. The data processing module is used to store the data to be processed in the target cache location and remove the data to be processed from the temporary data table when the target verification result is consistent with the preset verification result; The target docking type determination module includes a target association information determination unit and a docking parameter determination unit; The target association information determination unit is used to determine the target association information corresponding to the target data source based on the target data source to which the data to be processed belongs; The docking parameter determination unit is used to determine the target docking type based on the target data source, and to determine the docking parameters corresponding to the target docking type based on the target association information, so as to process the data to be processed based on the docking parameters; the target docking type includes file docking and database docking, and the docking parameters refer to the parameters that need to be set during the docking process, which are used to specify the docking target, method and data format; The device further includes: An association information configuration unit is used to determine at least one data source and the association information corresponding to the data source; The docking type configuration unit is used to configure the corresponding docking type to be invoked for the at least one data source, so as to determine the target docking type corresponding to the received data to be processed based on the configuration information and the association information; The target operating environment determination module is used to obtain the target operating environment from the source system based on the target docking type, the target association information to which the data to be processed belongs, and the configuration information, so as to perform data processing in the target operating environment; The pre-verification module includes: The pre-verification unit is used to perform preliminary verification on the data to be processed according to at least one level of verification sub-rules in the pre-verification rules. A storage unit is used to store the data to be processed into the temporary data table when the verification results of at least one verification sub-rule are consistent with the preset verification results; The post-verification module includes: The pending data retrieval unit is used to retrieve the pending data from the temporary data table when the amount of data of the target interface type to which the pending data belongs in the temporary data table reaches a preset data amount threshold, or when a timed task is received. The post-verification unit is used to retrieve the post-verification rules corresponding to the data to be processed, and to perform verification processing on the data to be processed according to at least one verification dimension in the post-verification rules to obtain the target verification result. The device further includes: An error information storage unit is used to store the data to be processed and the error information in a first data table for later retrieval when the verification result of the data to be processed is determined to be inconsistent with the preset verification result based on the pre-verification rules and / or post-verification rules.

7. An electronic device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the data processing method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the data processing method as described in any one of claims 1-5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1-5.

Images