Multi-source data automatic processing method and system

Abstract

This application discloses a method and system for automated processing of multi-source data. The method includes: collecting raw data from heterogeneous data sources and marking the raw data with privacy levels to obtain marked raw data; matching corresponding privacy rules in a preset privacy rule library according to the privacy level, and standardizing the marked raw data based on the privacy rules to obtain standardized data; responding to triggered business instructions, matching a target template from a preset business template library, and filling the target template with standardized data to generate a current result file; and delivering the current result file to the target business system associated with the business instruction to execute the business operation corresponding to the business instruction. This application can match differentiated processing rules based on data privacy levels to achieve targeted security processing of data with different levels of sensitivity, thereby ensuring automated closed-loop processing of the entire data process and effectively reducing the risk of data leakage.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and system for automated processing of multi-source data. Background Technology

[0002] With the development of digital transformation, enterprises and organizations have an increasingly urgent need for the integration and automated processing of multi-source, heterogeneous data. The data to be processed typically originates from various channels such as public websites, internal business systems, and offline documents, and its formats are diverse, containing varying levels of privacy information. To improve business efficiency, the industry is generally seeking to automate data collection, processing, content generation, and execution through technological means to replace traditional, highly manual operation models.

[0003] Current methods typically employ robotic process automation (RPA), application programming interface (API) synchronization, and scripting to automate specific steps. However, these methods are often designed for single data sources or independent processing steps, lacking effective collaboration between stages and failing to form a complete automated closed loop covering the entire data acquisition process from collection to final execution. Consequently, when processing multi-source heterogeneous data, the system struggles to dynamically adapt to changes in data sources to implement differentiated security processing based on data privacy attributes, thus posing a risk of data leakage. Summary of the Invention

[0004] The main purpose of this application is to provide an automated method and system for processing multi-source data, which aims to solve the technical problem that existing multi-source data processing methods are unable to meet the differentiated processing needs of privacy data from heterogeneous data sources.

[0005] To achieve the above objectives, this application proposes an automated multi-source data processing method, the method comprising: Raw data from heterogeneous data sources is collected, and the raw data is marked with a privacy level to obtain the marked raw data; According to the privacy level, the corresponding privacy rule is matched in the preset privacy rule library, and the original marked data is standardized based on the privacy rule to obtain standardized data; In response to a triggered business instruction, a target template is matched from a preset business template library, and the standardized data is filled into the target template to generate the current result file; The current result file is delivered to the target business system associated with the business instruction to execute the business operation corresponding to the business instruction.

[0006] In one embodiment, the step of collecting raw data from heterogeneous data sources includes: When the heterogeneous data source is a first type of data source, the page structure information of the first type of data source is parsed by a preset page structure parsing model, and a dynamic crawling strategy is generated according to the page structure information. Data is crawled in the first type of data source according to the dynamic crawling strategy to obtain the first raw data. When the heterogeneous data source is a second type of data source, data is retrieved from the second type of data source through a preset application interface according to pre-configured parameters to obtain the second original data; The first raw data and the second raw data are identified as the raw data.

[0007] In one embodiment, the step of marking the original data with a privacy level to obtain the marked original data includes: The original data is compared with several privacy features in a preset data privacy feature library; Based on the comparison results, data items with different privacy levels are identified in the original data, and each data item is marked with a privacy level according to its privacy level to obtain the marked original data.

[0008] In one embodiment, the tagged raw data includes several data items corresponding to different privacy levels; The step of matching the corresponding privacy rule in a preset privacy rule base according to the privacy level, and standardizing the tagged original data based on the privacy rule to obtain standardized data includes: Determine the privacy level corresponding to each data item, and query the privacy rules matching each privacy level in the preset privacy rule library. The preset privacy rule library is pre-configured by the management user and includes: high privacy data rules, medium privacy data rules and low privacy data rules. A data processing optimization strategy is obtained, and each privacy rule is updated according to the data processing optimization strategy to obtain the updated privacy rules. The data processing optimization strategy is generated based on historical processing results. Each data item is standardized according to the updated privacy rules to obtain standardized data.

[0009] In one embodiment, the step of matching a target template from a preset business template library and filling the target template with the standardized data in response to a triggered business instruction to generate the current result file includes: Upon receiving a business instruction, the current business scenario is determined based on the business instruction; Based on the current business scenario, a target template is matched in the preset business template library. The preset business template library is pre-configured by the management user through the template editing tool and includes several business templates and corresponding data processing rules. The data processing rules include at least one of summarization, classification, format conversion, and calculation. The standardized data is processed according to the data processing rules corresponding to the target template to obtain normalized data that fits the target template. The normalized data is filled into the target template according to the field mapping relationship of the target template to generate the initial version of the result file; The initial version of the deliverables undergoes dual verification, including content compliance and privacy compliance, and is determined as the current deliverables if the verification is passed.

[0010] In one embodiment, the step of delivering the current result file to the target business system associated with the business instruction to execute the business operation corresponding to the business instruction includes: Determine the target business system associated with the business instruction, and obtain the real-time load status of the target business system; When the real-time load status meets the load threshold condition corresponding to the target business system, the interaction type of the target business system is obtained; When the interaction type of the target business system is a file access support type, the current result file is sent to the target business system so that the target business system can execute the business operation corresponding to the business instruction based on the current result file.

[0011] In one embodiment, after the step of obtaining the interaction type of the target business system, the method further includes: When the interaction type of the target business system is a form filling type, the normalized data corresponding to the current result file is filled into the target business form according to the field mapping relationship of the target business system, so that the target business system executes the business operation corresponding to the business instruction based on the filled target business form.

[0012] In one embodiment, after the step of sending the result file to the target business system, the method further includes: Receive the business operation result fed back by the target business system, and determine whether the business operation was successfully executed based on the business operation result; When the business operation is successfully executed, a business operation log is generated and pushed to the management user; When the business operation fails, the exception type corresponding to the business operation result is determined, and the self-healing strategy corresponding to the exception type is matched in the exception scenario library, so that the target business system re-executes the business operation corresponding to the business instruction based on the self-healing strategy.

[0013] In one embodiment, after determining the exception type corresponding to the business operation result and matching the self-healing strategy corresponding to the exception type in the exception scenario library, so that the target business system re-executes the business operation corresponding to the business instruction based on the self-healing strategy, the method further includes: Receive the exception handling results fed back by the target business system; The business operation results and the exception handling results are identified as historical processing results; Data processing optimization strategies are generated based on the historical processing results. These strategies include at least: a data collection optimization strategy, a privacy processing optimization strategy, and a template optimization strategy.

[0014] Furthermore, to achieve the above objectives, this application also proposes a multi-source data automated processing system, the system comprising: The intelligent acquisition module is used to collect raw data from heterogeneous data sources and to mark the raw data with privacy levels to obtain the marked raw data. The privacy processing module is used to match the corresponding privacy rule in the preset privacy rule library according to the privacy level, and to perform standardization processing on the marked original data based on the privacy rule to obtain standardized data; The content generation module is used to respond to triggered business instructions, match a target template from a preset business template library, and fill the target template with the standardized data to generate the current result file; The business execution module is used to deliver the current result file to the target business system associated with the business instruction to execute the business operation corresponding to the business instruction.

[0015] This application discloses an automated processing method for multi-source data, specifically disclosing the following steps: collecting raw data from heterogeneous data sources and marking the raw data with privacy levels to obtain marked raw data; matching corresponding privacy rules in a preset privacy rule library according to the privacy levels, and standardizing the marked raw data based on the privacy rules to obtain standardized data; responding to triggered business instructions, matching a target template from a preset business template library and filling the target template with standardized data to generate the current result file; and delivering the current result file to the target business system associated with the business instruction to execute the business operation corresponding to the business instruction.

[0016] Because this application can match differentiated processing rules based on data privacy levels, it can achieve targeted standardization and security processing of data with different levels of sensitivity, thereby improving data processing efficiency while effectively reducing the risk of data leakage. Furthermore, by automatically matching and filling standardized data with business templates, it generates deliverables that can be directly submitted to the target business system, eliminating redundant identification and manual review caused by unclear privacy attributes. This achieves a fully automated closed loop from data collection and processing to generation and execution, reducing manual intervention and improving the accuracy of business execution while shortening the deliverables delivery cycle. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the first embodiment of the multi-source data automated processing method of this application; Figure 2 This is a flowchart illustrating the second embodiment of the multi-source data automated processing method of this application; Figure 3 This is a flowchart illustrating the third embodiment of the multi-source data automated processing method of this application; Figure 4 This is a schematic diagram of the module structure of the multi-source data automated processing system of this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] This application provides an automated multi-source data processing method, referencing... Figure 1 , Figure 1This is a flowchart illustrating the first embodiment of the multi-source data automated processing method of this application. In this embodiment, the method includes steps S10 to S40: Step S10: Collect raw data from heterogeneous data sources and mark the raw data with privacy levels to obtain marked raw data.

[0024] It should be noted that the executing entity in this embodiment can be a computing electronic device with data processing, network communication, and program execution functions, such as a desktop computer, a mainframe computer, a server cluster, etc., or it can be other system servers that can access heterogeneous data sources and realize automated data processing. Here, the system server (referred to as "system") of a multi-source data automated processing system can be used as an example to describe the various embodiments of this application.

[0025] It should be understood that heterogeneous data sources can be data sources that differ in structure, format, storage method, and access protocol. Heterogeneous data sources can include: public or semi-public unstructured data sources, such as website pages, PDF documents, and scanned images, whose data does not have a fixed field format; internal structured data sources within an enterprise, such as databases of business systems like ERP (Enterprise Resource Planning) and CRM (Customer Relationship Management); and open data platforms provided by third parties.

[0026] The raw data can be unprocessed, un-standardized, un-cleaned, or un-encrypted data obtained from the aforementioned heterogeneous data sources. Its format can be HTML, JSON, XML, image text, database records, etc.

[0027] For different heterogeneous data sources, differentiated data acquisition methods can be used to obtain raw data. Therefore, the steps for acquiring raw data from heterogeneous data sources specifically include: steps S101~S103: Step S101: When the heterogeneous data source is a first type of data source, the page structure information of the first type of data source is parsed by a preset page structure parsing model, and a dynamic crawling strategy is generated according to the page structure information. Data is crawled in the first type of data source according to the dynamic crawling strategy to obtain the first raw data.

[0028] It should be noted that the first type of data source can be a data source that does not provide a standardized, structured data interface and whose main or only data entry and exit point is the human-computer interaction interface (such as a webpage or client software interface). For example, the aforementioned public or semi-public unstructured data sources, whose corresponding data can be found in their visual page structure elements (such as buttons, tables, text boxes, etc.).

[0029] The preset page structure parsing model is a pre-trained algorithmic model used to understand and analyze the layout and element composition of visualized pages. This model integrates AI visual recognition technology and can automatically identify the page structure and data location of target data sources (such as websites, client software interfaces, offline scanned documents, and other first-type data sources) to generate dynamic crawling strategies. Furthermore, when page elements (such as button positions and field names) of the target data source change, the model can automatically re-parse and adjust the crawling strategy.

[0030] It should also be noted that the system can be configured with a Robotic Process Automation (RPA) component, which can then be used to automatically collect raw data from the first type of data source.

[0031] In its implementation, the system can perform real-time analysis on the current page or screenshot of the target data source based on a preset page structure parsing model. It can extract page structure information such as the location, type, attributes, and interrelationships of all interactive and readable elements contained therein, and generate a specific data extraction instruction sequence, i.e., a dynamic crawling strategy. Then, the RPA component is used to extract the first raw data from the first data source according to the dynamic crawling strategy. The first raw data can be a text string, image binary data, etc.

[0032] Step S102: When the heterogeneous data source is a second type of data source, data is retrieved from the second type of data source through a preset application interface according to the pre-configured parameters to obtain the second original data.

[0033] It should be understood that the second type of data source can be a data source that provides standardized application programming interfaces for external systems to exchange and integrate data, such as the aforementioned internal structured data source of an enterprise or an open data platform provided by a third party.

[0034] The pre-defined application programming interface (API) can be a set of functions, protocols, or tools predefined by the second type of data source provider. This system can connect to the target data source (internal business systems, open data platforms, etc.) by calling the APIs provided by each second type of data source.

[0035] It should also be noted that before the system calls data via the API, it needs to pre-configure the interface parameters related to the call, such as address, authentication information, synchronization frequency, etc. The system can also mark the interface data type according to privacy level (high / medium / low); for high-privacy data, an "encrypted transmission + two-way authentication" mechanism is used to ensure the security of the data synchronization process; for low-privacy data, an efficient transmission mode is used to balance efficiency and security.

[0036] In its implementation, the system constructs a protocol request conforming to the API specification according to pre-configured parameters and sends it to the second type of data source. For highly private data, the call process employs enhanced security mechanisms such as encrypted transmission and two-way authentication. Finally, the initial data in structured format (such as JSON or XML), i.e., the second raw data, is parsed from the API response returned by the second type of data source.

[0037] Step S103: Determine the first raw data and the second raw data as the raw data.

[0038] In practice, the system can aggregate the raw data obtained from the two different acquisition paths mentioned above for subsequent unified processing of the raw data.

[0039] It should also be noted that privacy level labeling is the process of labeling the raw data based on the degree of personal or commercially sensitive information contained within it. The system can have a pre-set data privacy feature library (covering privacy features such as ID card numbers, bank card numbers, and trade secret keywords) that can automatically determine the privacy level of the collected raw data.

[0040] Furthermore, to specifically illustrate how to perform security classification on the raw data, the step of marking the raw data with privacy levels to obtain the marked raw data may further include: steps S104~S105: Step S104: Compare the original data with several privacy features in a preset data privacy database.

[0041] It should be noted that the preset data privacy database may include: regular expression patterns: used to identify sensitive information with fixed formats such as ID card numbers, mobile phone numbers, and bank card numbers; keyword or term list: such as words indicating commercial sensitivity, such as "top secret," "confidential," "business plan," and "balance sheet"; named entity recognition model: can identify names of people, organizations, places, dates, etc. in text and judge their sensitivity in combination with the context; industry-specific rules: for example, in the medical industry, disease names, gene sequences, etc. can be defined as high privacy features.

[0042] In a practical implementation, the system can use the aforementioned preset data privacy library to scan the fields and text in each data item in the original data and check whether its content matches any feature patterns defined in the library.

[0043] Step S105: Based on the comparison results, determine the data items with different privacy levels in the original data, and mark each data item with a privacy level according to each privacy level to obtain the marked original data.

[0044] It should be noted that a data item can be the smallest unit of data in the original data, which can be an entire record (such as a customer information row), a single field (such as an "ID number" column), or a part of a field.

[0045] It should be understood that, based on the type, quantity, and contextual importance of the aforementioned matched privacy features, the privacy level of each data item can be determined. The level determination logic can also be built into the aforementioned preset data privacy database.

[0046] For example, the judgment logic can be as follows: if a field contains features that match both ID card number and mobile phone number, it is determined to be a high privacy data item; if a field only contains the company's public email address (without specific features), it can be determined to be a low privacy data item; if the title of a document contains the keyword "internal audit report", the data items associated with the entire document can be determined to be medium privacy or high privacy.

[0047] In its implementation, the system can attach a privacy label (high privacy, medium privacy, and low privacy) to each data item that has been judged, thereby obtaining the original data after labeling.

[0048] Furthermore, the original data, after being marked, can be stored in different locations within a temporary data pool according to the privacy level of different data items. This temporary data pool can employ a partitioned encrypted storage design, where the original data can be stored in corresponding encrypted partitions according to their privacy levels, with keys managed hierarchically by the administrator, thereby ensuring the security of the original data.

[0049] Specifically, high-privacy data is marked and then directly encrypted and stored in an encrypted partition of a temporary data pool, while medium / low-privacy data is marked according to its level and stored in the corresponding partition, thereby achieving privacy isolation of the raw data during the collection phase.

[0050] Step S20: Match the corresponding privacy rule in the preset privacy rule library according to the privacy level, and perform standardization processing on the marked original data based on the privacy rule to obtain standardized data.

[0051] It should be understood that the preset privacy rule base can be a predefined set of privacy rules bound to privacy levels, including high privacy rules, medium privacy rules, and low privacy rules. Management users can customize different processing rules based on the system's visual interface.

[0052] It should be noted that the system can read data items of different privacy levels from the temporary data pool, and then retrieve one or more privacy rules corresponding to the privacy label from the preset privacy rule library.

[0053] In its implementation, the system can perform standardized processing on each data item in the original data based on corresponding privacy rules, including a series of operations such as cleaning, transformation, desensitization, and encryption, and then output standardized data with uniform format (such as specific JSON or database table structure), controllable quality, and compliance with security requirements.

[0054] For example, for different data items in the same data table in the original data, considering that the privacy level may differ, different privacy rules can be used to process different data items: the "name" field (low privacy) in the customer table can be formatted uniformly, while the ID number field (high privacy) can be de-identified and encrypted.

[0055] It should also be noted that the standardized data mentioned above can be stored in hierarchical partitions of the core database according to the privacy levels of different data items. The core database can be a built-in system database. Unlike the aforementioned temporary data pool, this core database can provide long-term storage for standardized data and can also partition the standardized data according to different privacy levels. Furthermore, the high-privacy partitions in the core database can support "double encryption + access permission verification" to ensure that only authorized users can decrypt and access the data.

[0056] Step S30: In response to the triggered business instruction, match the target template from the preset business template library, and fill the standardized data into the target template to generate the current result file.

[0057] It should be understood that this business instruction can be manually triggered by the user through the system interface (such as clicking the "Generate Report" button), or it can be automatically issued by the upstream system or a preset scheduled task. This business instruction may include specific business operation content, business-related data, and the business systems involved. In this embodiment, the business-related data is illustrated using all the standardized data obtained above as an example.

[0058] The preset business template library can include several business templates. These templates are predefined format frameworks used to generate final output documents (such as contracts, reports, statements, and forms). Furthermore, the format frameworks of similar output documents can be categorized by privacy level (high-privacy templates for internal confidential reports, and low-privacy templates for publicly disclosed documents). High-privacy templates can include built-in sensitive information hiding controls and encrypted watermarks, while medium / low-privacy templates can retain necessary information according to compliance requirements. Users can also add and modify the structure framework of templates through the system's template editing tool in the interactive interface and associate them with corresponding data fields in the core database.

[0059] Understandably, the system can parse business instructions to determine their corresponding business operation content and business-related data, and then read the corresponding data items from the aforementioned temporary data pool. First, it determines the output type of the target result file according to the business operation content, and then combines the privacy tags of the data items to retrieve the target business template from the business template library.

[0060] It should also be noted that if the business-related data includes data items with different privacy levels, the data item with the highest privacy level can be used as the benchmark, and the format framework that meets the highest privacy level can be selected as the target business template.

[0061] In practice, once the target business template is determined, the required standardized data can be automatically retrieved and extracted from the core database based on the preset field mapping relationship in the target template, filled into the corresponding position in the template, and then the current result file can be generated.

[0062] Step S40: Deliver the current result file to the target business system associated with the business instruction to execute the business operation corresponding to the business instruction.

[0063] It should be noted that the target business system can be a third-party system that is distinct from the current system and needs to receive and use the deliverables to complete specific business operations, or other systems within the enterprise.

[0064] In practical implementation, the system can use the aforementioned RPA component to automatically perform the login operation to the target business system and navigate to a specified function page (such as the "File Upload" page) within the target business system. The system can then select the current deliverable file for upload, thus submitting the file to the target business system. The target business system can then respond to the aforementioned business instructions by performing the corresponding business operations on the received deliverable file.

[0065] This embodiment can match differentiated processing rules based on data privacy levels, enabling targeted standardization and security processing of data with varying degrees of sensitivity. This improves data processing efficiency while effectively reducing the risk of data leakage. Furthermore, by automatically matching and filling standardized data with business templates, it generates deliverables that can be directly submitted to the target business system. This eliminates redundant identification and manual review caused by unclear privacy attributes, achieving a fully automated closed loop from data collection and processing to generation and execution. This reduces manual intervention, improves the accuracy of business execution, and shortens the deliverables delivery cycle.

[0066] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 , Figure 2This is a flowchart illustrating the second embodiment of the multi-source data automated processing method of this application.

[0067] In this embodiment, to specifically illustrate how to standardize raw data containing data items with different privacy levels, step S20 specifically includes: steps S201~S203: Step S201: Determine the privacy level corresponding to each data item, and query the privacy rules matching each privacy level in the preset privacy rule library. The preset privacy rule library is pre-configured by the management user and includes: high privacy data rules, medium privacy data rules and low privacy data rules.

[0068] It should be understood that the privacy rule base maintains different privacy rules according to privacy level, and privacy rules can be a series of standard operating procedures for data at that privacy level.

[0069] High privacy data rules may include: data conversion (unified to JSON structured format), AES-256 encryption, sensitive field desensitization (such as keeping only the first 6 digits and the last 4 digits of the ID number), and privacy compliance verification (verifying the integrity of encryption and the effectiveness of desensitization).

[0070] Privacy rules in China may include: format conversion, duplicate data removal, partial anonymization (such as blocking the middle 4 digits of a phone number), range verification, etc.

[0071] Low privacy data rules can include: format conversion, duplicate data removal, null value validation, summary statistics, etc.

[0072] In practice, the system can use the privacy level label of the data item as a query index to quickly retrieve and call the corresponding privacy rule from the rule base (which can be a single rule or a combination of multiple rules).

[0073] Step S202: Obtain the data processing optimization strategy, and update each of the privacy rules according to the data processing optimization strategy to obtain the updated privacy rules. The data processing optimization strategy is generated based on historical processing results.

[0074] It should be noted that data processing optimization strategies can be a set of instructions or parameters used to adjust and improve existing privacy rules, and can be generated based on the analysis of feedback data from historical processing, i.e., historical processing results.

[0075] The historical processing results can include data anomaly rates (such as the proportion of format errors after data masking), compliance verification pass rates, and processing time for specific rules during historical data processing. Furthermore, the system can pre-configure a strategy optimization model that takes historical data processing as input and outputs corresponding optimization strategies.

[0076] For example, if the system determines, based on historical processing results, that a certain desensitization method in the "high privacy data rules" causes frequent failures in subsequent verifications, it can generate a data processing optimization strategy that includes suggestions to "strengthen the desensitization rules" or "adjust the desensitization algorithm parameters".

[0077] It should be noted that the system can update each privacy processing rule according to the data processing optimization strategy, thereby automatically or semi-automatically fine-tuning the rule parameters or logic in the rule base to obtain the updated privacy rules.

[0078] Step S203: Standardize each data item according to the updated privacy rules to obtain standardized data.

[0079] In practice, the system performs standard operating procedures on each data item according to the updated privacy rules, thereby converting all data items into a unified, structured data model within the system, thus obtaining standardized data.

[0080] In addition, the system can store this standardized data in the system's built-in core database, and specifically store it in different privacy level partitions within the database according to the privacy level of different data items for subsequent retrieval.

[0081] Furthermore, to illustrate how to achieve intelligent matching between standardized data and compliance templates and automatically generate compliance outcome documents, we can use all the standardized data stored in the core database as an example. Therefore, step S30 specifically includes: steps S301~S305: Step S301: Upon receiving a business instruction, determine the current business scenario based on the business instruction.

[0082] It should be noted that this business instruction can be a command used to trigger the content generation process. It can be manually selected and clicked by the user on the front-end interface (such as "Generate 2023 Annual Audit Report"), or it can be an instruction automatically sent by a scheduled task or upstream process.

[0083] The business instruction may also include specific business operation content, business-related data, and the business systems involved. In this embodiment, the business-related data is the aforementioned standardized data.

[0084] In its implementation, the system can parse business instructions and identify the type of business to be executed. For example, keywords, selected menu items, or accompanying parameters in the instructions can be parsed into specific business scenarios such as "financial report generation," "contract drafting," and "regulatory report filing."

[0085] Step S302: Match the target template in the preset business template library according to the current business scenario. The preset business template library is pre-configured by the management user through the template editing tool.

[0086] It should be understood that the preset business template library stores different business templates. Each business template can define the style of the output document, such as font and layout, and can also define data placeholders and compliance rules. In addition, the preset business template library can also store the data processing rules corresponding to each business template. These data processing rules can include at least one of the following: summarization, classification, format conversion, and calculation, so that fields that meet the data placeholders can be extracted from the input data and filled into the business template.

[0087] It should be noted that the system also provides a template editing tool. Administrators can use the template editing tool provided by the system to create or modify business templates in a low-code manner, such as "drag and drop" or "form configuration", on the system's interactive interface, and establish a mapping relationship between specific locations in the business templates and standardized data fields in the core database.

[0088] It should also be noted that business templates can first be divided into different template sets corresponding to different business types according to business scenarios (business types), such as template sets corresponding to business types such as contracts, reports, statements, and forms; and then further divided according to privacy levels within different template sets, such as confidential report templates and public disclosure templates.

[0089] In practice, using the current business scenario as the query condition, a set of templates that match the business category can be matched first in the template library. Then, based on the highest privacy level among the aforementioned data items, the target template with the highest privacy level can be determined from this set of templates.

[0090] Step S303: Process the standardized data according to the data processing rules corresponding to the target template to obtain normalized data adapted to the target template.

[0091] Step S304: Fill the normalized data into the target template according to the field mapping relationship of the target template to generate the initial result file.

[0092] It should be understood that the field mapping relationship is the mapping relationship between data placeholders in the target template and standardized data fields.

[0093] In practice, the system can first perform further processing on the standardized data according to the above data processing rules, such as summarizing, classifying, format conversion and calculation, to obtain normalized data that fits the placeholder data requirements of each data placeholder in the target template; then fill each data item or data field in the normalized data into the corresponding position of the target template, and finally generate a preliminary, complete file, namely the first version of the result file.

[0094] Step S305: Perform dual verification on the initial version of the deliverables, including content compliance and privacy compliance, and determine the initial version of the deliverables as the current deliverables if the verification is passed.

[0095] It should be noted that the system can also have a built-in dynamic compliance rule base, which can be synchronized in real time with the latest industry data compliance requirements on the network. This rule base can include several check items used to perform content compliance verification and privacy compliance verification on the aforementioned initial version of the deliverables.

[0096] Content compliance verification can be used to check whether the content of the generated file itself conforms to business logic and format requirements. Corresponding check items may include: whether the data format is correct (e.g., whether the date is YYYY-MM-DD), whether the required fields are complete, whether the values ​​are within a reasonable range, and whether the calculation logic (e.g., whether the sum of the items equals the total) is correct, etc.

[0097] Privacy compliance verification can be used to check whether the processing and final state of generated files meet data privacy protection requirements. Corresponding check items may include: the matching degree of privacy levels between data and templates, such as checking whether high-privacy data was mistakenly filled into a low-privacy public template; verification of the effectiveness of sensitive information masking, used to check whether sensitive information that should be masked (such as ID numbers) is correctly masked in the output file; and encryption and watermark checks, such as checking whether high-privacy files have been password protected or digitally watermarked according to rules.

[0098] In its implementation, the system performs dual compliance verification on the initial deliverables based on each check item in the dynamic compliance rule base. Only when the initial compliance verification passes all check items is the deliverable determined to be the current deliverable that can be delivered and used.

[0099] Furthermore, the current outcome document can also be stored in the aforementioned core database and associated with its corresponding standardized data.

[0100] This embodiment achieves precise adaptation and continuous self-optimization of data processing strategies by matching and dynamically updating a pre-configured rule base categorized by privacy level for tagged data items. This ensures high privacy data security while improving the adaptability and standardization of the overall processing flow. By matching pre-set templates according to business scenarios and automatically filling standardized data based on field mapping relationships, and then performing dual compliance checks on the generated initial file for both content and privacy, the output file is generated quickly and accurately. This effectively ensures the reliability of the output content in terms of business specifications and data security, reducing manual verification steps and compliance risks.

[0101] Based on the first and second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to that in embodiments one and two above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 , Figure 3 This is a flowchart illustrating the third embodiment of the multi-source data automated processing method of this application.

[0102] In this embodiment, to specifically illustrate how to achieve cross-system automated execution of the output documents, step S40 specifically includes: steps S401~S403': Step S401: Determine the target business system associated with the business instruction, and obtain the real-time load status of the target business system.

[0103] It should be understood that the system can determine the system object to be interacted with by parsing the parameters and context in the business instruction. For example, if the business instruction is "submit the annual report to the China Securities Regulatory Commission (CSRC) system", then the target business system can be the "CSRC online filing platform".

[0104] It should be noted that before the delivery operation is executed, the system can first probe or query the current operating pressure or resource usage of the target business system: for example, by querying the monitoring indicators published by the target business system through API (such as CPU utilization, memory utilization, current concurrent requests); or by testing the response latency and success rate of the target business system through simple probe requests (such as Ping, accessing a lightweight interface).

[0105] Step S402: When the real-time load status meets the load threshold condition corresponding to the target business system, obtain the interaction type of the target business system.

[0106] It should be understood that load threshold conditions can be pre-configured by the management user for the target business system to determine whether it is in an acceptable workload state, such as "when the CPU utilization of the target business system is below 80% and the average response time is less than 2 seconds". In addition, corresponding process triggering conditions can be set based on the management user's needs for using the deliverables, such as "trigger immediately after the current deliverable is generated".

[0107] It should be noted that the interaction type can include file access support classes and form filling classes.

[0108] The file access support category is for target business systems that provide standardized file upload interfaces and can accept files that meet specific format requirements (e.g., from external systems) to complete business operations. The form entry category is for target business systems that do not provide batch file upload interfaces and require users to fill out online forms item by item through an interactive interface to complete business operations.

[0109] Step S403: When the interaction type of the target business system is the file access support type, the current result file is sent to the target business system so that the target business system executes the business operation corresponding to the business instruction based on the current result file.

[0110] In practical implementation, when the process triggering conditions and load threshold conditions are met, the system can deliver the previously generated current result file to the target business system by directly calling the access API provided by the target business system or by using an RPA component to simulate uploading. After receiving the file, the target system will automatically trigger its corresponding subsequent processes based on business instructions, such as parsing the file, updating the status, and generating receipts.

[0111] Step S403': When the interaction type of the target business system is form filling, the normalized data corresponding to the current result file is filled into the target business form according to the field mapping relationship of the target business system, so that the target business system executes the business operation corresponding to the business instruction based on the filled target business form.

[0112] It should be noted that for target business systems that require form filling, since the aforementioned generated output files cannot be used directly, the system can first obtain the normalized data on which the aforementioned generated output files depend to fill in the target business forms within the target business system.

[0113] In its implementation, the system first obtains the mapping relationships of each field in the target business form within the target business system. Then, by calling an RPA component, it fills the normalized data into the target business form according to these field mapping relationships, thus completing the entire form. After the form is filled out, the RPA component simulates clicking the "Submit" or "Confirm" button, generating the filled target business form in the target business system. Ultimately, this triggers the target business system to execute the corresponding business operation based on the business instructions.

[0114] Furthermore, to achieve data closure and ensure process stability and traceability, after step S402, the following steps are also included: steps S501~S503: Step S501: Receive the business operation result fed back by the target business system, and determine whether the business operation was successfully executed based on the business operation result.

[0115] It should be noted that the system can continuously monitor the delivery process of the aforementioned deliverables and the execution process of business operations in the target business system based on RPA components, thereby obtaining feedback results from the target business system. Specifically, the system can receive script feedback from the RPA execution unit (such as "Clicking the submit button was successful, and the page redirected to the confirmation page"), parse the return page from the target system (such as detecting the words "Operation successful"), or obtain the return code based on the API of the target business system.

[0116] It should be understood that the system can pre-maintain the expected results for different types of business instructions (business operations). Then, the results of the business operations can be compared with the expected results to determine whether the aforementioned business operation was executed successfully.

[0117] Step S502: When the business operation is successfully executed, generate a business operation log and push it to the management user.

[0118] It should be understood that the system can create structured records for successfully executed business operations. These records can include at least: the operator (system account or triggerer), the operation time, the content of the business operation (e.g., "upload file A to system B"), and the identifiers of the data / files involved. This forms a complete audit log.

[0119] Furthermore, the system can store the generated audit logs in the aforementioned core database and notify relevant management personnel through messages, dashboards, or reports.

[0120] Step S503: When the business operation fails, determine the exception type corresponding to the business operation result, and match the self-healing strategy corresponding to the exception type in the exception scenario library, so that the target business system re-executes the business operation corresponding to the business instruction based on the self-healing strategy.

[0121] It should be noted that the system can determine the exception type of business operation failure based on the business operation result, and then call the pre-configured self-healing strategy from the exception scenario library.

[0122] It is understandable that the system can also pre-configure an exception scenario library. The self-healing strategies in this exception scenario library can include, for example, the following: when the exception type is network timeout, the self-healing strategy can be: wait for 30 seconds and then automatically retry, with a maximum of 3 retries; when the exception type is login failure, the self-healing strategy can be: verify the validity of the account and password, and if valid, log in again; if invalid, issue an alarm; when the exception type is page element mismatch, the self-healing strategy can be: trigger the page structure to be re-parsed, update the element positioning information, and then re-execute.

[0123] In practical implementation, when a business operation fails, the system can automatically execute the matched self-healing strategy based on the exception type and re-initiate the entire or part of the business operation process based on the RPA component.

[0124] Furthermore, considering that the re-initiated business operation process may also fail, after step S503, the process also includes: steps S601~S603: Step S601: Receive the exception handling result fed back by the target business system.

[0125] It should be noted that after the self-healing strategy is executed and the business operation process is re-initiated, the system can still receive the exception handling result from the target business system. This exception handling result can be the result of whether the business operation was successful or failed.

[0126] Step S602: Determine the business operation result and the exception handling result as historical processing results.

[0127] In its implementation, the system can aggregate the results of all original business operations and the results of subsequent re-initiated and responded business operations as the execution result of the current business instruction, and mark them as historical processing results to be stored in the core database.

[0128] Step S603: Generate a data processing optimization strategy based on the historical processing results. The data processing optimization strategy includes at least: a data collection optimization strategy, a privacy processing optimization strategy, and a template optimization strategy.

[0129] It should be noted that data processing optimization strategies can be specific suggestions or instructions used to guide the adjustment of parameters or rules at each stage of the system's business operation process, such as: data collection optimization strategies, privacy processing optimization strategies, and template optimization strategies.

[0130] For example, the data collection optimization strategy can be: if historical processing results show that the data crawling failure rate of a certain website is abnormally high, a strategy suggestion can be generated to increase the page structure parsing frequency of the data source or adjust the crawling waiting time during the aforementioned data collection stage.

[0131] Privacy processing optimization strategies could include: if historical processing results show that some highly private data still frequently reports errors during compliance verification (e.g., the desensitized format is not accepted), then strategy recommendations could be generated to strengthen the desensitization rules for such data or update the privacy feature library during the aforementioned standardized processing stage based on privacy rules.

[0132] Template optimization strategies can be as follows: If a report generated using a certain business template is frequently rejected when uploaded to the target business system due to incompatible field formats based on historical processing results, it can be suggested to modify the mapping rules or format requirements of specific fields in the target template during the successful file generation stage.

[0133] This embodiment effectively avoids execution failures caused by system congestion by monitoring the load status of the target business system in real time and intelligently scheduling delivery times based on preset threshold conditions, thereby improving the success rate and overall stability of the automated process. After delivery, monitoring the business operation results and automatically matching and executing pre-set self-healing strategies for execution failures enables automated handling of execution anomalies and self-recovery of the process, reducing reliance on manual intervention. Furthermore, by collecting and analyzing historical processing results, including anomaly handling results, a full-process optimization strategy for data collection, privacy processing, and content templates is generated, driving adaptive iteration in each stage of the system to continuously improve the accuracy, security, and compliance of data processing and business execution.

[0134] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the multi-source data automated processing method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0135] This application also provides a multi-source data automated processing system; please refer to [link / reference]. Figure 4 , Figure 4 This is a schematic diagram of the module structure of the multi-source data automated processing system of this application. The system includes: The intelligent acquisition module 401 is used to acquire raw data from heterogeneous data sources and to mark the raw data with a privacy level to obtain the marked raw data. The privacy processing module 402 is used to match the corresponding privacy rule in the preset privacy rule library according to the privacy level, and to perform standardization processing on the marked original data based on the privacy rule to obtain standardized data; Content generation module 403 is used to respond to triggered business instructions, match target templates from a preset business template library, and fill the standardized data into the target templates to generate the current result file; The business execution module 404 is used to deliver the current result file to the target business system associated with the business instruction to execute the business operation corresponding to the business instruction.

[0136] This embodiment can match differentiated processing rules based on data privacy levels, enabling targeted standardization and security processing of data with varying degrees of sensitivity. This improves data processing efficiency while effectively reducing the risk of data leakage. Furthermore, by automatically matching and filling standardized data with business templates, it generates deliverables that can be directly submitted to the target business system. This eliminates redundant identification and manual review caused by unclear privacy attributes, achieving a fully automated closed loop from data collection and processing to generation and execution. This reduces manual intervention, improves the accuracy of business execution, and shortens the deliverables delivery cycle.

[0137] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other elements in the process, method, article, or system that includes that element.

[0138] The above embodiment numbers are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. They are only some embodiments of this application and do not limit the scope of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of this application.

Claims

1. A method for automated processing of multi-source data, characterized in that, The method includes: Raw data from heterogeneous data sources is collected, and the raw data is marked with a privacy level to obtain the marked raw data; Based on the privacy level, the corresponding privacy rule is matched in the preset privacy rule library, and the original marked data is standardized based on the privacy rule to obtain standardized data; In response to a triggered business instruction, a target template is matched from a preset business template library, and the standardized data is filled into the target template to generate the current result file; The current result file is delivered to the target business system associated with the business instruction to execute the business operation corresponding to the business instruction.

2. The method as described in claim 1, characterized in that, The steps for collecting raw data from heterogeneous data sources include: When the heterogeneous data source is a first type of data source, the page structure information of the first type of data source is parsed by a preset page structure parsing model, and a dynamic crawling strategy is generated according to the page structure information. Data is crawled in the first type of data source according to the dynamic crawling strategy to obtain the first raw data. When the heterogeneous data source is a second type of data source, data is retrieved from the second type of data source through a preset application interface according to pre-configured parameters to obtain the second original data; The first raw data and the second raw data are identified as the raw data.

3. The method as described in claim 2, characterized in that, The step of marking the original data with a privacy level to obtain the marked original data includes: The original data is compared with several privacy features in a preset data privacy feature library; Based on the comparison results, data items with different privacy levels are identified in the original data, and each data item is marked with a privacy level according to its privacy level to obtain the marked original data.

4. The method as described in claim 1, characterized in that, The marked raw data includes several data items corresponding to different privacy levels; The step of matching the corresponding privacy rule in a preset privacy rule base according to the privacy level, and standardizing the tagged original data based on the privacy rule to obtain standardized data includes: Determine the privacy level corresponding to each data item, and query the privacy rules matching each privacy level in the preset privacy rule library. The preset privacy rule library is pre-configured by the management user and includes: high privacy data rules, medium privacy data rules and low privacy data rules. A data processing optimization strategy is obtained, and each privacy rule is updated according to the data processing optimization strategy to obtain the updated privacy rules. The data processing optimization strategy is generated based on historical processing results. Each data item is standardized according to the updated privacy rules to obtain standardized data.

5. The method as described in claim 1, characterized in that, The step of responding to a triggered business instruction, matching a target template from a preset business template library, filling the target template with the standardized data, and generating the current result file includes: Upon receiving a business instruction, the current business scenario is determined based on the business instruction; Based on the current business scenario, a target template is matched in the preset business template library. The preset business template library is pre-configured by the management user through the template editing tool and includes several business templates and corresponding data processing rules. The data processing rules include at least one of summarization, classification, format conversion, and calculation. The standardized data is processed according to the data processing rules corresponding to the target template to obtain normalized data that fits the target template. The normalized data is filled into the target template according to the field mapping relationship of the target template to generate the initial version of the result file; The initial version of the deliverables undergoes dual verification, including content compliance and privacy compliance, and is determined as the current deliverables if the verification is passed.

6. The method as described in claim 5, characterized in that, The step of delivering the current result file to the target business system associated with the business instruction to execute the business operation corresponding to the business instruction includes: Determine the target business system associated with the business instruction, and obtain the real-time load status of the target business system; When the real-time load status meets the load threshold condition corresponding to the target business system, the interaction type of the target business system is obtained; When the interaction type of the target business system is a file access support type, the current result file is sent to the target business system so that the target business system can execute the business operation corresponding to the business instruction based on the current result file.

7. The method as described in claim 6, characterized in that, After the step of obtaining the interaction type of the target business system, the method further includes: When the interaction type of the target business system is a form filling type, the normalized data corresponding to the current result file is filled into the target business form according to the field mapping relationship of the target business system, so that the target business system executes the business operation corresponding to the business instruction based on the filled target business form.

8. The method as described in claim 6, characterized in that, After the step of sending the result file to the target business system, the method further includes: Receive the business operation result fed back by the target business system, and determine whether the business operation was successfully executed based on the business operation result; When the business operation is successfully executed, a business operation log is generated and pushed to the management user; When the business operation fails, the exception type corresponding to the business operation result is determined, and the self-healing strategy corresponding to the exception type is matched in the exception scenario library, so that the target business system re-executes the business operation corresponding to the business instruction based on the self-healing strategy.

9. The method as described in claim 8, characterized in that, After the step of determining the exception type corresponding to the business operation result and matching the self-healing strategy corresponding to the exception type in the exception scenario library, so that the target business system re-executes the business operation corresponding to the business instruction based on the self-healing strategy, the method further includes: Receive the exception handling results fed back by the target business system; The business operation results and the exception handling results are identified as historical processing results; Data processing optimization strategies are generated based on the historical processing results. These strategies include at least: a data collection optimization strategy, a privacy processing optimization strategy, and a template optimization strategy.

10. A multi-source data automated processing system, characterized in that, The system includes: The intelligent acquisition module is used to collect raw data from heterogeneous data sources and to mark the raw data with privacy levels to obtain the marked raw data. The privacy processing module is used to match the corresponding privacy rule in the preset privacy rule library according to the privacy level, and to perform standardization processing on the marked original data based on the privacy rule to obtain standardized data; The content generation module is used to respond to triggered business instructions, match a target template from a preset business template library, and fill the target template with the standardized data to generate the current result file; The business execution module is used to deliver the current result file to the target business system associated with the business instruction to execute the business operation corresponding to the business instruction.

Images