Data processing method and system for custom forms

Abstract

The present application discloses a data processing method and system for custom forms, the method comprising: receiving custom forms and form data from different product lines, the form data comprising at least one key-value pair; identifying form fields of the custom form, the form fields are obtained by dynamic parsing using a pre-trained dynamic parsing algorithm model, the pre-trained dynamic parsing algorithm model is a neural network for automatically identifying and parsing key fields in the custom form; using each identified form field as a key, extracting the value corresponding to the key from the key-value pair, and obtaining matching data of the custom form; converting the matching data of the custom form into structured data according to a preset data conversion strategy; storing the structured data in a ClickHouse database, and visually displaying the stored structured data. Therefore, by adopting the embodiment of the present application, human resources can be saved, and the data processing efficiency of the custom form can be accelerated.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method and system for custom forms. Background Art

[0002] In modern enterprise operations, especially in cross-line business environments, the demand for data collection and processing is growing. To meet these needs, enterprises have developed custom form functions that allow personalized form items to be configured according to the specific needs of different user groups. The data collected by these forms is essential for understanding customer needs, optimizing service processes, and improving customer satisfaction.

[0003] In the prior art, custom form functions usually involve creating semi-structured data storage methods to meet the diverse data collection requirements of different forms. When a new form is created, the system administrator needs to design a new database table structure to store the data of the form and write or modify the corresponding data parsing code to ensure that the data can be correctly parsed and stored, resulting in a waste of human resources; at the same time, as the number of custom forms increases, the storage method of semi-structured data makes data parsing and analysis more complicated, affecting the efficiency of data processing. Summary of the invention

[0004] The embodiments of the present application provide a data processing method and system for custom forms. In order to have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a general review, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0005] In a first aspect, an embodiment of the present application provides a data processing method for a custom form, the method comprising:

[0006] Receive custom forms and form data from different product lines, where the form data includes at least one key-value pair;

[0007] Identify form fields of a custom form. Form fields are dynamically parsed using a pre-trained dynamic parsing algorithm model. The pre-trained dynamic parsing algorithm model is a neural network used to automatically identify and parse key fields in a custom form.

[0008] Use each identified form field as a key, extract the value corresponding to the key from the key-value pair, and obtain the matching data of the custom form;

[0009] According to the preset data conversion strategy, the matching data of the custom form is converted into structured data;

[0010] Store structured data in the ClickHouse database and visualize the stored structured data.

[0011] Optionally, visualize the stored structured data, including:

[0012] Read structured data from the ClickHouse database;

[0013] Analyze the data feature points of structured data;

[0014] According to the data feature points, select the best chart type from the chart type library;

[0015] According to the optimal chart type, the structured data is processed to obtain a chart;

[0016] Send the chart to the client for visual display.

[0017] Optionally, identify the form fields for the custom form, including:

[0018] Load the pre-trained dynamic parsing algorithm model;

[0019] When the custom form is in image format or PDF format, optical character recognition technology is used to convert the custom form into a readable text format to obtain a form description text;

[0020] Input the form description text into the pre-trained dynamic parsing algorithm model to dynamically parse the form description text;

[0021] Output the field name sequence corresponding to the form description text;

[0022] Takes a sequence of field names to identify the form fields.

[0023] Optionally, the pre-trained dynamic parsing algorithm model includes a parameter extraction layer, a feature extraction layer, a feature parsing layer, a field matching layer, and a key field determination layer;

[0024] Dynamically parse the form description text, including:

[0025] The parameter extraction layer extracts phrases, titles, and tags from the form description text to obtain unstructured data;

[0026] The feature extraction layer extracts the features of unstructured data to obtain form feature information;

[0027] The feature parsing layer performs parsing based on form feature information to identify all fields in the custom form;

[0028] The field matching layer matches all fields with predefined field templates to obtain multiple matching fields;

[0029] The key field determination layer uses a classifier to classify multiple matching fields to identify key fields of preset key types existing in the multiple matching fields, and obtain a field name sequence corresponding to the form description text.

[0030] Optionally, follow these steps to generate a pre-trained dynamic parsing algorithm model, including:

[0031] Collect custom form samples covering various business scenarios and including different formats;

[0032] Preprocess custom form samples in different formats to obtain historical form description text. The preprocessing includes image conversion based on optical character recognition technology, text data cleaning and standardization.

[0033] Create a parameter extraction layer for extracting phrases, titles, and tags; create a feature extraction layer for extracting form feature information; create a feature parsing layer using natural language processing technology; create a field matching layer for matching identified fields with predefined field templates; use a random forest neural network to create a classifier as a key field determination layer; integrate the parameter extraction layer, feature extraction layer, feature parsing layer, field matching layer, and key field determination layer into a dynamic parsing algorithm model;

[0034] Generate model training samples based on historical form description text;

[0035] Input the model training samples into the dynamic parsing algorithm model for machine learning and output the model loss value;

[0036] When the model loss value reaches the minimum, a pre-trained dynamic parsing algorithm model is generated.

[0037] Optionally, generate model training samples based on historical form description text, including:

[0038] The parameter extraction layer is used to obtain phrases, titles, and tags from the historical form description text to obtain historical unstructured data;

[0039] In response to a data labeling instruction for unstructured data, obtaining historical field name labels for historical unstructured data labeling, where the historical field name labels include positive samples and negative samples;

[0040] Map and associate historical unstructured data and historical field name labels to obtain associated unstructured data and field name labels as model training samples;

[0041] Input the model training samples into the dynamic parsing algorithm model for machine learning and output the model loss value, including:

[0042] The associated unstructured data and field name sequence are sequentially input into the feature extraction layer, feature parsing layer, field matching layer, and key field determination layer for machine learning, and the model loss value is output.

[0043] Optionally, the loss function of the classifier is:

[0044]

[0045] in, is the model loss value, is the predicted probability of the classifier for the positive sample, is the weight to balance positive and negative samples, is the focus parameter, When the value of is greater than 0, it is used to reduce the attention paid to samples with high prediction probability of the classifier.

[0046] Optionally, the preset data conversion strategy includes a data preprocessing module and a field mapping module, and the matching data of the custom form is the value corresponding to each identified form field;

[0047] According to the preset data conversion strategy, the matching data of the custom form is converted into structured data, including:

[0048] The data preprocessing module is used to remove noise, unify the format, and process missing values ​​for the values ​​corresponding to each identified form field, so as to obtain the final value corresponding to each identified form field;

[0049] Using the field mapping module, obtain the standard fields corresponding to each identified form field from the standard fields in the database;

[0050] Acquire the data format of the standard field corresponding to each identified form field as the target data format corresponding to each identified form field;

[0051] According to the target data format corresponding to each identified form field, the final value corresponding to each identified form field is format-converted to obtain structured data corresponding to each identified form field;

[0052] The structured data corresponding to each identified form field is used as the structured data of the custom form.

[0053] Optionally, obtain the standard fields corresponding to each identified form field from the standard fields of the database, including:

[0054] From the standard fields of the database, obtain the first field name, the first field type, and the first value range of each standard field;

[0055] Obtain the second field name and the second field type of each identified form field;

[0056] Obtaining a second value range corresponding to each identified form field from the final values ​​corresponding to each identified form field;

[0057] Calculating the name similarity between each standard field and each identified form field according to the first field name of each standard field and the second field name of each identified form field;

[0058] Calculating type similarity between each standard field and each identified form field according to a first field type of each standard field and a second field type of each identified form field;

[0059] Calculating the similarity between the value ranges of each standard field and each identified form field based on the first value range of each standard field and the second value range of each identified form field;

[0060] Based on name similarity, type similarity, and value range similarity, calculate the comprehensive similarity between each identified form field and each standard field;

[0061] From the comprehensive similarities between each identified form field and each standard field, the standard field corresponding to the maximum comprehensive similarity is selected as the standard field corresponding to each identified form field.

[0062] In a second aspect, an embodiment of the present application provides a data processing system for a custom form, the system comprising:

[0063] A data receiving module, used for receiving custom forms and form data from different product lines, wherein the form data includes at least one key-value pair;

[0064] A field recognition module is used to recognize form fields of a custom form. The form fields are obtained by dynamic parsing using a pre-trained dynamic parsing algorithm model. The pre-trained dynamic parsing algorithm model is a neural network used to automatically recognize and parse key fields in a custom form.

[0065] The form data determination module is used to use each identified form field as a key, extract the value corresponding to the key from the key-value pair, and obtain the matching data of the custom form;

[0066] The form data conversion module is used to convert the matching data of the custom form into structured data according to the preset data conversion strategy;

[0067] The visualization module is used to store structured data in the ClickHouse database and visualize the stored structured data.

[0068] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0069] In the embodiment of the present application, on the one hand, the form fields of the custom form can be identified by using a pre-trained dynamic parsing algorithm model, which can adapt to custom forms of different formats, making the system more flexible and able to handle diversified data inputs, thereby reducing manual intervention and improving the efficiency and accuracy of data processing. On the other hand, through the preset data conversion strategy, the matching data is converted into structured data to ensure the consistency and standardization of the data. At the same time, the structured data is stored in the ClickHouse database, which is a high-performance database designed for online analytical processing, so that the data can support real-time query and analysis. Through real-time data analysis and intuitive visual display, decision makers can gain insights faster and make more accurate business decisions.

[0070] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0072] Figure 1 It is a flowchart of a data processing method for a custom form provided in an embodiment of the present application;

[0073] Figure 2 It is a data representation intent of a key-value pair provided by this application;

[0074] Figure 3 It is a structural diagram of a pre-trained dynamic parsing algorithm model provided by the present application;

[0075] Figure 4 It is a data representation intent of matching data provided by this application;

[0076] Figure 5 It is a schematic diagram of a visual interface provided by this application;

[0077] Figure 6 It is a structural diagram of a data processing system for custom forms provided in an embodiment of the present application;

[0078] Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] The following description and the accompanying drawings sufficiently illustrate specific embodiments of the present application to enable those skilled in the art to practice them.

[0080] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0081] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of systems and methods consistent with some aspects of the present application as detailed in the attached claims.

[0082] In the description of the present application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0083] The present application provides a data processing method and system for custom forms to solve the problems existing in the above-mentioned related technical issues. In the embodiment of the present application, on the one hand, the form fields of the custom form can be identified by using a pre-trained dynamic parsing algorithm model. The dynamic parsing algorithm model can adapt to custom forms of different formats, making the system more flexible and able to handle diversified data inputs, thereby reducing manual intervention and improving the efficiency and accuracy of data processing. On the other hand, through a preset data conversion strategy, the matching data is converted into structured data to ensure the consistency and standardization of the data. At the same time, the structured data is stored in the ClickHouse database. The ClickHouse database is a high-performance database designed for online analytical processing, so that the data can support real-time query and analysis. Through real-time data analysis and intuitive visual display, decision makers can gain insights faster and make more accurate business decisions. The following exemplary embodiments are described in detail.

[0084] The following will be combined with the attached Figure 1 -Attached Figure 5, a data processing method for a custom form provided in an embodiment of the present application is introduced in detail. The method can be implemented by a computer program and can be run on a data processing system for a custom form based on the von Neumann system. The computer program can be integrated into an application or run as an independent tool application.

[0085] See also Figure 1 , which is a flow chart of a data processing method for a custom form according to an embodiment of the present application. Figure 1 As shown, the method of the embodiment of the present application may include the following steps:

[0086] S101, receiving custom forms and form data from different product lines, where the form data includes at least one key-value pair;

[0087] Among them, product line refers to a series of related products developed by an enterprise for different market needs or different customer groups. Different product lines involve different business areas. Custom forms refer to forms designed and created according to specific business needs. These forms usually contain specific fields for collecting specific data. Form data refers to the information entered by the user when filling out the form, which is usually used for subsequent data processing and analysis. A key-value pair is a data structure in which the "key" is a unique identifier and the "value" is the data associated with the key. In the form data of this application, the key corresponds to the name of the form field, and the value is the specific information entered by the user.

[0088] In some embodiments, such as multiple product lines of a manufacturing company, each product line has its own customer feedback form. These forms are designed as custom forms to collect product-specific data. For example, a form for the cosmetics department may contain fields such as "Product Satisfaction" and "Frequency of Use", while a form for the electronics department may contain fields such as "Technical Issues" and "Product Performance". When customers fill out and submit these forms, the system collects form data, where each field (key) and the corresponding user input (value) form a key-value pair. For example, a form for the cosmetics department may generate the following key-value pairs: {"Product Satisfaction": "Very Satisfied"} and {"Frequency of Use": "Every Day"}. These key-value pairs are then used for data analysis and customer service improvements. Among them, the data table of key-value pairs is, for example, Figure 2 shown.

[0089] S102, identifying form fields of the custom form, where the form fields are obtained by dynamically parsing using a pre-trained dynamic parsing algorithm model, where the pre-trained dynamic parsing algorithm model is a neural network for automatically identifying and parsing key fields in the custom form;

[0090] Among them, form fields refer to the input areas or item identifiers used to collect specific information in a form. Neural networks are computational models that mimic the structure and function of human brain neuron networks and are used to process complex data patterns and relationships.

[0091] In some embodiments of the present application, the specific process of identifying the form fields of a custom form includes: loading a pre-trained dynamic parsing algorithm model; when the custom form is in image format or PDF format, using optical character recognition technology to convert the custom form into a readable text format to obtain a form description text; inputting the form description text into a pre-trained dynamic parsing algorithm model to dynamically parse the form description text; outputting a field name sequence corresponding to the form description text; and using the field name sequence as each identified form field.

[0092] Among them, the pre-trained dynamic parsing algorithm model is a model pre-trained using machine learning technology, which can automatically identify and parse fields in the form. This model is usually based on a neural network and can handle forms of different formats and layouts. Optical character recognition technology (OCR) can extract text information from images or PDF files and convert text in scanned documents or pictures into machine-readable text data. Form description text refers to the text information obtained after OCR processing, which describes the content of the form, including the field names in the form and the data filled in by the user. The field name sequence refers to an ordered list of all field names in the form, which are usually the names of the form fields recognized by the dynamic parsing algorithm model.

[0093] In the embodiment of the present application, the pre-trained dynamic parsing algorithm model and OCR technology are used to process custom forms, which can realize automatic and efficient form data extraction and parsing. This method not only improves the speed and accuracy of data processing, reduces manual intervention and potential errors, but also enhances the flexibility and adaptability of the system, enabling it to process forms of various formats and layouts.

[0094] For example, a large bank needs to process loan application forms from different branches, which may be submitted in image or PDF format with different formats and layouts. When branches submit images or PDF files of loan application forms, the bank uses OCR technology to convert these files into readable text format to obtain form description text. The OCR-processed form description text is input into the pre-trained dynamic parsing algorithm model, and the model automatically identifies the fields in the form. The model outputs a sequence of recognized field names, such as ["Applicant Name", "ID Number", "Loan Amount"].

[0095] Among them, for example Figure 3As shown, the pre-trained dynamic parsing algorithm model includes a parameter extraction layer, a feature extraction layer, a feature parsing layer, a field matching layer, and a key field determination layer.

[0096] In some embodiments of the present application, the specific process of dynamically parsing the form description text includes: the parameter extraction layer extracts phrases, titles, and tags from the form description text to obtain unstructured data; the feature extraction layer extracts features of the unstructured data to obtain form feature information; the feature parsing layer performs parsing based on the form feature information to identify all fields in the custom form; the field matching layer matches all fields with predefined field templates to obtain multiple matching fields; the key field determination layer uses a classifier to classify multiple matching fields to identify key fields of preset key types existing in multiple matching fields, and obtains a field name sequence corresponding to the form description text.

[0097] Among them, the predefined field template refers to the field format and type pre-set in the system, which is used to guide the field matching process and ensure that the extracted data conforms to the expected structure. The classifier is used to identify and classify fields to determine which fields are key fields.

[0098] In some embodiments of the present application, the specific process of generating a pre-trained dynamic parsing algorithm model includes: collecting custom form samples covering multiple preset business scenarios and including different formats; pre-processing the custom form samples in different formats to obtain historical form description text, the preprocessing includes image conversion based on optical character recognition technology, text data cleaning and standardization; creating a parameter extraction layer for extracting phrases, titles and labels; creating a feature extraction layer for extracting form feature information; using natural language processing technology to create a feature parsing layer; creating a field matching layer for matching identified fields with predefined field templates; using a random forest neural network to create a classifier as a key field determination layer; integrating the parameter extraction layer, feature extraction layer, feature parsing layer, field matching layer and key field determination layer into a dynamic parsing algorithm model; generating model training samples based on the historical form description text; inputting the model training samples into the dynamic parsing algorithm model for machine learning, and outputting the model loss value; when the model loss value reaches the minimum, generating a pre-trained dynamic parsing algorithm model.

[0099] Among them, custom form samples refer to forms collected from different business scenarios, which may contain multiple data formats, such as images, PDF, etc.

[0100] In the embodiment of the present application, by collecting and preprocessing a variety of custom form samples, combined with advanced natural language processing technology and machine learning algorithms, a dynamic parsing algorithm model that can automatically identify and parse key fields is created. This model can significantly improve the automation of form data processing, reduce manual intervention, improve the efficiency and accuracy of data processing, and provide enterprises with powerful data-driven decision support. Through continuous machine learning training and model optimization, the model can adapt to changing business needs and form formats to ensure long-term business adaptability.

[0101] In some embodiments of the present application, the specific process of generating model training samples based on historical form description text includes: using a parameter extraction layer to obtain phrases, titles, and labels from the historical form description text to obtain historical unstructured data; in response to data labeling instructions for unstructured data, obtaining historical field name labels for the historical unstructured data, the historical field name labels include positive samples and negative samples; mapping and associating the historical unstructured data and the historical field name labels to obtain associated unstructured data and field name labels as model training samples.

[0102] In some embodiments of the present application, the model training samples are input into the dynamic parsing algorithm model for machine learning, and the specific process of outputting the model loss value includes: inputting the associated unstructured data and field name sequence into the feature extraction layer, feature parsing layer, field matching layer and key field determination layer in sequence for machine learning, and outputting the model loss value.

[0103] Among them, the loss function of the classifier is:

[0104]

[0105] in, is the model loss value, is the predicted probability of the classifier for the positive sample, is the weight to balance positive and negative samples, is the focus parameter, When the value of is greater than 0, it is used to reduce the attention paid to samples with high prediction probability of the classifier.

[0106] S103, using each identified form field as a key, extracting the value corresponding to the key from the key-value pair, and obtaining matching data of the custom form;

[0107] Among them, form fields refer to the areas in the form used to collect specific information. Each field has a clear identifier or name, and users can enter data in these fields. In form data processing, the key corresponds to the name of the form field. Matched data refers to a set of data formed by the values ​​corresponding to the keys extracted based on the matching of the keys in the key-value pair. These data correspond to the form fields one by one and contain all the information filled in by the user in the form.

[0108] In some embodiments, for example, an online recruitment platform has a job application form that includes the following fields: name, email address, work experience, and expected salary. After the applicant fills out and submits the form, the system needs to process this data. The system forms a key-value pair with each field and its corresponding user input data, for example: {"name": "Zhang San"}, {"email address": "zhangsan@example.com"}, {"work experience": "5 years"}, {"expected salary": "10,000 yuan / month"}. The system uses the identified form fields as keys, extracts the values ​​corresponding to each key from the key-value pair, and obtains the matching data for the custom form. These data include: Name: Zhang San, Email address: zhangsan@example.com, Work experience: 5 years, Expected salary: 10,000 yuan / month. Among them, matching data, such as Figure 4 shown.

[0109] S104, converting the matching data of the custom form into structured data according to a preset data conversion strategy;

[0110] The preset data conversion strategy includes a data preprocessing module and a field mapping module, and the matching data of the custom form is the value corresponding to each identified form field.

[0111] In an embodiment of the present application, according to a preset data conversion strategy, a specific process of converting the matching data of a custom form into structured data includes: using a data preprocessing module to remove noise, unify the format, and process missing values ​​for the values ​​corresponding to each identified form field, so as to obtain the final value corresponding to each identified form field; using a field mapping module to obtain the standard field corresponding to each identified form field from the standard field of the database; obtaining the data format of the standard field corresponding to each identified form field as the target data format corresponding to each identified form field; according to the target data format corresponding to each identified form field, format-converting the final value corresponding to each identified form field to obtain the structured data corresponding to each identified form field; using the structured data corresponding to each identified form field as the structured data of the custom form.

[0112] Among them, noise removal refers to the process of removing irrelevant or erroneous information from the data, which may be caused by erroneous, abnormal or incomplete data input. Format unification refers to converting data into a consistent format for easy processing and analysis, such as unifying all date formats into YYYY-MM-DD. Missing value processing refers to identifying and processing missing values ​​in the data, which may be done by filling in default values, deleting missing value records, or estimating missing values ​​using statistical methods. The field mapping module is a component used to match custom form fields with standard fields in the database to ensure data consistency and comparability. Standard fields refer to fields that are predefined in the database and follow a specific format and type. They are used to store and organize data. The target data format refers to the format that the data should conform to after conversion, which is consistent with the standard field format in the database. Structured data refers to data organized into a predefined format.

[0113] For example, an e-commerce platform needs to process product information forms from different suppliers, which contain fields such as product name, price, and stock quantity. The system uses a data preprocessing module to process the form data submitted by suppliers, including removing currency symbols and commas in the price field (noise removal), unifying all product names to lowercase (format unification), and filling default values ​​for missing stock quantities (missing value processing). The system uses a field mapping module to map the fields in the supplier form (such as "product name", "price", "stock quantity") to standard fields in the database (such as "product_name", "price", "stock_quantity"). The system obtains the data format of the standard fields in the database, for example, the price field is stored as a floating point number in the database, and the stock quantity is an integer. The system converts the preprocessed form field values ​​to the target data format, for example, converting the price from the string "$19.99" to the floating point number 19.99, and converting the stock quantity from the text "twenty" to the integer 20. The system stores the converted field values ​​as structured data.

[0114] In some embodiments of the present application, the specific process of obtaining the standard fields corresponding to each identified form field from the standard fields of the database includes: obtaining the first field name, the first field type and the first value range of each standard field from the standard fields of the database; obtaining the second field name and the second field type of each identified form field; obtaining the second value range corresponding to each identified form field from the final value corresponding to each identified form field; calculating the name similarity between each standard field and each identified form field according to the first field name of each standard field and the second field name of each identified form field; calculating the type similarity between each standard field and each identified form field according to the first field type of each standard field and the second field type of each identified form field; calculating the value range similarity between each standard field and each identified form field according to the first value range of each standard field and the second value range of each identified form field; calculating the comprehensive similarity between each identified form field and each standard field based on the name similarity, type similarity and value range similarity; selecting the standard field corresponding to the maximum comprehensive similarity from the comprehensive similarities between each identified form field and each standard field as the standard field corresponding to each identified form field.

[0115] S105, storing the structured data in the ClickHouse database, and visually displaying the stored structured data.

[0116] Among them, structured data refers to data organized in a predefined format, usually stored in a relational database, such as rows and columns. ClickHouse is a high-performance columnar database management system (DBMS) designed for online analytical processing (OLAP). It can handle large amounts of data, support real-time queries, and optimize the speed of writing and reading data. Visualization refers to displaying data in the form of graphics or charts so that users can intuitively understand the meaning and trends of the data.

[0117] In some embodiments, a ClickHouse database is used to store the converted structured data. The structured data is imported into the ClickHouse database, and a table containing columns of "sales date", "product ID", "sales quantity", and "sales amount" is created. A BI tool (such as Grafana or Tableau) is connected to the ClickHouse database, and a series of dashboards and charts are created to visualize the sales data, such as Figure 5 shown.

[0118] In some embodiments of the present application, the specific process of visualizing the stored structured data includes: reading the structured data from the ClickHouse database; analyzing the data feature points of the structured data; selecting the optimal chart type from the chart type library based on the data feature points; processing the structured data based on the optimal chart type to obtain a chart; and sending the chart to the client for visual display.

[0119] Data feature points refer to key attributes or statistical features in a data set, such as mean, median, maximum, minimum, distribution, and trend. The chart type library is a collection of multiple chart types and their applicable scenarios. You can select the appropriate chart type based on the characteristics of the data and analysis requirements.

[0120] For example, the website traffic data for the past year was read from the ClickHouse database, including structured data such as visits, page views, user dwell time, and conversion rate. Statistical tools were used to analyze this data to identify key feature points, such as seasonal trends in visits, peak periods of page views, and high and low conversion rates. Based on the data feature points, the analyst decided to use a line chart to show the seasonal trend of visits, a bar chart to show the peak period of page views, and a pie chart to show the conversion rate distribution of different pages. According to the selected chart type, data processing tools and programming languages ​​(such as Python's Pandas and Matplotlib libraries) are used to process the data and generate corresponding charts. The generated charts are uploaded to the server, and the marketing department can access these charts through the Web client for data analysis and decision support.

[0121] In the embodiment of the present application, on the one hand, the form fields of the custom form can be identified by using a pre-trained dynamic parsing algorithm model, which can adapt to custom forms of different formats, making the system more flexible and able to handle diversified data inputs, thereby reducing manual intervention and improving the efficiency and accuracy of data processing. On the other hand, through the preset data conversion strategy, the matching data is converted into structured data to ensure the consistency and standardization of the data. At the same time, the structured data is stored in the ClickHouse database, which is a high-performance database designed for online analytical processing, so that the data can support real-time query and analysis. Through real-time data analysis and intuitive visual display, decision makers can gain insights faster and make more accurate business decisions.

[0122] The following is a system embodiment of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the system embodiment of the present application, please refer to the method embodiment of the present application.

[0123] See also Figure 6 , which shows a schematic diagram of the structure of a data processing system for a custom form provided by an exemplary embodiment of the present application. The data processing system for a custom form can be implemented as all or part of an electronic device through software, hardware, or a combination of both. The system 1 includes a data receiving module 10, a field identification module 20, a form data determination module 30, a form data conversion module 40, and a visualization module 50.

[0124] A data receiving module 10, for receiving custom forms and form data from different product lines, wherein the form data includes at least one key-value pair;

[0125] A field recognition module 20 is used to recognize form fields of a custom form, where the form fields are obtained by dynamic parsing using a pre-trained dynamic parsing algorithm model, where the pre-trained dynamic parsing algorithm model is a neural network used to automatically recognize and parse key fields in a custom form;

[0126] The form data determination module 30 is used to use each identified form field as a key, extract the value corresponding to the key from the key-value pair, and obtain the matching data of the custom form;

[0127] The form data conversion module 40 is used to convert the matching data of the custom form into structured data according to a preset data conversion strategy;

[0128] The visualization module 50 is used to store the structured data in the ClickHouse database and to visualize the stored structured data.

[0129] It should be noted that the data processing system for custom forms provided in the above embodiment only uses the division of the above functional modules as an example when executing the data processing method for custom forms. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data processing system for custom forms provided in the above embodiment and the data processing method embodiment for custom forms belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.

[0130] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0131] In the embodiment of the present application, on the one hand, the form fields of the custom form can be identified by using a pre-trained dynamic parsing algorithm model, which can adapt to custom forms of different formats, making the system more flexible and able to handle diversified data inputs, thereby reducing manual intervention and improving the efficiency and accuracy of data processing. On the other hand, through the preset data conversion strategy, the matching data is converted into structured data to ensure the consistency and standardization of the data. At the same time, the structured data is stored in the ClickHouse database, which is a high-performance database designed for online analytical processing, so that the data can support real-time query and analysis. Through real-time data analysis and intuitive visual display, decision makers can gain insights faster and make more accurate business decisions.

[0132] The present application also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implements the data processing method for customizing a form provided by the above-mentioned various method embodiments.

[0133] The present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the data processing method for customizing a form in the above-mentioned various method embodiments.

[0134] See also Figure 7 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device 1000 may include: at least one processor 1001 , at least one network interface 1004 , a user interface 1003 , a memory 1005 , and at least one communication bus 1002 .

[0135] The communication bus 1002 is used to realize the connection and communication between these components.

[0136] The user interface 1003 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.

[0137] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0138] Among them, the processor 1001 may include one or more processing cores. The processor 1001 uses various interfaces and lines to connect various parts of the entire electronic device 1000, and executes various functions and processes data of the electronic device 1000 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1005, and calling data stored in the memory 1005. Optionally, the processor 1001 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1001 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1001, and it can be implemented separately through a chip.

[0139] Among them, the memory 1005 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may also be optionally at least one storage system located away from the aforementioned processor 1001. As Figure 7 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a data processing application for custom forms.

[0140] exist Figure 7In the electronic device 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 1001 can be used to call the data processing application for the custom form stored in the memory 1005, and specifically perform the following operations:

[0141] Receive custom forms and form data from different product lines, where the form data includes at least one key-value pair;

[0142] Identify form fields of a custom form. Form fields are dynamically parsed using a pre-trained dynamic parsing algorithm model. The pre-trained dynamic parsing algorithm model is a neural network used to automatically identify and parse key fields in a custom form.

[0143] Use each identified form field as a key, extract the value corresponding to the key from the key-value pair, and obtain the matching data of the custom form;

[0144] According to the preset data conversion strategy, the matching data of the custom form is converted into structured data;

[0145] Store structured data in the ClickHouse database and visualize the stored structured data.

[0146] In one embodiment, when the processor 1001 performs visualization on the stored structured data, the processor 1001 specifically performs the following operations:

[0147] Read structured data from the ClickHouse database;

[0148] Analyze the data feature points of structured data;

[0149] According to the data feature points, select the best chart type from the chart type library;

[0150] According to the optimal chart type, the structured data is processed to obtain a chart;

[0151] Send the chart to the client for visual display.

[0152] In one embodiment, when the processor 1001 identifies the form fields of the custom form, the processor 1001 specifically performs the following operations:

[0153] Load the pre-trained dynamic parsing algorithm model;

[0154] When the custom form is in image format or PDF format, optical character recognition technology is used to convert the custom form into a readable text format to obtain a form description text;

[0155] Input the form description text into the pre-trained dynamic parsing algorithm model to dynamically parse the form description text;

[0156] Output the field name sequence corresponding to the form description text;

[0157] Takes a sequence of field names to identify the form fields.

[0158] In one embodiment, when the processor 1001 performs dynamic parsing of the form description text, the processor 1001 specifically performs the following operations:

[0159] The parameter extraction layer extracts phrases, titles, and tags from the form description text to obtain unstructured data;

[0160] The feature extraction layer extracts the features of unstructured data to obtain form feature information;

[0161] The feature parsing layer performs parsing based on form feature information to identify all fields in the custom form;

[0162] The field matching layer matches all fields with predefined field templates to obtain multiple matching fields;

[0163] The key field determination layer uses a classifier to classify multiple matching fields to identify key fields of preset key types existing in the multiple matching fields, and obtain a field name sequence corresponding to the form description text.

[0164] In one embodiment, when the processor 1001 generates a pre-trained dynamic parsing algorithm model, the processor 1001 specifically performs the following operations:

[0165] Collect custom form samples covering various business scenarios and including different formats;

[0166] Preprocess custom form samples in different formats to obtain historical form description text. The preprocessing includes image conversion based on optical character recognition technology, text data cleaning and standardization.

[0167] Create a parameter extraction layer for extracting phrases, titles, and tags; create a feature extraction layer for extracting form feature information; create a feature parsing layer using natural language processing technology; create a field matching layer for matching identified fields with predefined field templates; use a random forest neural network to create a classifier as a key field determination layer; integrate the parameter extraction layer, feature extraction layer, feature parsing layer, field matching layer, and key field determination layer into a dynamic parsing algorithm model;

[0168] Generate model training samples based on historical form description text;

[0169] Input the model training samples into the dynamic parsing algorithm model for machine learning and output the model loss value;

[0170] When the model loss value reaches the minimum, a pre-trained dynamic parsing algorithm model is generated.

[0171] In one embodiment, when the processor 1001 generates a model training sample according to the historical form description text, the processor 1001 specifically performs the following operations:

[0172] The parameter extraction layer is used to obtain phrases, titles, and tags from the historical form description text to obtain historical unstructured data;

[0173] In response to a data labeling instruction for unstructured data, obtaining historical field name labels for historical unstructured data labeling, where the historical field name labels include positive samples and negative samples;

[0174] Map and associate historical unstructured data and historical field name labels to obtain associated unstructured data and field name labels as model training samples;

[0175] Input the model training samples into the dynamic parsing algorithm model for machine learning and output the model loss value, including:

[0176] The associated unstructured data and field name sequence are sequentially input into the feature extraction layer, feature parsing layer, field matching layer, and key field determination layer for machine learning, and the model loss value is output.

[0177] In one embodiment, when the processor 1001 converts the matching data of the custom form into structured data according to the preset data conversion strategy, the processor 1001 specifically performs the following operations:

[0178] The data preprocessing module is used to remove noise, unify the format, and process missing values ​​for the values ​​corresponding to each identified form field, so as to obtain the final value corresponding to each identified form field;

[0179] Using the field mapping module, obtain the standard fields corresponding to each identified form field from the standard fields in the database;

[0180] Acquire the data format of the standard field corresponding to each identified form field as the target data format corresponding to each identified form field;

[0181] According to the target data format corresponding to each identified form field, the final value corresponding to each identified form field is format-converted to obtain structured data corresponding to each identified form field;

[0182] The structured data corresponding to each identified form field is used as the structured data of the custom form.

[0183] In one embodiment, when the processor 1001 obtains the standard fields corresponding to the identified form fields from the standard fields in the database, the processor 1001 specifically performs the following operations:

[0184] From the standard fields of the database, obtain the first field name, the first field type, and the first value range of each standard field;

[0185] Obtain the second field name and the second field type of each identified form field;

[0186] Obtaining a second value range corresponding to each identified form field from the final values ​​corresponding to each identified form field;

[0187] Calculating the name similarity between each standard field and each identified form field according to the first field name of each standard field and the second field name of each identified form field;

[0188] Calculating type similarity between each standard field and each identified form field according to a first field type of each standard field and a second field type of each identified form field;

[0189] Calculating the similarity between the value ranges of each standard field and each identified form field based on the first value range of each standard field and the second value range of each identified form field;

[0190] Based on name similarity, type similarity, and value range similarity, calculate the comprehensive similarity between each identified form field and each standard field;

[0191] From the comprehensive similarities between each identified form field and each standard field, the standard field corresponding to the maximum comprehensive similarity is selected as the standard field corresponding to each identified form field.

[0192] In the embodiment of the present application, on the one hand, the form fields of the custom form can be identified by using a pre-trained dynamic parsing algorithm model, which can adapt to custom forms of different formats, making the system more flexible and able to handle diversified data inputs, thereby reducing manual intervention and improving the efficiency and accuracy of data processing. On the other hand, through the preset data conversion strategy, the matching data is converted into structured data to ensure the consistency and standardization of the data. At the same time, the structured data is stored in the ClickHouse database, which is a high-performance database designed for online analytical processing, so that the data can support real-time query and analysis. Through real-time data analysis and intuitive visual display, decision makers can gain insights faster and make more accurate business decisions.

[0193] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program for processing the data of the custom form can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium of the program for processing the data of the custom form can be a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0194] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method for a custom form, characterized in that: The method comprises: Receiving custom forms and form data thereof from different product lines, wherein the form data includes at least one key-value pair; Identifying form fields of the custom form, wherein the form fields are obtained by dynamic parsing using a pre-trained dynamic parsing algorithm model, wherein the pre-trained dynamic parsing algorithm model is a neural network for automatically identifying and parsing key fields in the custom form; Using each identified form field as a key, extracting the value corresponding to the key from the key-value pair to obtain matching data of the custom form; The identifying the form fields of the custom form includes: Load the pre-trained dynamic parsing algorithm model; In the case where the custom form is in an image format or a PDF format, the custom form is converted into a readable text format using an optical character recognition technology to obtain a form description text; Inputting the form description text into a pre-trained dynamic parsing algorithm model to dynamically parse the form description text; Output the field name sequence corresponding to the form description text; The field name sequence is used as each form field to be identified; wherein, The pre-trained dynamic parsing algorithm model includes a parameter extraction layer, a feature extraction layer, a feature parsing layer, a field matching layer, and a key field determination layer; The dynamically parsing the form description text includes: The parameter extraction layer extracts phrases, titles and tags from the form description text to obtain unstructured data; The feature extraction layer extracts features of the unstructured data to obtain form feature information; The feature parsing layer performs parsing based on the form feature information to identify all fields in the custom form; The field matching layer matches all the fields with a predefined field template to obtain a plurality of matching fields; The key field determination layer uses a classifier to classify the multiple matching fields to identify key fields of a preset key type existing in the multiple matching fields, and obtain a field name sequence corresponding to the form description text; According to a preset data conversion strategy, the matching data of the custom form is converted into structured data; wherein, The preset data conversion strategy includes a data preprocessing module and a field mapping module, and the matching data of the custom form is the value corresponding to each identified form field; The step of converting the matching data of the custom form into structured data according to a preset data conversion strategy includes: The data preprocessing module is used to remove noise, unify the format, and process missing values ​​for the values ​​corresponding to the identified form fields, so as to obtain the final values ​​corresponding to the identified form fields; Using the field mapping module, obtain the standard fields corresponding to each identified form field from the standard fields in the database; Acquire the data format of the standard field corresponding to each identified form field as the target data format corresponding to each identified form field; According to the target data format corresponding to each identified form field, the final value corresponding to each identified form field is format-converted to obtain structured data corresponding to each identified form field; Using the structured data corresponding to each identified form field as the structured data of the custom form; The structured data is stored in the ClickHouse database, and the stored structured data is visualized.

2. The method according to claim 1, characterized in that The visualization of the stored structured data includes: Read structured data from the ClickHouse database; Analyzing data feature points of the structured data; According to the data feature points, selecting an optimal chart type from a chart type library; Processing the structured data according to the optimal chart type to obtain a chart; The chart is sent to the client for visual display.

3. The method according to claim 1, characterized in that Follow these steps to generate a pre-trained dynamic parsing algorithm model, including: Collect custom form samples covering various business scenarios and including different formats; Preprocessing the custom form samples in different formats to obtain historical form description text, wherein the preprocessing includes image conversion based on optical character recognition technology, cleaning and standardization of text data; Create a parameter extraction layer for extracting phrases, titles and tags; create a feature extraction layer for extracting form feature information; create a feature parsing layer using natural language processing technology; create a field matching layer for matching the identified fields with predefined field templates; use a random forest neural network to create a classifier as a key field determination layer; integrate the parameter extraction layer, the feature extraction layer, the feature parsing layer, the field matching layer and the key field determination layer into a dynamic parsing algorithm model; Generate a model training sample according to the historical form description text; Input the model training samples into the dynamic parsing algorithm model for machine learning, and output the model loss value; When the model loss value reaches the minimum, a pre-trained dynamic parsing algorithm model is generated.

4. The method according to claim 3, characterized in that The step of generating a model training sample according to the historical form description text includes: The parameter extraction layer is used to obtain phrases, titles and tags from the historical form description text to obtain historical unstructured data; In response to a data labeling instruction for the unstructured data, obtaining a historical field name label labeled for the historical unstructured data, wherein the historical field name label includes a positive sample and a negative sample; Mapping and associating the historical unstructured data and the historical field name labels to obtain associated unstructured data and field name labels as model training samples; wherein, The step of inputting the model training sample into the dynamic parsing algorithm model for machine learning and outputting the model loss value comprises: The associated unstructured data and field name sequence are sequentially input into the feature extraction layer, feature parsing layer, field matching layer and key field determination layer for machine learning, and the model loss value is output.

5. The method according to claim 4, characterized in that The loss function of the classifier is: in, is the model loss value, is the predicted probability of the classifier for the positive sample, is the weight to balance the positive samples and the negative samples, is the focus parameter, When the value of is greater than 0, it is used to reduce the attention paid to samples with high prediction probability of the classifier.

6. The method according to claim 1, characterized in that The step of obtaining the standard fields corresponding to the identified form fields from the standard fields of the database includes: From the standard fields of the database, obtain the first field name, the first field type, and the first value range of each standard field; Obtain the second field name and the second field type of each identified form field; Obtaining a second value range corresponding to each identified form field from the final values ​​corresponding to each identified form field; Calculating the name similarity between each standard field and each identified form field according to the first field name of each standard field and the second field name of each identified form field; Calculating type similarity between each standard field and each identified form field according to the first field type of each standard field and the second field type of each identified form field; Calculating the similarity between the value ranges of each standard field and each identified form field according to the first value range of each standard field and the second value range of each identified form field; Calculating the comprehensive similarity between each identified form field and each standard field based on the name similarity, the type similarity and the value range similarity; From the comprehensive similarities between the identified form fields and each standard field, the standard field corresponding to the maximum comprehensive similarity is selected as the standard field corresponding to the identified form fields.

7. A data processing system for custom forms implemented using the method described in any one of claims 1 to 6, characterized in that: The system comprises: A data receiving module, used for receiving custom forms and form data from different product lines, wherein the form data includes at least one key-value pair; A field recognition module, used to recognize the form fields of the custom form, wherein the form fields are obtained by dynamic parsing using a pre-trained dynamic parsing algorithm model, wherein the pre-trained dynamic parsing algorithm model is a neural network for automatically recognizing and parsing key fields in the custom form; A form data determination module, used to use each identified form field as a key, extract the value corresponding to the key from the key-value pair, and obtain matching data of the custom form; A form data conversion module, used to convert the matching data of the custom form into structured data according to a preset data conversion strategy; The visualization module is used to store the structured data in the ClickHouse database and to visualize the stored structured data.

Images