Data processing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a data processing method and device, electronic equipment, a storage medium and a program product, and is applied to the technical field of artificial intelligence. The method comprises the following steps: receiving data query demand information of a user terminal; inputting the data query demand information into a large model agent, and analyzing the data query demand information by using the large model agent to obtain unstructured demand information and structured demand information; according to the unstructured demand information, querying first data information matched with the unstructured demand information from a pre-constructed vector database, and querying second data information matched with the structured demand information from a pre-constructed structured database; determining target data information according to the first data information and the second data information; and sending the target data information to a user terminal. The method improves the accuracy of data matching.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data processing method, device, electronic device, storage medium, and program product. Background Art

[0002] The circulation of data elements is committed to breaking down traditional data silos and promoting the safe and efficient circulation and transaction of data in a trusted environment. Data access terminals in the data circulation infrastructure play the dual roles of data provision and demand. With the large-scale promotion and application, not only will massive amounts of data be accessed, but also diverse and complex data demands will be derived. Data matching in related technologies is mainly achieved through keyword fuzzy matching methods. Keyword fuzzy matching refers to matching target data in data with certain patterns by identifying and analyzing keywords input by users. However, in the multi-dimensional cross-data calculation scale, as well as complex, dynamic, and changeable query scenarios, the single-dimensional keyword fuzzy method is not only limited in efficiency when processing massive amounts of data, but also difficult to accurately capture the user's true intentions, resulting in low accuracy of the matched data. Summary of the Invention

[0003] The embodiments of the present application provide a data processing method and apparatus to solve the problem of low accuracy of data matching in related technologies.

[0004] In order to solve the above technical problems, this application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a data processing method, the method comprising:

[0006] Receiving data query requirement information from a user terminal, wherein the data query requirement information includes data quality requirement information;

[0007] Inputting the data query demand information into the large model agent, and using the large model agent to parse the data query demand information to obtain unstructured demand information and structured demand information;

[0008] According to the unstructured demand information, querying a pre-built vector database for first data information matching the unstructured demand information, wherein the vector database stores vector data information of multimodal data;

[0009] According to the structured demand information, querying a pre-built structured database for second data information that matches the structured demand information, wherein the structured database stores structured data information of the multimodal data;

[0010] determining target data information according to the first data information and the second data information;

[0011] Sending the target data information to the user terminal.

[0012] Optionally, the querying, based on the unstructured demand information, from a pre-built vector database for first data information matching the unstructured demand information includes:

[0013] Demand vector information corresponding to the unstructured demand information is generated, and based on the demand vector information, a vector similarity search is performed through a pre-built vector database to determine first data information matching the demand vector information.

[0014] Optionally, the querying, based on the structured demand information, from a pre-built structured database for second data information that matches the structured demand information includes:

[0015] A demand structured query statement SQL corresponding to the structured demand information is generated, and based on the demand SQL, a query is performed through a pre-built structured database to determine second data information matching the demand SQL.

[0016] Optionally, generating a demand structured query statement SQL corresponding to the structured demand information, and performing a query based on the demand SQL through a pre-built structured database to determine the second data information matching the demand SQL includes:

[0017] Performing semantic analysis on the structured demand information to obtain a semantic demand text corresponding to the structured demand information;

[0018] Generate a requirement SQL corresponding to the semantic requirement text;

[0019] The demand SQL is escaped, and based on the escaped demand SQL, a pre-built structured database is searched for second data information that matches the escaped demand SQL.

[0020] Optionally, before inputting the data query demand information into the large model agent and parsing the data query demand information using the large model agent to obtain unstructured demand information and structured demand information, the method further includes:

[0021] Searching for target information matching the data query requirement information from a pre-built knowledge base, wherein the similarity between the target information and the data query requirement information is greater than a preset value;

[0022] The step of inputting the data query requirement into the large model agent and using the large model agent to parse the data query requirement to obtain unstructured requirement information and structured requirement information includes:

[0023] The data query requirements and target information matching the data query requirements are input into a large model intelligent agent, and the data query requirements and target information matching the data query requirements are parsed by the large model intelligent agent to obtain unstructured demand information and structured demand information.

[0024] Optionally, before receiving the user's data query request, the method further includes:

[0025] Inputting the multimodal data of the data access terminal into the large model agent, parsing the multimodal data using the large model agent to obtain structured statistical information and unstructured text information of the multimodal data;

[0026] performing vectorization processing on the unstructured text information and the unstructured assessment information of the quality assessment result of the multimodal data to obtain vector data information of the multimodal data, and storing the vector data information in a vector database;

[0027] The structured statistical information and a quality assessment database table corresponding to the structured assessment information of the quality assessment result of the multimodal data are stored in a structured database, and the structured data information of the multimodal data includes the structured statistical information and a quality assessment database table corresponding to the structured assessment information of the quality assessment result of the multimodal data.

[0028] Optionally, after storing the structured statistical information and the quality assessment database table corresponding to the structured assessment information of the quality assessment result of the multimodal data in a structured database, the method further includes:

[0029] Building a semantic model based on the mapping relationship between the quality assessment fields in the quality assessment database table and the quality assessment terms in the Chinese term library;

[0030] Extract pattern information from the semantic model, and construct a dictionary and index based on the pattern information to obtain a knowledge base, wherein the pattern information includes the meaning of the quality assessment field, the meaning of the quality assessment term, and the mapping relationship between the quality assessment field and the quality assessment term.

[0031] Optionally, before inputting the multimodal data of the data access terminal into the large model agent and parsing the multimodal data using the large model agent to obtain structured statistical information and unstructured text information of the multimodal data, the method further includes:

[0032] receiving multimodal data and quality processing requirements of the multimodal data sent by a data access terminal;

[0033] Inputting the quality processing requirement into a large model agent, identifying a quality assessment type corresponding to the quality processing requirement through the large model agent, determining a basic function matching the quality assessment type from a predefined basic function list, and generating function call information according to the basic function;

[0034] Calling the basic function according to the function call information to perform quality assessment on the multimodal data according to the quality processing requirement, thereby obtaining a quality assessment result of the multimodal data;

[0035] Sending a quality assessment result of the multimodal data to the data access terminal.

[0036] Optionally, after sending the quality assessment result of the multimodal data to the data access terminal, the method further includes:

[0037] generating a quality assessment template according to the data content of the multimodal data, the quality processing requirement, the basis function, and the quality assessment result;

[0038] The quality assessment template is stored.

[0039] In a second aspect, an embodiment of the present application further provides a data processing device, the device comprising:

[0040] A first receiving module is configured to receive data query requirement information from a user terminal, wherein the data query requirement information includes data quality requirement information;

[0041] A first parsing module is configured to input the data query demand information into a large model agent, and use the large model agent to parse the data query demand information to obtain unstructured demand information and structured demand information;

[0042] a first query module, configured to query, based on the unstructured demand information, a pre-built vector database for first data information matching the unstructured demand information, wherein the vector database stores vector data information of multimodal data;

[0043] a second query module, configured to query a pre-built structured database for second data information matching the structured demand information according to the structured demand information, wherein the structured database stores the structured data information of the multimodal data;

[0044] A first determining module, configured to determine target data information according to the first data information and the second data information;

[0045] The first sending module is configured to send the target data information to the user terminal.

[0046] In a third aspect, an embodiment of the present application further provides an electronic device, including a processor and a transceiver, wherein the transceiver is configured to:

[0047] Receiving data query requirement information from a user terminal, wherein the data query requirement information includes data quality requirement information;

[0048] The processor is configured to:

[0049] Inputting the data query demand information into the large model agent, and using the large model agent to parse the data query demand information to obtain unstructured demand information and structured demand information;

[0050] According to the unstructured demand information, querying a pre-built vector database for first data information matching the unstructured demand information, wherein the vector database stores vector data information of multimodal data;

[0051] According to the structured demand information, querying a pre-built structured database for second data information that matches the structured demand information, wherein the structured database stores structured data information of the multimodal data;

[0052] determining target data information according to the first data information and the second data information;

[0053] The transceiver is used for:

[0054] Sending the target data information to the user terminal.

[0055] In a fourth aspect, an embodiment of the present application further provides an electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the above-mentioned data processing method when executed by the processor.

[0056] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned data processing method are implemented.

[0057] In a sixth aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the data processing method as described above.

[0058] The data processing method of an embodiment of the present application includes receiving data query demand information from a user terminal, wherein the data query demand information includes data quality demand information; inputting the data query demand information into a large model intelligent agent, and using the large model intelligent agent to parse the data query demand information to obtain unstructured demand information and structured demand information; based on the unstructured demand information, querying first data information matching the unstructured demand information from a pre-constructed vector database, wherein the vector database stores vector data information of multimodal data; based on the structured demand information, querying second data information matching the structured demand information from a pre-constructed structured database, wherein the structured database stores structured data information of the multimodal data; determining target data information based on the first data information and the second data information; and sending the target data information to the user terminal.

[0059] In this approach, a large-scale intelligent agent performs in-depth analysis of the data query request information input by the user terminal, generating unstructured and structured request information. A vector database is then used to retrieve the first data matching the unstructured request information, and a structured database is used to accurately retrieve the second data matching the structured request information. This combined approach improves adaptability to complex query scenarios, combining unstructured semantic understanding with structured logic verification, and overcomes the dimensional limitations of keyword matching, thereby enhancing data matching accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0061] Figure 1 is a flow chart of the data processing method provided in an embodiment of the present application;

[0062] Figure 2 Schematic diagram of data quality assessment provided in an embodiment of the present application;

[0063] Figure 3 is a schematic diagram of data processing provided by an embodiment of the present application;

[0064] Figure 4 is a structural diagram of a data processing device provided in one embodiment of the present application;

[0065] Figure 5 This is a structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0067] The present application embodiment provides a data processing method. Figure 1 , Figure 1 is a flow chart of the data processing method provided in the embodiment of the present application, such as Figure 1 As shown, the following steps are included:

[0068] Step 101: Receive data query requirement information from a user terminal, where the data query requirement information includes data quality requirement information;

[0069] In this step, the data processing system receives data query demand information from the user terminal. The data query demand information is used to indicate the specific information of the data that the user terminal needs to query, such as the type of data and the content of the data; the data quality demand information is used to indicate the quality requirements of the user terminal for the data to be queried, such as the integrity, accuracy, consistency and other requirements of the data.

[0070] Step 102: input the data query demand information into the large model agent, and use the large model agent to parse the data query demand information to obtain unstructured demand information and structured demand information;

[0071] In this step, the large-model agent is an intelligent system based on a large-scale language model, capable of natural language understanding, semantic parsing, logical reasoning, and multimodal data processing. Through extensive knowledge and experience pre-trained on large-scale datasets, the large-model agent can adapt to multimodal data of different types, including structured, unstructured, and semi-structured data.

[0072] The data query requirement information is input into the large-scale intelligent agent, which then parses the data query requirement information to obtain unstructured and structured requirement information. For example, unstructured requirement information can include natural language descriptions of the requirement and business scenario information, such as "user behavior data" and "high integrity." Structured requirement information can include structured numerical information, such as "field missing rate ≤ 5%."

[0073] Step 103: According to the unstructured demand information, query a pre-built vector database for first data information that matches the unstructured demand information, wherein the vector database stores vector data information of multimodal data;

[0074] In this step, the vector database stores vector data information of multimodal data. Through semantic similarity retrieval, first data information related to the unstructured demand information can be matched from the vector database. Specifically, the unstructured demand information is converted into a vector, and similarity calculation is performed with the vector data information of the multimodal data in the vector database to determine the first data information with the highest similarity to the unstructured demand information. For example, the unstructured demand information can be "user behavior data," and the first data information can be "user click log" or "order behavior data."

[0075] Step 104: According to the structured demand information, query a pre-built structured database for second data information that matches the structured demand information, wherein the structured database stores structured data information of the multimodal data;

[0076] In this step, the structured database stores structured data information of the multimodal data. The structured data information includes structured statistical information and a quality assessment database table corresponding to the structured assessment information of the quality assessment results of the multimodal data. Structured statistical information refers to the results of quantitative statistics of multimodal data (such as text, images, and structured fields), such as field missing rate, data distribution, and numerical range, which are used to describe the basic characteristics of the data. The quality assessment database table refers to a structured table that stores the quality assessment results of the multimodal data.

[0077] Specifically, the structured requirement information may be converted into a Structured Query Language (SQL) statement, and then a database query is executed. For example, the structured requirement information is WHERE metric_type='completeness' AND score>90, and the second data information with a completeness score higher than 90% is queried.

[0078] Step 105: Determine target data information based on the first data information and the second data information;

[0079] In this step, the first data matched in the vector database is correlated with the second data matched in the structured database to extract the target data that meets the user's needs. For example, the first data is a sample of "user click logs," the second data is a sample of "user behavior data" with an integrity score greater than 90%, and the target data is a sample of "user click logs" with an integrity score greater than 90%.

[0080] Step 106: Send the target data information to the user terminal.

[0081] In this step, the target data information finally screened out is returned to the user terminal. The target data information may include specific data content and data quality assessment results.

[0082] Furthermore, after sending the target data information to the user terminal, satisfaction information based on the target data information can be received from the user terminal. The vector database and structured database can then be adjusted based on the satisfaction information. This allows the vector database and structured database to be continuously improved, thereby enhancing the accuracy of data matching.

[0083] In one embodiment, data query demand information of a user terminal is received, wherein the data query demand information includes data quality demand information; the data query demand information is input into a large model intelligent agent, and the data query demand information is parsed using the large model intelligent agent to obtain unstructured demand information and structured demand information; based on the unstructured demand information, first data information matching the unstructured demand information is queried from a pre-constructed vector database, wherein the vector database stores vector data information of multimodal data; based on the structured demand information, second data information matching the structured demand information is queried from a pre-constructed structured database, wherein the structured database stores structured data information of the multimodal data; based on the first data information and the second data information, target data information is determined; and the target data information is sent to the user terminal.

[0084] In this implementation, a large-scale intelligent agent performs in-depth analysis of the data query request information input by the user terminal, generating unstructured and structured request information. A vector database is then used to retrieve first data matching the unstructured request information, and a structured database is used to accurately retrieve second data matching the structured request information. This combination improves adaptability to complex query scenarios, combining unstructured semantic understanding with structured logic verification, transcending the dimensional limitations of keyword matching and enhancing data matching accuracy.

[0085] Optionally, the querying, based on the unstructured demand information, from a pre-built vector database for first data information matching the unstructured demand information includes:

[0086] Demand vector information corresponding to the unstructured demand information is generated, and based on the demand vector information, a vector similarity search is performed through a pre-built vector database to determine first data information matching the demand vector information.

[0087] In one implementation, unstructured user demand information (e.g., a natural language description of "querying for high-integrity user behavior data") is converted into a vector representation (i.e., a numerated semantic vector) by a large model agent. This process utilizes natural language processing techniques to convert text into high-dimensional vectors, capturing its semantic characteristics. A vector database stores vector data information for multimodal data. By calculating the similarity between the user terminal's demand vector information and the vector data information in the vector database, the data sample that most closely matches the demand semantics can be found and used as the first data information.

[0088] In this implementation, traditional keyword matching relies on literal similarity and is susceptible to synonyms, ambiguity, or context. Vector similarity search, on the other hand, captures the deeper meaning of requirements through semantic vectors, enabling a more accurate match to user intent.

[0089] Optionally, the querying, based on the structured demand information, from a pre-built structured database for second data information that matches the structured demand information includes:

[0090] A demand structured query statement SQL corresponding to the structured demand information is generated, and based on the demand SQL, a query is performed through a pre-built structured database to determine second data information matching the demand SQL.

[0091] In one embodiment, structured demand information refers to clear and resolvable query conditions (such as field names, numerical ranges, logical conditions, etc.). The data processing system converts these structured demand information into demand SQL. SQL statements are instructions that can be directly executed by the database and can accurately match data fields and conditions. The structured database stores structured data information of multimodal data. The structured data information includes structured statistical information and a quality assessment database table corresponding to the structured assessment information of the quality assessment results of the multimodal data. Structured statistical information refers to the results of quantitative statistics of multimodal data (such as text, images, structured fields), such as field missing rate, data distribution, numerical range, etc., which are used to describe the basic characteristics of the data. The quality assessment database table refers to a structured table that stores the quality assessment results of multimodal data. The data processing system then executes the generated demand SQL and queries the structured database for the second data information that meets the demand SQL.

[0092] In this implementation, the SQL statement is a structured query logic that can directly locate fields and conditions in the database, thereby avoiding errors in traditional keyword fuzzy matching.

[0093] Optionally, generating a demand structured query statement SQL corresponding to the structured demand information, and performing a query based on the demand SQL through a pre-built structured database to determine the second data information matching the demand SQL includes:

[0094] Performing semantic analysis on the structured demand information to obtain a semantic demand text corresponding to the structured demand information;

[0095] Generate a requirement SQL corresponding to the semantic requirement text;

[0096] The demand SQL is escaped, and based on the escaped demand SQL, a pre-built structured database is searched for second data information that matches the escaped demand SQL.

[0097] In one implementation, natural language processing can be used to semantically parse structured requirement information and convert it into semantic requirement text, providing a foundation for subsequent SQL statement generation. Semantic requirement text can be more professional business terms that are easier for users to understand, such as dimensions (completeness), indicators (field missing rate), and tags (high completeness).

[0098] Generate executable requirement SQL based on the semantic requirement text. For example, the semantic requirement text "data integrity score must reach above 90%" is converted to "SELECT * FROM data_quality WHERE metric_type = 'integrity' AND score>90". That is, combine the semantic parsing results and the database table structure to generate SQL statements that conform to the database logic, thereby ensuring that the query conditions match the data fields.

[0099] Then, the required SQL is escaped and translated into an SQL statement that can be executed on the physical data model. The escaped SQL statement is then executed to retrieve the second data information that meets the requirements from the structured database.

[0100] In this implementation, structured requirements are converted into semantic text through semantic parsing to improve the accuracy of requirement understanding; corresponding SQL is generated to achieve precise data query, avoiding the errors of traditional fuzzy matching; combined with efficient retrieval of structured databases, query efficiency and result reliability are improved.

[0101] Optionally, before inputting the data query demand information into the large model agent and parsing the data query demand information using the large model agent to obtain unstructured demand information and structured demand information, the method further includes:

[0102] Searching for target information matching the data query requirement information from a pre-built knowledge base, wherein the similarity between the target information and the data query requirement information is greater than a preset value;

[0103] The step of inputting the data query requirement into the large model agent and using the large model agent to parse the data query requirement to obtain unstructured requirement information and structured requirement information includes:

[0104] The data query requirements and target information matching the data query requirements are input into a large model intelligent agent, and the data query requirements and target information matching the data query requirements are parsed by the large model intelligent agent to obtain unstructured demand information and structured demand information.

[0105] In one implementation, the Retrieval-Augmented Generation (RAG) technology of a large-scale agent can be utilized to enable the large-scale agent to more accurately resolve data query requirements from user terminals. RAG technology is a method that combines external knowledge with large-scale agents. Its core is to extract relevant textual information from an external knowledge base through vector semantic retrieval and input this relevant textual information as contextual information into the large-scale agent, thereby enhancing the generation capabilities of the large-scale agent. Furthermore, the knowledge base can be improved based on the accuracy of the generated unstructured and structured demand information, thereby further optimizing the accuracy of knowledge base interactions.

[0106] In this application, similarity retrieval can be used to search a pre-built knowledge base for target information that matches the data query requirement information, i.e., relevant text information corresponding to the data query requirement information. The target information is then input into the large model agent as contextual information along with the data query requirement information, thereby enhancing the large model agent's ability to understand the data query requirement information and improving the accuracy of the obtained unstructured and structured requirement information.

[0107] Optionally, before receiving the user's data query request, the method further includes:

[0108] Inputting the multimodal data of the data access terminal into the large model agent, parsing the multimodal data using the large model agent to obtain structured statistical information and unstructured text information of the multimodal data;

[0109] performing vectorization processing on the unstructured text information and the unstructured assessment information of the quality assessment result of the multimodal data to obtain vector data information of the multimodal data, and storing the vector data information in a vector database;

[0110] The structured statistical information and a quality assessment database table corresponding to the structured assessment information of the quality assessment result of the multimodal data are stored in a structured database, and the structured data information of the multimodal data includes the structured statistical information and a quality assessment database table corresponding to the structured assessment information of the quality assessment result of the multimodal data.

[0111] In one embodiment, the data access terminal loads multimodal data. Multimodal data refers to data containing multiple types or forms, usually referring to a combination of data of different modes such as text, images, audio, video, structured fields, etc., which differ in form, structure or source. In order to facilitate subsequent data queries, a large model intelligent agent can be used to parse complex multimodal data to obtain structured statistical information and unstructured text information. The structured statistical information and unstructured text information have unique identifiers, and the identifiers of the structured statistical information and unstructured text information obtained for the same multimodal data are the same. Structured statistical information can be field missing rate, data distribution, quality score (such as completeness score, accuracy score), etc., and unstructured text information can be field definition (such as "user unique identifier"), business rules (such as "user churn needs to be combined with historical behavior analysis"), data source description, etc.

[0112] Unstructured information (such as text descriptions, business rules) and unstructured assessment information of quality assessment results (such as "completeness score must be higher than 90%") are then converted into vector data information and stored in a vector database to facilitate subsequent semantic retrieval. The quality assessment database table corresponding to the structured statistical information and the structured assessment information of the quality assessment results of multimodal data is stored in a structured database to support subsequent precise queries. The quality assessment database table stores structured data such as the completeness score, accuracy score, consistency score, etc. of the field, which can be used to sort out tables and fields and describe the fields.

[0113] In this implementation, multimodal data is parsed by a large-model intelligent agent, which can efficiently extract structured statistical information and unstructured text information to achieve intelligent data governance. After vectorization processing, unstructured information (such as text descriptions, SQL statements) is stored in a vector database, which supports semantic similarity retrieval and improves the relevance and matching accuracy of multimodal data. The structured database stores quality assessment results and field definitions to facilitate accurate queries and data value exploration. This solution integrates semantic understanding and structured queries to address the shortcomings of related technologies in multimodal data processing, semantic fuzzy matching and quality assessment, improves data processing efficiency, accuracy and flexibility, and adapts to the needs of complex business scenarios.

[0114] Optionally, after storing the structured statistical information and the quality assessment database table corresponding to the structured assessment information of the quality assessment result of the multimodal data in a structured database, the method further includes:

[0115] Building a semantic model based on the mapping relationship between the quality assessment fields in the quality assessment database table and the quality assessment terms in the Chinese term library;

[0116] Extract pattern information from the semantic model, and construct a dictionary and index based on the pattern information to obtain a knowledge base, wherein the pattern information includes the meaning of the quality assessment field, the meaning of the quality assessment term, and the mapping relationship between the quality assessment field and the quality assessment term.

[0117] In one embodiment, the quality assessment database table stores the quality assessment results of multimodal data, such as the completeness score, accuracy score, consistency score, etc. of the field (such as missing_percentage = 3%, data_consistency = 95%). The Chinese terminology library contains Chinese terms related to quality assessment (such as "completeness", "accuracy", "field missing rate", etc.), which may correspond to fields or quality assessment indicators in the database table. By analyzing the semantic association between quality assessment fields (such as missing_percentage) and Chinese terms (such as "field missing rate"), a correspondence between fields and terms is established. For example, the missing_percentage field corresponds to the term "field missing rate". Based on this mapping relationship, a semantic model is constructed to understand the semantic association between quality assessment fields and Chinese terms, which can provide a basis for subsequent semantic retrieval and knowledge management.

[0118] Schema information is extracted from the semantic model, including: the meaning of quality assessment fields (e.g., the missing_percentage field represents "field missing rate"); the meaning of quality assessment terms (e.g., "field missing rate" refers to "the proportion of missing fields in the data"); and the mapping between fields and terms (e.g., missing_percentage → "field missing rate"). This schema information clarifies the semantic associations between quality assessment data and Chinese terms, providing structured data for knowledge base construction.

[0119] The quality assessment fields, terms, and their mappings in the schema information are organized into a structured dictionary. The terms and fields in the dictionary are indexed to support fast retrieval (for example, quickly locating the missing_percentage field by searching for "field missing rate"). The dictionary and index are integrated into a knowledge base for subsequent data governance, semantic retrieval, and intelligent querying. For example, if a user enters "query field missing rate," the system automatically associates it with the missing_percentage field and executes the query.

[0120] In this implementation, quality assessment data is associated with the Chinese terminology library through semantic modeling to build a structured knowledge base, which solves problems such as inconsistency between terms and fields and semantic ambiguity in data processing, and provides basic support for intelligent query, quality assessment and multimodal data management.

[0121] Optionally, before inputting the multimodal data of the data access terminal into the large model agent and parsing the multimodal data using the large model agent to obtain structured statistical information and unstructured text information of the multimodal data, the method further includes:

[0122] receiving multimodal data and quality processing requirements of the multimodal data sent by a data access terminal;

[0123] Inputting the quality processing requirement into a large model agent, identifying a quality assessment type corresponding to the quality processing requirement through the large model agent, determining a basic function matching the quality assessment type from a predefined basic function list, and generating function call information according to the basic function;

[0124] Calling the basic function according to the function call information to perform quality assessment on the multimodal data according to the quality processing requirement, thereby obtaining a quality assessment result of the multimodal data;

[0125] Sending a quality assessment result of the multimodal data to the data access terminal.

[0126] In one embodiment, see Figure 2 , receiving multimodal data sent by the data access terminal and the quality processing requirements for multimodal data. Multimodal data refers to data that includes multiple types, and quality processing requirements refer to specific requirements for data quality, such as completeness, accuracy, and consistency.

[0127] The large-scale model agent then identifies the quality assessment type corresponding to the quality processing requirements, such as completeness assessment (which requires checking the field missing rate of multimodal data), accuracy assessment (which requires verifying the numerical range of multimodal data), and consistency assessment (which compares field values across different data sources). The large-scale model agent then identifies the base function that matches the quality assessment type from a predefined list of base functions and generates function call information based on the base function. This function call information includes the name of the base function to be called, the representation of the multimodal data, and the request parameters. The large-scale model agent can transmit this function call information to the data processing system in JSON format.

[0128] After receiving the call request, the data processing system can call the corresponding basic function according to the function call information. Then, based on the basic function, the multimodal data is quality assessed according to the quality processing requirements to obtain a quality assessment result. For example, if the quality processing requirement is to "check the field missing rate", the calculate_missing_rate() function is called, and the field name (such as user_id) and the data set are passed in. The quality assessment result of the multimodal data is "field missing rate = 3%". The field assessment result of the multimodal data is then sent to the data access terminal.

[0129] In this implementation, by integrating large-model intelligent agent technology and automatic function combination capabilities, a multimodal data quality assessment system for data circulation scenarios is constructed to achieve automation, personalization and group intelligence of data quality assessment, and flexibly adapt to customized assessment needs of multiple types of data and multiple access terminals.

[0130] Optionally, after sending the quality assessment result of the multimodal data to the data access terminal, the method further includes:

[0131] generating a quality assessment template according to the data content of the multimodal data, the quality processing requirement, the basis function, and the quality assessment result;

[0132] The quality assessment template is stored.

[0133] In one implementation, a quality assessment template can be generated based on the multimodal data's content, quality processing requirements, basic functions, and quality assessment results. The generated template can be stored in a structured format in a database or knowledge base for easy subsequent use. This ensures consistent evaluation logic for the same or similar requirements, avoiding duplication of development. Subsequent data quality assessments provide a directly callable template, reducing configuration time.

[0134] One implementation method, see Figure 3 ,pass Figure 3 A complete description of the embodiments of this application is given below:

[0135] First, the user's data query demand information is input into the large model intelligent agent, and the large model intelligent agent is used to perform qualitative intent recognition on the data query demand information to obtain unstructured demand information, and the large model intelligent agent is used to perform quantitative intent recognition on the data query demand information to obtain structured demand information.

[0136] Secondly, demand vector information corresponding to the unstructured demand information is generated, and based on the demand vector information, a vector similarity search is performed through a pre-built vector database to determine first data information that matches the demand vector information, wherein the vector database stores vector data information of multimodal data;

[0137] Furthermore, semantic analysis is performed on the structured demand information to obtain a semantic demand text corresponding to the structured demand information; demand SQL corresponding to the semantic demand text is generated; the demand SQL is escaped, and based on the escaped demand SQL, a second data information matching the escaped demand SQL is searched from a pre-built structured database;

[0138] Finally, target data information is determined according to the first data information and the second data information.

[0139] Prior to this, a semantic model was constructed based on the mapping relationship between the quality assessment fields in the quality assessment database table and the quality assessment terms in the Chinese terminology database. Pattern information in the semantic model was extracted, and a dictionary and index were constructed based on the pattern information to obtain a knowledge base. The pre-built knowledge base was searched for target information that matched the data query requirement information. The data query requirement and the target information that matched the data query requirement were input into the large model agent. The large model agent then parsed the data query requirement and the target information that matched the data query requirement to obtain unstructured and structured requirement information.

[0140] Prior to this, the multimodal data of the data access terminal is input into the large model intelligent agent, and the large model intelligent agent is used to parse the multimodal data to obtain structured statistical information and unstructured text information of the multimodal data; the unstructured text information and the unstructured evaluation information of the quality evaluation results of the multimodal data are vectorized to obtain vector data information of the multimodal data, and the vector data information is stored in a vector database; the structured statistical information and the quality evaluation database table corresponding to the structured evaluation information of the quality evaluation results of the multimodal data are stored in a structured database.

[0141] See also Figure 4 , Figure 4 This is a structural diagram of a data processing device provided by another embodiment of the present application. Figure 4 As shown, the data processing device 400 includes:

[0142] The first receiving module 401 is configured to receive data query requirement information from a user terminal, wherein the data query requirement information includes data quality requirement information;

[0143] A first parsing module 402 is configured to input the data query demand information into a large model agent, and use the large model agent to parse the data query demand information to obtain unstructured demand information and structured demand information;

[0144] A first query module 403 is configured to query a pre-built vector database for first data information matching the unstructured demand information based on the unstructured demand information, wherein the vector database stores vector data information of multimodal data;

[0145] A second query module 404 is configured to query a pre-built structured database for second data information matching the structured requirement information based on the structured requirement information, wherein the structured database stores the structured data information of the multimodal data;

[0146] A first determining module 405 is configured to determine target data information based on the first data information and the second data information;

[0147] The first sending module 406 is configured to send the target data information to the user terminal.

[0148] Optionally, the first query module includes:

[0149] The first determining unit is configured to generate demand vector information corresponding to the unstructured demand information, and perform vector similarity retrieval based on the demand vector information through a pre-built vector database to determine first data information matching the demand vector information.

[0150] Optionally, the second query module includes:

[0151] The second determining unit is configured to generate a demand structured query statement SQL corresponding to the structured demand information, and based on the demand SQL, perform a query through a pre-built structured database to determine second data information matching the demand SQL.

[0152] Optionally, the second determining unit includes:

[0153] A first parsing subunit is configured to perform semantic parsing on the structured demand information to obtain a semantic demand text corresponding to the structured demand information;

[0154] A first generating subunit is configured to generate a requirement SQL corresponding to the semantic requirement text;

[0155] The first query sub-unit is configured to perform an escape process on the demand SQL, and based on the escaped demand SQL, query a pre-built structured database for second data information that matches the escaped demand SQL.

[0156] Optionally, the device further comprises:

[0157] A first search module is configured to search a pre-built knowledge base for target information that matches the data query requirement information, wherein the similarity between the target information and the data query requirement information is greater than a preset value;

[0158] The first parsing module includes:

[0159] The first parsing unit is used to input the data query requirements and the target information matching the data query requirements into the large model intelligent agent, and use the large model intelligent agent to parse the data query requirements and the target information matching the data query requirements to obtain unstructured demand information and structured demand information.

[0160] Optionally, the device further comprises:

[0161] A second parsing module is configured to input the multimodal data of the data access terminal into the large model agent, and parse the multimodal data using the large model agent to obtain structured statistical information and unstructured text information of the multimodal data;

[0162] a first storage module, configured to perform vectorization processing on the unstructured text information and the unstructured assessment information of the quality assessment result of the multimodal data to obtain vector data information of the multimodal data, and store the vector data information in a vector database;

[0163] The second storage module is used to store the structured statistical information and the quality assessment database table corresponding to the structured assessment information of the quality assessment result of the multimodal data in a structured database, wherein the structured data information of the multimodal data includes the structured statistical information and the quality assessment database table corresponding to the structured assessment information of the quality assessment result of the multimodal data.

[0164] Optionally, the device further comprises:

[0165] A first building module is used to build a semantic model based on the mapping relationship between the quality assessment fields in the quality assessment database table and the quality assessment terms in the Chinese term library;

[0166] The first extraction module is used to extract pattern information from the semantic model and construct a dictionary and index based on the pattern information to obtain a knowledge base, wherein the pattern information includes the meaning of the quality assessment field, the meaning of the quality assessment term, and the mapping relationship between the quality assessment field and the quality assessment term.

[0167] Optionally, the device further comprises:

[0168] A second receiving module, configured to receive multimodal data and quality processing requirements of the multimodal data sent by a data access terminal;

[0169] a second determination module, configured to input the quality processing requirement into a large model agent, identify a quality assessment type corresponding to the quality processing requirement through the large model agent, determine a basic function matching the quality assessment type from a predefined basic function list, and generate function call information based on the basic function;

[0170] a first evaluation module, configured to call the basic function according to the function call information, perform quality evaluation on the multimodal data according to the quality processing requirement, and obtain a quality evaluation result of the multimodal data;

[0171] The second sending module is used to send the quality assessment result of the multimodal data to the data access terminal.

[0172] Optionally, the device further comprises:

[0173] A first generating module, configured to generate a quality assessment template according to the data content of the multimodal data, the quality processing requirement, the basis function, and the quality assessment result;

[0174] The third storage module is used to store the quality assessment template.

[0175] The present application also provides an electronic device. Since the principle of solving the problem by the electronic device is similar to the data processing method in the present application, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be repeated. Figure 5 As shown, the electronic device in the embodiment of the present application includes: a processor 500, which is used to read the program in the memory 520 and execute the following process: through the transceiver 510:

[0176] Receiving data query requirement information from a user terminal, wherein the data query requirement information includes data quality requirement information;

[0177] The processor 500 is configured to read the program in the memory 520 and execute the following process:

[0178] Inputting the data query demand information into the large model agent, and using the large model agent to parse the data query demand information to obtain unstructured demand information and structured demand information;

[0179] According to the unstructured demand information, querying a pre-built vector database for first data information matching the unstructured demand information, wherein the vector database stores vector data information of multimodal data;

[0180] According to the structured demand information, querying a pre-built structured database for second data information that matches the structured demand information, wherein the structured database stores structured data information of the multimodal data;

[0181] determining target data information according to the first data information and the second data information;

[0182] The processor 500 is configured to read the program in the memory 520 and execute the following process: via the transceiver 510:

[0183] Sending the target data information to the user terminal.

[0184] Among them, Figure 5 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 500 and memory represented by memory 520. The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are all well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 510 may be a plurality of components, i.e., a transmitter and a transceiver, providing a unit for communicating with various other devices on a transmission medium. The processor 500 is responsible for managing the bus architecture and general processing, and the memory 520 may store data used by the processor 500 when performing operations.

[0185] Optionally, the processor 500 is configured to read a program in the memory 520 and execute the following process:

[0186] Demand vector information corresponding to the unstructured demand information is generated, and based on the demand vector information, a vector similarity search is performed through a pre-built vector database to determine first data information matching the demand vector information.

[0187] Optionally, the processor 500 is configured to read a program in the memory 520 and execute the following process:

[0188] A demand structured query statement SQL corresponding to the structured demand information is generated, and based on the demand SQL, a query is performed through a pre-built structured database to determine second data information matching the demand SQL.

[0189] Optionally, the processor 500 is configured to read a program in the memory 520 and execute the following process:

[0190] Performing semantic analysis on the structured demand information to obtain a semantic demand text corresponding to the structured demand information;

[0191] Generate a requirement SQL corresponding to the semantic requirement text;

[0192] The demand SQL is escaped, and based on the escaped demand SQL, a pre-built structured database is searched for second data information that matches the escaped demand SQL.

[0193] Optionally, the processor 500 is configured to read a program in the memory 520 and execute the following process:

[0194] Searching for target information matching the data query requirement information from a pre-built knowledge base, wherein the similarity between the target information and the data query requirement information is greater than a preset value;

[0195] The step of inputting the data query requirement into the large model agent and using the large model agent to parse the data query requirement to obtain unstructured requirement information and structured requirement information includes:

[0196] The data query requirements and target information matching the data query requirements are input into a large model intelligent agent, and the data query requirements and target information matching the data query requirements are parsed by the large model intelligent agent to obtain unstructured demand information and structured demand information.

[0197] Optionally, the processor 500 is configured to read a program in the memory 520 and execute the following process:

[0198] Inputting the multimodal data of the data access terminal into the large model agent, parsing the multimodal data using the large model agent to obtain structured statistical information and unstructured text information of the multimodal data;

[0199] performing vectorization processing on the unstructured text information and the unstructured assessment information of the quality assessment result of the multimodal data to obtain vector data information of the multimodal data, and storing the vector data information in a vector database;

[0200] The structured statistical information and a quality assessment database table corresponding to the structured assessment information of the quality assessment result of the multimodal data are stored in a structured database, and the structured data information of the multimodal data includes the structured statistical information and a quality assessment database table corresponding to the structured assessment information of the quality assessment result of the multimodal data.

[0201] Optionally, the processor 500 is configured to read a program in the memory 520 and execute the following process:

[0202] Building a semantic model based on the mapping relationship between the quality assessment fields in the quality assessment database table and the quality assessment terms in the Chinese term library;

[0203] Extract pattern information from the semantic model, and construct a dictionary and index based on the pattern information to obtain a knowledge base, wherein the pattern information includes the meaning of the quality assessment field, the meaning of the quality assessment term, and the mapping relationship between the quality assessment field and the quality assessment term.

[0204] Optionally, the processor 500 is configured to read a program in the memory 520 and execute the following process: via the transceiver 510:

[0205] receiving multimodal data and quality processing requirements of the multimodal data sent by a data access terminal;

[0206] The processor 500 is configured to read the program in the memory 520 and execute the following process:

[0207] Inputting the quality processing requirement into a large model agent, identifying a quality assessment type corresponding to the quality processing requirement through the large model agent, determining a basic function matching the quality assessment type from a predefined basic function list, and generating function call information according to the basic function;

[0208] Calling the basic function according to the function call information to perform quality assessment on the multimodal data according to the quality processing requirement, thereby obtaining a quality assessment result of the multimodal data;

[0209] The processor 500 is configured to read the program in the memory 520 and execute the following process: via the transceiver 510:

[0210] Sending a quality assessment result of the multimodal data to the data access terminal.

[0211] Optionally, the processor 500 is configured to read a program in the memory 520 and execute the following process:

[0212] generating a quality assessment template according to the data content of the multimodal data, the quality processing requirement, the basis function, and the quality assessment result;

[0213] The quality assessment template is stored.

[0214] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the various processes of the above-mentioned data processing method embodiment and can achieve the same technical effect. To avoid repetition, the details are not described here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0215] The present application also provides a computer program product including computer instructions, which, when executed by a processor, implement the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and to avoid repetition, they will not be described here.

[0216] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0217] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0218] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A data processing method, characterized in that: The method comprises: Receiving data query requirement information from a user terminal, wherein the data query requirement information includes data quality requirement information; Inputting the data query demand information into the large model agent, and using the large model agent to parse the data query demand information to obtain unstructured demand information and structured demand information; According to the unstructured demand information, querying a pre-built vector database for first data information matching the unstructured demand information, wherein the vector database stores vector data information of multimodal data; According to the structured demand information, querying a pre-built structured database for second data information that matches the structured demand information, wherein the structured database stores structured data information of the multimodal data; determining target data information according to the first data information and the second data information; Sending the target data information to the user terminal.

2. The data processing method according to claim 1, wherein: The step of searching a pre-built vector database for first data information matching the unstructured demand information according to the unstructured demand information includes: Demand vector information corresponding to the unstructured demand information is generated, and based on the demand vector information, a vector similarity search is performed through a pre-built vector database to determine first data information matching the demand vector information.

3. The data processing method according to claim 1, wherein: The step of searching a pre-built structured database for second data information matching the structured demand information according to the structured demand information includes: A demand structured query statement SQL corresponding to the structured demand information is generated, and based on the demand SQL, a query is performed through a pre-built structured database to determine second data information matching the demand SQL.

4. The data processing method according to claim 3, wherein: The step of generating a demand structured query statement SQL corresponding to the structured demand information, and performing a query based on the demand SQL through a pre-built structured database to determine second data information matching the demand SQL includes: Performing semantic analysis on the structured demand information to obtain a semantic demand text corresponding to the structured demand information; Generate a requirement SQL corresponding to the semantic requirement text; The demand SQL is escaped, and based on the escaped demand SQL, a pre-built structured database is searched for second data information that matches the escaped demand SQL.

5. The data processing method according to claim 1, wherein: Before inputting the data query demand information into the large model agent and parsing the data query demand information using the large model agent to obtain unstructured demand information and structured demand information, the method further includes: Searching for target information matching the data query requirement information from a pre-built knowledge base, wherein the similarity between the target information and the data query requirement information is greater than a preset value; The step of inputting the data query requirement into the large model agent and using the large model agent to parse the data query requirement to obtain unstructured requirement information and structured requirement information includes: The data query requirements and target information matching the data query requirements are input into a large model intelligent agent, and the data query requirements and target information matching the data query requirements are parsed by the large model intelligent agent to obtain unstructured demand information and structured demand information.

6. The data processing method according to claim 5, characterized in that: Before receiving the user's data query request, the method further includes: Inputting the multimodal data of the data access terminal into the large model agent, parsing the multimodal data using the large model agent to obtain structured statistical information and unstructured text information of the multimodal data; performing vectorization processing on the unstructured text information and the unstructured assessment information of the quality assessment result of the multimodal data to obtain vector data information of the multimodal data, and storing the vector data information in a vector database; The structured statistical information and a quality assessment database table corresponding to the structured assessment information of the quality assessment result of the multimodal data are stored in a structured database, and the structured data information of the multimodal data includes the structured statistical information and a quality assessment database table corresponding to the structured assessment information of the quality assessment result of the multimodal data.

7. The data processing method according to claim 6, characterized in that: After storing the structured statistical information and the quality assessment database table corresponding to the structured assessment information of the quality assessment result of the multimodal data in a structured database, the method further includes: Building a semantic model based on the mapping relationship between the quality assessment fields in the quality assessment database table and the quality assessment terms in the Chinese term library; Extract pattern information from the semantic model, and construct a dictionary and index based on the pattern information to obtain a knowledge base, wherein the pattern information includes the meaning of the quality assessment field, the meaning of the quality assessment term, and the mapping relationship between the quality assessment field and the quality assessment term.

8. The data processing method according to claim 6, characterized in that: Before inputting the multimodal data of the data access terminal into the large model agent and parsing the multimodal data using the large model agent to obtain structured statistical information and unstructured text information of the multimodal data, the method further includes: receiving multimodal data and quality processing requirements of the multimodal data sent by a data access terminal; Inputting the quality processing requirement into a large model agent, identifying a quality assessment type corresponding to the quality processing requirement through the large model agent, determining a basic function matching the quality assessment type from a predefined basic function list, and generating function call information according to the basic function; Calling the basic function according to the function call information to perform quality assessment on the multimodal data according to the quality processing requirement, thereby obtaining a quality assessment result of the multimodal data; Sending a quality assessment result of the multimodal data to the data access terminal.

9. The data processing method according to claim 8, characterized in that: After sending the quality assessment result of the multimodal data to the data access terminal, the method further includes: generating a quality assessment template according to the data content of the multimodal data, the quality processing requirement, the basis function, and the quality assessment result; The quality assessment template is stored.

10. A data processing device, characterized in that: The device comprises: A first receiving module is configured to receive data query requirement information from a user terminal, wherein the data query requirement information includes data quality requirement information; A first parsing module is configured to input the data query demand information into a large model agent, and use the large model agent to parse the data query demand information to obtain unstructured demand information and structured demand information; a first query module, configured to query, based on the unstructured demand information, a pre-built vector database for first data information matching the unstructured demand information, wherein the vector database stores vector data information of multimodal data; a second query module, configured to query a pre-built structured database for second data information matching the structured demand information according to the structured demand information, wherein the structured database stores the structured data information of the multimodal data; A first determining module, configured to determine target data information according to the first data information and the second data information; The first sending module is configured to send the target data information to the user terminal.

11. An electronic device, characterized in that: The electronic device includes a transceiver and a processor, wherein the transceiver is used to receive data query demand information of a user terminal, wherein the data query demand information includes data quality demand information; The processor is configured to: Inputting the data query demand information into the large model agent, and using the large model agent to parse the data query demand information to obtain unstructured demand information and structured demand information; According to the unstructured demand information, querying a pre-built vector database for first data information matching the unstructured demand information, wherein the vector database stores vector data information of multimodal data; According to the structured demand information, querying a pre-built structured database for second data information that matches the structured demand information, wherein the structured database stores structured data information of the multimodal data; determining target data information according to the first data information and the second data information; The transceiver is used for: Sending the target data information to the user terminal.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 9.

Images