Data processing method and device, equipment, storage medium and program product
By segmenting and semantically matching user input questions, and combining the similarity calculation between a large language model and a table database, the shortcomings of large language models in cross-table question answering are solved, and effective processing of multi-table questions is achieved.
Patent Information
- Application Number
- CN202511688767.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-17
- Publication Date
- 2026-02-27
AI Technical Summary
Large language model-driven table-based question answering cannot effectively output when dealing with multi-table questions, limiting its practical application in the field of complex data question answering.
By segmenting the user-input question information into words and semantically matching it with a pre-configured lexicon, the second question information is determined. After normalization and vectorization, similarity is calculated with a pre-configured table database. A large language model is then used to perform multi-table queries to generate response information.
It enables effective answers to questions spanning multiple tables, improving the application capabilities of large-scale language models in the field of complex data question answering.
Smart Images
Figure CN121579632A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and in particular to a data processing method, apparatus, device, storage medium, and program product. Background Technology
[0002] Large-scale language model-driven table-based question answering primarily focuses on query processing for single-table structured data. However, this single-table question answering paradigm has significant limitations when handling business queries, as it can only answer questions on a single table. When table-based question answering questions require answers across multiple tables, it cannot provide effective output. This single-table question answering approach hinders the practical application of large-scale language models in the domain of complex data question answering.
[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0004] This disclosure provides a data processing method, apparatus, device, storage medium, and program product that at least to some extent overcomes the limitations of related technologies where large language model-driven table-based question answering is mainly designed for single tables and cannot effectively handle questions that require answers across multiple tables.
[0005] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.
[0006] According to one aspect of this disclosure, a data processing method is provided, comprising: performing word segmentation on first question information input by a user and semantic matching with a pre-configured lexicon to determine second question information; normalizing and vectorizing the second question information and calculating similarity with a pre-configured table database to determine entities that meet a similarity threshold and a standardized query template, wherein the pre-configured table database includes multiple tables, and the standardized query template includes a structured query language and a skeleton question; inputting the first question information, entities, and standardized query template into a first large language model to determine multi-table query information; and inputting the first question information and multi-table query information into a second large language model to determine response information.
[0007] In some embodiments, before performing word segmentation on the first question information input by the user and semantic matching with a pre-configured thesaurus to determine the second question information, the method further includes: performing sensitive word verification on the first question information; if the sensitive word verification of the first question information passes, determining whether the first question information has undergone structured query language interaction; if so, performing word segmentation on the first question information input by the user and semantic matching with a pre-configured thesaurus to determine the second question information.
[0008] In some embodiments, the pre-configured table library includes: normalizing and vectorizing each table to determine the vectorized table, and calculating the association information between each table; outputting the vectorized table and the association information between the tables in a structured format to determine the pre-configured table library.
[0009] In some embodiments, normalizing and vectorizing each table to determine a vectorized table and calculating the association information between each table includes: extracting the entity and skeleton question of each table; normalizing and vectorizing the entity and skeleton question of each table to determine a vectorized table; and calculating the vectors of different tables using cosine similarity to determine the association information between each table.
[0010] In some embodiments, inputting the first question information, entities, and standardized query template into the first large language model to determine the multi-table query information includes: inputting the first question information, entities, and standardized query template into the first large language model and outputting a multi-table structured query language; verifying the security of the multi-table structured query language; if the multi-table structured query language passes the verification, determining whether the multi-table structured query language is executable; if so, executing the multi-table structured query language to determine the multi-table query information.
[0011] In some embodiments, inputting multi-table query information and first question information into a second large language model to determine response information includes: inputting multi-table query information and first question information into a second large language model and outputting natural language; determining whether the transmission method of natural language is streaming; if yes, adding multi-table query information after the last tag return of natural language to determine it as response information; if no, concatenating natural language and multi-table query information into response information.
[0012] According to another aspect of this disclosure, a data processing apparatus is also provided, comprising: a semantic matching module, used to perform word segmentation processing on first question information input by a user and semantic matching with a pre-configured lexicon to determine second question information; a similarity calculation module, used to normalize and vectorize the second question information and perform similarity calculation with a pre-configured table library to determine entities that meet a similarity threshold and a standardized query template, wherein the pre-configured table library includes multiple tables, and the standardized query template includes a structured query language and a skeleton question; a multi-table query information determination module, used to input the first question information, entities, and standardized query template into a first large language model to determine multi-table query information; and a response information determination module, used to input the first question information and multi-table query information into a second large language model to determine response information.
[0013] According to another aspect of this disclosure, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the data processing method described in any of the preceding claims by executing the executable instructions.
[0014] According to another aspect of this disclosure, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the data processing method described in any of the preceding claims.
[0015] According to another aspect of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the data processing method described in any of the above.
[0016] The data processing method provided in the embodiments of this disclosure involves: segmenting the first question information input by the user and semantically matching it with a pre-configured lexicon to determine the second question information; normalizing and vectorizing the second question information and calculating its similarity with a pre-configured table database to determine entities that meet the similarity threshold and standardized query templates. The pre-configured table database includes multiple tables, and the standardized query templates include structured query language and skeleton questions. The first question information, entities, and standardized query templates are input into a first large language model to determine multi-table query information; the first question information and multi-table query information are input into a second large language model to determine response information. This application, based on a large language model, effectively answers cross-table questions by deeply parsing user questions and combining them with a table database generated through multi-table preprocessing. This addresses the problem that large language model-driven table-based question answering primarily targets single tables and cannot effectively handle cross-table questions.
[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.
[0019] Figure 1 This diagram illustrates a data processing system structure according to an embodiment of the present disclosure. Figure 2 This diagram illustrates a data processing method according to an embodiment of the present disclosure. Figure 3A flowchart illustrating a specific example of a data processing method according to an embodiment of this disclosure is shown. Figure 4 A flowchart illustrating another specific example of a data processing method according to an embodiment of the present disclosure; Figure 5 A flowchart illustrating yet another specific example of a data processing method according to an embodiment of the present disclosure; Figure 6 A flowchart illustrating another specific example of a data processing method according to an embodiment of this disclosure; Figure 7 A flowchart illustrating another specific example of a data processing method according to an embodiment of this disclosure is shown; Figure 8 This diagram illustrates a data processing apparatus according to an embodiment of the present disclosure. Figure 9 A structural block diagram of a computer device according to an embodiment of the present disclosure is shown. Detailed Implementation
[0020] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0021] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.
[0022] To facilitate understanding, before introducing the embodiments of this disclosure, the following explanations are provided for several terms involved in the embodiments of this disclosure: Large Language Models (LLM); Table Question Answering (Table QA); Structured Query Language (SQL).
[0023] The specific implementation methods of the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.
[0024] Figure 1 A schematic diagram of an exemplary application system architecture to which the data processing methods of the embodiments of this disclosure can be applied is shown. For example... Figure 1 As shown, the system architecture may include terminal device 101, network 102 and server 103.
[0025] Network 102 is a medium used to provide a communication link between terminal device 101 and server 103, and can be a wired network or a wireless network.
[0026] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to Local Area Networks (LANs), Metropolitan Area Networks (MANs), Wide Area Networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats, including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPSec) can be used to encrypt all or some links. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.
[0027] Terminal device 101 can be various electronic devices, including but not limited to smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, wearable devices, augmented reality devices, virtual reality devices, etc.
[0028] Optionally, the client of the application installed on different terminal devices 101 may be the same, or the client of the same type of application based on different operating systems. Depending on the terminal platform, the specific form of the application client may also be different; for example, the application client may be a mobile client, a PC client, etc.
[0029] Server 103 can be a server that provides various services, such as a backend management server that supports the device operated by the user using terminal device 101. The backend management server can analyze and process received requests and other data, and feed the processing results back to the terminal device.
[0030] Optionally, the server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.
[0031] In one example of this disclosure, the server segments the first question information input by the user through a terminal device and performs semantic matching with a pre-configured lexicon to determine the second question information. The second question information is then normalized and vectorized, and its similarity is calculated with a pre-configured table database to determine entities that meet the similarity threshold and standardized query templates. The pre-configured table database includes multiple tables, and the standardized query templates include a structured query language and a skeleton question. The first question information, entities, and standardized query templates are input into a first large language model to determine multi-table query information. Finally, the first question information and multi-table query information are input into a second large language model to determine the response information.
[0032] Those skilled in the art will know that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative; any number of terminal devices, networks, and servers can be included depending on actual needs. This disclosure does not limit the scope of the embodiments.
[0033] Figure 2 A flowchart of a data processing method according to an embodiment of this disclosure is shown, such as Figure 2 As shown, the data processing method provided in this embodiment includes the following steps: S202, after processing the first question information input by the user through word segmentation, semantically match it with the pre-configured lexicon to determine the second question information.
[0034] It should be noted that the aforementioned first question information can be multimodal. For example, multimodal input reception can receive a user's mixed input of voice, text, or images, and use speech recognition and optical character recognition technologies to convert the non-text input into a structured text question. The aforementioned word segmentation process can segment the text question into words, generate a vocabulary array, and perform semantic similarity matching between the vocabulary and a keyword vector database (equivalent to the aforementioned pre-configured vocabulary) to obtain second question information that optimizes the expression of the user's intent, thus accurately recognizing the intent of the user's first question information input.
[0035] S204. After normalizing and vectorizing the information of the second question, perform similarity calculation with the pre-configured table database to determine the entities that meet the similarity threshold and the standardized query template. The pre-configured table database includes multiple tables, and the standardized query template includes structured query language and skeleton question.
[0036] It should be noted that the normalization mentioned above can refer to the process of converting text into a standard form, which can eliminate surface differences in textual information. For example, normalization may include eliminating redundant noise: removing characters that do not affect semantics (such as punctuation and special symbols); unifying format: for example, converting all letters to lowercase; and simplifying structure: for example, removing stop words. Vectorization can refer to converting data into a numerical vector representation that is easy for computers to process. Similarity calculation can refer to calculating the cosine similarity between the normalized and vectorized information of the second question and a pre-configured database of tables. For example, the calculated cosine similarity is compared with a preset threshold; if it is greater than the preset threshold, it indicates a high similarity between the two. Connectivity queries between highly correlated tables and compatibility between two tables can be represented as the most joinable column pairs between the two tables.
[0037] In a specific instance, if there is a cosine similarity score greater than 0.9, the first index value is extracted, and the entity and standardized query template corresponding to the index value are obtained; if the cosine similarity scores are all less than 0.9, the top three index values are extracted, and the entities and standardized query templates corresponding to the top three index values are obtained.
[0038] S206, input the first problem information, entity and standardized query template into the first major language model to determine the multi-table query information.
[0039] It should be noted that the first major language model mentioned above can be a deep learning model. The multi-table query information mentioned above can be a structured query language retrieved from multiple related tables.
[0040] S208: Input the first question information and multi-table query information into the second language model to determine the response information.
[0041] It should be noted that the second major language model mentioned above can be a deep learning model. The response information mentioned above can be the reply information output in response to the user's input of the first question.
[0042] This application is based on a large language model. By deeply parsing user questions and combining them with a table library generated by multi-table preprocessing, it can effectively answer questions that cross multiple tables. This solves the problem that large language model-driven table question answering mainly targets single tables and cannot effectively handle questions that cross multiple tables.
[0043] In one embodiment of this disclosure, such as Figure 3 As shown, the data processing method provided in this embodiment of the disclosure includes the following steps before determining the second problem information, which not only proactively intercepts risks and prevents sensitive content from entering core business logic, but also reduces resource consumption and improves response speed: S302, perform sensitive word verification on the first question information.
[0044] S304, when the sensitive word verification of the first question information passes, determine whether the first question information should be used for structured query language interaction.
[0045] It should be noted that if the first issue fails the sensitive word verification, an exception will be thrown and the process will end.
[0046] S306, If so, the first question information input by the user is segmented and semantically matched with the pre-configured dictionary to determine the second question information.
[0047] It should be noted that if the first question is not answered using structured query language, the conversation will end after a routine dialogue.
[0048] In one embodiment of this disclosure, such as Figure 4 As shown, the data processing method provided in this embodiment of the present disclosure for constructing a pre-configured table database includes the following steps, which can express multiple tables in a structured and unified manner, thereby improving query efficiency: S402, normalize and vectorize each table, determine the vectorized table, and calculate the relationship information between each table.
[0049] It should be noted that the normalization of the tables mentioned above can be interpreted as normalizing the table information structure: extracting and normalizing the structural information of each table (table name, field names, field types, foreign key relationships, etc.). The vectorization of the tables mentioned above can be interpreted as vectorizing the table information: using an embedding model to vectorize the structural information of the entire table, calculating the degree of association between tables (cosine similarity), and identifying foreign key relationships.
[0050] S404 outputs the vectorized tables and the relationships between the tables in a structured format to determine the pre-configured table library.
[0051] It should be noted that the above-mentioned output of vectorized tables and the relationship information between tables in a structured format can be structured table information: outputting the vectorized table information and the relationship between tables in a structured JSON format for use by the neural network model. The calculation of the relationship information between the tables can be performed by performing connectivity queries on the relationships between highly correlated tables, and the compatibility between two tables can be represented as the most joinable column pairs between the two tables. In a specific example, the table relationship matrix (equivalent to the above-mentioned relationship information) is shown in Table 1 below: Table 1
[0052] In a specific example, the database has three tables storing different business data: 1. User information table, including user ID and user name; 2. Order record table, including order ID, user ID, product ID, and order date; 3. Product information table, including product ID and product name. When a user asks "to find the names of the electronic products purchased by user 'Zhang San' this year," a single table cannot provide the answer; all three tables must be joined to obtain a complete result.
[0053] In one embodiment of this disclosure, such as Figure 5 As shown, the data processing method provided in this embodiment of the disclosure includes the following steps: normalizing and vectorizing each table, determining the vectorized table, and calculating the association information between each table. S502, extract the entities and skeleton questions for each table; S504, normalize and vectorize the entities and skeleton questions of each table to determine the vectorized table; S506 calculates the vectors of different tables using cosine similarity to determine the association information between the tables.
[0054] For example, because the same entity may have different names in different tables, queries may fail to obtain accurate results. This application addresses this by extracting entities and constructing a global mapping to eliminate semantic conflicts across tables. Furthermore, by creating an efficient vector index from entities and skeleton questions, and then normalizing and vectorizing them, this application enables fast and accurate retrieval of the database.
[0055] In one embodiment of this disclosure, such as Figure 6 As shown, the data processing method provided in this embodiment of the disclosure, which inputs the first problem information, entity, and standardized query template into the first large language model to determine multi-table query information, includes the following steps: S602: Input the first problem information, entity and standardized query template into the first large language model, and output a multi-table structured query language; S604, verify the security of multi-table structured query language; S606: When the multi-table structured query language validation passes, determine whether the multi-table structured query language is executable. S608, if so, execute Multi-Table Structured Query Language to determine multi-table query information.
[0056] It should be noted that the aforementioned language security may include any one or more of the following: syntax validation, checking the correctness of the multi-table structured query language structure; and permission validation, verifying user data access permissions. The aforementioned executable functionality may include any one or more of the following: table existence; field type matching. Inputting the first question information, entity, and standardized query template into the first large language model may involve concatenating the first question information, entity, and standardized query template as prompt words and inputting them into the first large language model. Executing the multi-table structured query language may involve executing the structured query language in a pre-configured table database to obtain the query results.
[0057] This disclosure avoids single points of failure by decoupling security verification and feasibility assessment; it also forms a closed-loop optimization by feeding back the execution results of the structured query language to train the large language model, and continuously iterates to correct and optimize the execution of the user's structured query language until the expected operational results are achieved.
[0058] In one embodiment of this disclosure, such as Figure 7 As shown, the data processing method provided in this embodiment of the disclosure inputs multi-table query information and first question information into a second large language model to determine the response information, including the following steps: S702 inputs the multi-table query information and the first question information into the second language model and outputs natural language. S704, determine whether natural language is being streamed; S706, if it is streaming, add multi-table query information to determine the response information after the last tag of the natural language is returned; S708, if not streaming, combines natural language and multi-table query information into a response message.
[0059] In this application, different processing is performed according to different types of natural language, which can improve transmission efficiency.
[0060] Based on the same inventive concept, this disclosure also provides a data processing apparatus, as described in the following embodiments. Since the principle by which this apparatus solves the problem is similar to that of the method embodiments described above, the implementation of this apparatus embodiment can refer to the implementation of the method embodiments described above, and repeated details will not be repeated.
[0061] Figure 8This diagram illustrates a data processing apparatus according to an embodiment of the present disclosure, such as... Figure 8 As shown, the device includes: a semantic matching module 81, a similarity calculation module 82, a multi-table query information determination module 83, and a response information determination module 84.
[0062] The semantic matching module is used to perform word segmentation on the first question information input by the user and then perform semantic matching with a pre-configured word vector library to determine the second question information.
[0063] The similarity calculation module is used to perform semantic normalization on the second question information and then perform similarity calculation with a pre-configured database to determine entities that meet the similarity threshold and standardized query templates. The pre-configured database includes multiple tables, and the standardized query templates include structured query language and skeleton questions.
[0064] The multi-table query information determination module is used to input the first question information, entities, and standardized query templates into the first language model to determine the multi-table query information.
[0065] The response information determination module is used to input the first question information and multi-table query information into the second language model to determine the response information.
[0066] In one example of this disclosure, the data processing device further includes a verification module for performing sensitive word verification on the first question information; when the sensitive word verification of the first question information passes, it is determined whether the first question information is used for structured query language interaction; if so, the first question information input by the user is processed by word segmentation and semantically matched with a pre-configured word vector library to determine the second question information.
[0067] In one example of this disclosure, the pre-configured table library in the aforementioned data processing apparatus includes: normalizing and vectorizing each table to determine the vectorized table, and calculating the association information between each table; outputting the vectorized table and the association information between the tables in a structured format to determine the pre-configured table library.
[0068] In one example of this disclosure, the aforementioned similarity calculation module is further used to: extract the entities and skeleton questions of each table; normalize and vectorize the entities and skeleton questions of each table to determine the vectorized table; and calculate the vectors of different tables using cosine similarity to determine the association information between the tables.
[0069] In one example of this disclosure, the aforementioned multi-table query information determination module is further configured to: input the first question information, entity, and standardized query template into the first large language model, and output the multi-table structured query language; verify the security of the multi-table structured query language; when the multi-table structured query language passes the verification, determine whether the multi-table structured query language is executable; if so, execute the multi-table structured query language to determine the multi-table query information.
[0070] In one example of this disclosure, the above-mentioned response information determination module is further configured to: input multi-table query information and first question information into a second large language model and output natural language; determine whether the natural language is streaming; if it is streaming, add multi-table query information after the last tag return of the natural language to determine it as response information; if it is not streaming, combine the natural language and multi-table query information into response information.
[0071] It should be noted that the semantic matching module 81, similarity calculation module 82, multi-table query information determination module 83, and response information determination module 84 mentioned above correspond to S202 to S208 in the method embodiment. The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above method embodiment. It should be noted that the above modules, as part of the apparatus, can be executed in a computer system such as a set of computer-executable instructions.
[0072] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."
[0073] The following reference Figure 9 To describe an electronic device 900 according to such an embodiment of the present disclosure. Figure 9 The electronic device 900 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.
[0074] like Figure 9 As shown, the electronic device 900 is manifested in the form of a general-purpose computing device. The components of the electronic device 900 may include, but are not limited to: at least one processing unit 910, at least one storage unit 920, and a bus 930 connecting different system components (including storage unit 920 and processing unit 910).
[0075] The storage unit stores program code that can be executed by the processing unit 910, causing the processing unit 910 to perform the steps described in the "Exemplary Methods" section above according to various exemplary embodiments of this disclosure.
[0076] For example, the processing unit 910 can execute the following steps of the above method embodiment: after performing word segmentation on the first question information input by the user, perform semantic matching with a pre-configured word vector library to determine the second question information; after semantic normalization on the second question information, perform similarity calculation with a pre-configured table library to determine entities that meet the similarity threshold and standardized query templates, wherein the pre-configured table library includes multiple tables, and the standardized query template includes structured query language and skeleton questions; input the first question information, entities, and standardized query templates into a first large language model to determine multi-table query information; input the first question information and multi-table query information into a second large language model to determine response information.
[0077] For example, the processing unit 910 can perform the following steps in the above method embodiment: perform sensitive word verification on the first question information; when the sensitive word verification of the first question information passes, determine whether the first question information is subject to structured query language interaction; if so, perform word segmentation processing on the first question information input by the user and semantic matching with the pre-configured word vector library to determine the second question information.
[0078] For example, the processing unit 910 can perform the following steps in the above method embodiment: normalize and vectorize each table, determine the vectorized table, and calculate the association information between each table; output the vectorized table and the association information between the tables in a structured format, and determine the pre-configured table library.
[0079] For example, the processing unit 910 can perform the following steps in the above method embodiment: extract the entities and skeleton questions of each table; normalize and vectorize the entities and skeleton questions of each table to determine the vectorized table; calculate the vectors of different tables by using cosine similarity to determine the association information between the tables.
[0080] For example, the processing unit 910 can perform the following steps in the above method embodiment: inputting the first problem information, entity and standardized query template into the first large language model, and outputting a multi-table structured query language; verifying the security of the multi-table structured query language; when the multi-table structured query language passes the verification, determining whether the multi-table structured query language is executable; if so, executing the multi-table structured query language to determine the multi-table query information.
[0081] For example, the processing unit 910 can perform the following steps in the above method embodiment: inputting multi-table query information and first question information into the second language model and outputting natural language; determining whether the natural language is streaming; if it is streaming, adding multi-table query information after the last tag return of the natural language to determine it as response information; if it is not streaming, concatenating the natural language and multi-table query information into response information.
[0082] Storage unit 920 may include readable media in the form of volatile storage units, such as random access memory (RAM) 9201 and / or cache 9202, and may further include read-only memory (ROM) 9203.
[0083] The storage unit 920 may also include a program / utility 9204 having a set (at least one) program module 9205, such program module 9205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.
[0084] Bus 930 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.
[0085] Electronic device 900 can also communicate with one or more external devices 940 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 900, and / or with any device that enables electronic device 900 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 950. Furthermore, electronic device 900 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 960. As shown, network adapter 960 communicates with other modules of electronic device 900 via bus 930. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0086] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.
[0087] In particular, according to embodiments of this disclosure, the process described above with reference to the flowchart can be implemented as a computer program product, which includes a computer program that, when executed by a processor, implements the above-described data processing method.
[0088] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the methods described above is stored thereon. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.
[0089] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0090] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.
[0091] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.
[0092] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java and C++, and conventional procedural programming languages such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0093] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.
[0094] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.
[0095] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.
[0096] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.
Claims
1. A data processing method, characterized in that, include: The first question information input by the user is segmented into words and then semantically matched with a pre-configured dictionary to determine the second question information; After normalizing and vectorizing the second question information, similarity is calculated with a pre-configured table database to determine entities that meet the similarity threshold and standardized query templates. The pre-configured table database includes multiple tables, and the standardized query template includes a structured query language and a skeleton question. Input the first question information, the entity, and the standardized query template into the first large language model to determine multi-table query information; Input the first question information and the multi-table query information into the second language model to determine the response information.
2. The data processing method according to claim 1, characterized in that, Before performing word segmentation on the first question information input by the user and semantic matching with a pre-configured lexicon to determine the second question information, the method further includes: Perform sensitive word verification on the first problem information; Once the sensitive word verification of the first question information passes, determine whether the first question information should be used for structured query language interaction; If so, the first question information entered by the user is segmented and semantically matched with a pre-configured thesaurus to determine the second question information.
3. The data processing method according to claim 1, characterized in that, The pre-configured table database includes: Normalize and vectorize each table to determine the vectorized table, and calculate the relationship information between each table; The vectorized tables and the association information between the tables are output in a structured format to determine the pre-configured table library.
4. The data processing method according to claim 3, characterized in that, The process of normalizing and vectorizing each table, determining the vectorized table, and calculating the association information between the tables includes: Extract the entities and skeleton questions from each table; The entities and skeleton questions of each table are normalized and vectorized to determine the vectorized table; By calculating the vectors of different tables using cosine similarity, the association information between the tables can be determined.
5. The data processing method according to claim 1, characterized in that, The step of inputting the first question information, the entity, and the standardized query template into the first large language model to determine multi-table query information includes: Input the first question information, the entity, and the standardized query template into the first large language model, and output a multi-table structured query language; Verify the security of the multi-table structured query language; Once the multi-table structured query language passes the validation, determine whether the multi-table structured query language is executable. If so, execute the multi-table structured query language to determine the multi-table query information.
6. The data processing method according to claim 1, characterized in that, The multi-table query information and the first question information are input into the second large language model to determine the response information, including: Input the multi-table query information and the first question information into the second large language model, and output natural language; Determine whether the transmission method of the output natural language is streaming; If so, add the multi-table query information as the response information after the last tag of the natural language is returned; If not, combine the natural language with the multi-table query information to form the response information.
7. A data processing apparatus, characterized in that, include: The semantic matching module is used to perform word segmentation on the first question information input by the user and semantic matching with a pre-configured lexicon to determine the second question information; The similarity calculation module is used to normalize and vectorize the second question information and perform similarity calculation with a pre-configured table database to determine entities that meet the similarity threshold and standardized query templates. The pre-configured table database includes multiple tables, and the standardized query template includes a structured query language and a skeleton question. The multi-table query information determination module is used to input the first question information, the entity, and the standardized query template into the first large language model to determine the multi-table query information. The response information determination module is used to input the first question information and the multi-table query information into the second language model to determine the response information.
8. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the data processing method of any one of claims 1 to 6 by executing the executable instructions.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data processing method according to any one of claims 1 to 6.
10. A computer program product comprising: A computer program or instruction, characterized in that, when executed by a processor, the computer program or instruction implements the data processing method according to any one of claims 1 to 6.