Data processing method and device and electronic equipment
By receiving user input non-text modal object files and text modal problem statements, searching the knowledge base and learning association relationships, the problem of large language models generating false answers is solved, and the accuracy and adaptability of answers is improved.
Patent Information
- Application Number
- CN202410918248.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-05-31
- Filing Date
- 2024-07-09
- Publication Date
- 2024-11-08
AI Technical Summary
Large language models are prone to "illusion" when processing user queries, generating false answers, and it is difficult to provide accurate answers.
By receiving user-entered non-text modal object files and text modal problem statements, search the knowledge base of multiple knowledge content, obtain the target knowledge content that matches the problem, and input it into a large language model, and output it after learning the association relationship Answer to reduce hallucination.
It improves the accuracy of large language models for user questions, reduces false answers, and enhances the ability to answer questions in niche and special areas.
Smart Images

Figure CN118916442A_ABST
Abstract
Description
Technical Field
[0001] This document belongs to the field of computer technology, and specifically relates to a data processing method, device and electronic equipment. Background Art
[0002] With the advent of various large language models (LLM), it is not difficult to find that, under the condition of respecting user privacy, large language models have gradually brought many new changes to various industries and made it possible to achieve tasks that were previously unimaginable. However, after experiencing these large language models, it is not difficult to find that we still have a long way to go before we can achieve true general artificial intelligence (AGI).
[0003] An important problem with current large language models is that they have a serious "hallucination" phenomenon. This problem is mainly reflected in the fact that large language models will fabricate some false answers based on user questions. For example, when a user asks a large language model which country issued the currency in a certain image, the large language model may generate an answer by guessing and provide the answer to the user in a very affirmative way. Therefore, a better data processing solution is needed to avoid the "hallucination" phenomenon of large language models. Summary of the invention
[0004] The embodiments of this specification provide a data processing method, device, and electronic device to provide a data processing solution for avoiding the "hallucination" phenomenon of a large language model.
[0005] In a first aspect, an embodiment of the present specification provides a method for processing data, the method comprising: receiving a question input by a user comprising a target file in a non-text mode and a question statement in a text mode, the question statement being a statement asking a question about the target file; retrieving target knowledge content matching the question from a knowledge base comprising multiple pieces of knowledge content, the knowledge content comprising a file in a non-text mode and descriptive information in a text mode that describes the file; inputting the target knowledge content into a large language model to obtain a large language model that has learned the association relationship between the file and the descriptive information in the target knowledge content; inputting the question into the large language model that has learned the association relationship, and using the output of the large language model for the question as the answer corresponding to the question.
[0006] In a second aspect, an embodiment of the present specification provides a data processing device, comprising: a question receiving module, for receiving a question input by a user comprising a target file in a non-text mode and a question statement in a text mode, wherein the question statement is a statement asking a question about the target file; a knowledge retrieval module, for retrieving target knowledge content matching the question from a knowledge base comprising multiple pieces of knowledge content, wherein the knowledge content comprises a file in a non-text mode and descriptive information in a text mode that describes the file; a model learning module, for inputting the target knowledge content into a large language model to obtain a large language model that has learned the association relationship between the file and the descriptive information in the target knowledge content; and a model answering module, for inputting the question into the large language model that has learned the association relationship, and using the output of the large language model for the question as the answer corresponding to the question statement.
[0007] In a third aspect, an embodiment of the present specification provides an electronic device, comprising: a processor, and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor is enabled to: receive a question input by a user comprising a target file in a non-text mode and a question statement in a text mode, wherein the question statement is a statement asking a question about the target file; retrieve target knowledge content matching the question from a knowledge base comprising multiple pieces of knowledge content, wherein the knowledge content comprises a file in a non-text mode and description information in a text mode that describes the file; input the target knowledge content into a large language model to obtain a large language model that has learned the association relationship between the file and the description information in the target knowledge content; input the question into the large language model that has learned the association relationship, and use the output of the large language model for the question as the answer corresponding to the question.
[0008] In a fourth aspect, an embodiment of the present specification provides a storage medium for storing a computer program, wherein the computer program can be executed by a processor to implement the following process: receiving a question input by a user, comprising a target file in a non-text mode and a question statement in a text mode, wherein the question statement is a statement asking a question about the target file; retrieving target knowledge content matching the question from a knowledge base comprising multiple pieces of knowledge content, wherein the knowledge content comprises a file in a non-text mode and descriptive information in a text mode that describes the file; inputting the target knowledge content into a large language model to obtain a large language model that has learned the association relationship between the file and the descriptive information in the target knowledge content; inputting the question into the large language model that has learned the association relationship, and using the output of the large language model for the question as the answer corresponding to the question.
[0009] In a fifth aspect, an embodiment of the present specification provides a computer program product, including a computer program, which implements the following process when executed by a processor: receiving a question input by a user, including a target file in a non-text mode and a question statement in a text mode, wherein the question statement is a statement asking a question about the target file; retrieving target knowledge content matching the question from a knowledge base including multiple knowledge contents, wherein the knowledge content includes a file in a non-text mode and description information in a text mode that describes the file; inputting the target knowledge content into a large language model to obtain a large language model that has learned the association relationship between the file and the description information in the target knowledge content; inputting the question into the large language model that has learned the association relationship, and using the output of the large language model for the question as the answer corresponding to the question. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0011] Figure 1 It is a flowchart of a data processing method provided in an embodiment of this specification;
[0012] Figure 2 It is a schematic diagram of a solution using a large language model provided in an embodiment of this specification;
[0013] Figure 3 It is a schematic diagram of a process of answering a question input by a user provided by an embodiment of this specification;
[0014] Figure 4 It is a schematic diagram of a process of answering a question input by a user provided by an embodiment of this specification;
[0015] Figure 5 It is a schematic diagram of a solution based on a large number of samples to train a small model provided in an embodiment of this specification;
[0016] Figure 6 This is a schematic diagram of an application scenario of a data processing method provided in an embodiment of this specification;
[0017] Figure 7 It is a structural schematic diagram of a data processing device provided in an embodiment of this specification;
[0018] Figure 8It is a structural diagram of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION
[0019] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this document.
[0020] The terms "first", "second", etc. in the specification and claims of this document are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of this specification can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0021] The data processing method, device and electronic device provided in the embodiments of this specification are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.
[0022] Figure 1 A data processing method provided by an embodiment of the present invention is shown. The method can be executed by an electronic device, and the electronic device may include: a server and / or a terminal device, wherein the terminal device may be, for example, a vehicle-mounted terminal or a mobile phone terminal. In other words, the method can be executed by software or hardware installed in the aforementioned electronic device, and the method includes the following steps:
[0023] Step S102: receiving a question input by a user, which includes a target file in a non-text mode and a question statement in a text mode.
[0024] Among them, the question statement is a statement asking questions about the target file. The target file is a file in non-text mode, specifically a file in multimedia form such as pictures, voice and video. Since the question statement in the question input by the user is in text mode and the target file is in non-text mode, the question input by the user is a multimodal question. The number of question statements contained in the question can be one or more, and the content of each question statement can be one question content or multiple question contents, which is not specifically limited in this specification.
[0025] To enable large language models to better grasp the key points in the problem statement, when users input a problem, they can use preset symbols to mark the words and other elements that the user deems important in the problem statement. For example, for the problem statement "Please determine whether this passport has the risk of being PS synthesized", the words "PS synthesized" can be marked with the preset symbol "<>". The problem statement will then become "Please determine whether this passport has the risk of <PS synthesized>".
[0026] To better illustrate this specification and highlight its main idea, the specific embodiments in this specification take the application scenario of processing data based on the image modality as the description object. Those skilled in the art should understand that for other application scenarios of processing data in modalities such as speech and video, this specification can also be implemented in the same or similar ways.
[0027] Step S104: Retrieve the target knowledge content that matches the problem from a knowledge base containing multiple knowledge contents.
[0028] Among them, the knowledge base is a collection of multiple multimodal knowledge contents. Similar to the problem input by the user, the knowledge content can include non-textual modality files and textual modality description information. Then the knowledge base is a multimodal database. The knowledge base can store multimodal data in ways such as A-MMKG and N-MMKG. The specific data storage method of the knowledge base in this specification is not specifically limited and can be determined according to the actual situation.
[0029] The description information corresponding to each knowledge content in the knowledge base can be the annotation information describing the files in the knowledge content, such as information about objects, scenes, entities, attributes, etc. in the files. When constructing a multimodal knowledge base using images, knowledge extraction systems such as NEIL, GAIA, and RESIN for images can be used to extract knowledge (i.e., description information) from the images to annotate the images with the extracted knowledge, obtaining the knowledge content that associates text and image information (including annotation information) corresponding to the image. In addition to knowledge extraction systems, manual methods can also be used to annotate image information.
[0030] Step S106: Input the target knowledge content into a large language model to obtain a large language model that has learned the association relationship between the files and description information in the target knowledge content.
[0031] Step S108: Input the problem into the large language model that has learned the association relationship, and use the output of the large language model for the problem as the answer corresponding to the problem.
[0032] Zero-shot learning is a machine learning technique used to achieve classification, recognition, or information extraction without relevant sample data. Figure 2 A scheme using a large language model is shown. Figure 2 As shown in the figure, the solution adopts zero-shot generation technology. The user directly inputs the question (including the target file and the question sentence) into the large language model to obtain the judgment result of the large language model. The large language model (LLM) is a model pre-trained with large-scale data. It has strong generalization ability but lacks some specific domain knowledge. Therefore, in Figure 2 When the questions input by users belong to specific business scenarios or niche fields, large language models often find it difficult to give accurate answers. Figure 2 The multimodal large model here refers to a large language model that can process and fuse multiple types of input data such as images and texts. Usually, various modalities are tokenized and sent to the large language model for processing, analysis, and output of corresponding results.
[0033] Steps S104-S108 show a new dimension of data processing solution, which is a solution based on contextual inference of the knowledge base: by inputting several pictures and corresponding text content into the large language model, the large language model can understand the relationship between pictures and texts, and then the new pictures are input into the large language model to output the answer corresponding to the question. Combine the user's question with the knowledge base containing knowledge content. Specifically, when the user inputs a question, the question is first matched with the knowledge content in the knowledge base, and the target knowledge content that is more relevant to the question input by the user is queried, and then the target knowledge content is first input into the large language model, so that the large language model learns the relationship between the files and description information in the target knowledge content, understands the user's general intention, and then the question input by the user is input into the large language model to obtain the answer output by the large language model as the answer corresponding to the question input by the user.
[0034] The knowledge base can conveniently provide multimodal knowledge content for large language models. In order to enable large language models to obtain accurate knowledge content that keeps pace with the times and improve user experience, in one example, based on the user's questions and answers, knowledge content related to the user's questions can be obtained from the Internet or related databases, and the relevant knowledge content can be deposited in the knowledge base to achieve real-time updating of the knowledge base.
[0035] The data processing method disclosed in the embodiments of this specification can be used to provide personal virtual assistants, high-tech robots with conversation capabilities, etc. with personalized services to users, so as to respond to questions raised by users in seconds and at low cost around the clock.
[0036] For example, the data processing method of the embodiments of this specification can be applied to the following scenarios:
[0037] Scenario 1: Identify image risks. The image risks may include whether the image is modified or synthesized using network image processing software.
[0038] Figure 3 A schematic diagram of the process of answering questions to identify image risks is shown. Figure 3 As shown in the figure, the process answers the question raised by the user, "Please judge whether this passport has the risk of PS synthesis." Specifically, two pieces of knowledge content matching the question raised by the user can be obtained from the knowledge base first, one piece of knowledge content contains a passport picture taken normally and the description information "normal shooting", and the other piece of knowledge content contains a passport picture synthesized by PS and the description information "PS synthesis". These two pieces of knowledge content are input into the large language model so that the large language model can initially establish the association relationship between "PS synthesis", "normal shooting" and passport pictures. Then, the question raised by the user (including the photo provided by the user) can be input into the large language model that has been fully contextualized (i.e., association relationship learning). This process can obtain a more accurate image risk recognition result, and can provide the user with a more accurate answer.
[0039] Scene 2: Identify object type and locate it.
[0040] Figure 4 FIG. 1 shows a schematic diagram of the process of identifying the object type and answering questions about the location of the object. Figure 4 As shown, the process answers the question raised by the user, "What kind of dog is this picture and where is it?" Specifically, multiple pieces of knowledge content matching the question raised by the user can be obtained from the knowledge base, each piece of knowledge content including a picture of a dog and description information describing the type of dog in the picture, and these pieces of knowledge content are input into the large language model so that the large language model can initially establish the association between the type of dog and the picture of the dog. Then, the question raised by the user (including the photo provided by the user) can be input into the large language model that has learned the above association. This process can obtain more accurate object type recognition results and object location results, and can provide users with more accurate answers.
[0041] For the task of identifying and locating object types in scenario 2, we usually use a large amount of sample data to train a small model. In this way, when we encounter a problem similar to the sample data, we input the problem into the trained small model to get the answer to the problem. Figure 5 A schematic diagram of a solution for training a small model based on a large number of samples is shown. Figure 5As shown in Figure a, during the model training phase, a large number of dog pictures are used to train the dog detector, so that the dog detector has the functions of extracting the location of the dog and identifying the dog species. Figure 5 As shown in Figure b, in the model inference stage, a new image is input into the trained dog detector, and the dog detector can accurately extract the location and type of the dog. However, the disadvantage of this solution is that it requires a lot of data training, and it cannot be implemented for types not included in the training set or other types of tasks (such as identifying cats). Figure 4 The solution in the above example only needs to prepare some knowledge content in the knowledge base, so that the large language model can give a more accurate answer. Figure 5 The solution in is more flexible, does not require a lot of training data, and can adapt to more tasks.
[0042] The above are only examples of application scenarios of the embodiments of this specification and examples of questions raised by users. The data processing method of the embodiments of this specification can also be applied to other scenarios, and the questions raised by users in each scenario can also be in other forms, which is not limited by this specification.
[0043] In the embodiment of the present specification, after receiving the question input by the user, the question is first matched with the knowledge content in the knowledge base, so that the large language model learns the matched target knowledge content, and then the question input by the user is input into the large language model to obtain the answer corresponding to the question. This process enables the large language model to understand the associated knowledge corresponding to the target knowledge content matched with the question input by the user through the combination of the knowledge base and the large language model, greatly reducing the hallucination of the large language model, and improving the accuracy of the answer corresponding to the input question inferred by the large language model. When the large language model faces questions that are difficult to answer directly (such as questions in niche fields or special fields, etc.), it can output accurate replies based on the assistance of the knowledge content deposited in the knowledge base. At the same time, the powerful generalization ability of the large language model can enable the data processing method to quickly adapt to various types of tasks and have high robustness.
[0044] Figure 6 A schematic diagram of an application scenario of a data processing method is provided, such as Figure 6 As shown, the user sends a service request for answering the target question to the server through a terminal such as a mobile phone. After receiving the service request, the server uses the information extraction method in this manual to obtain the answer corresponding to the target question, and sends the obtained answer to the terminal for the user's reference.
[0045] In one implementation, step S104 may be performed as the following steps A1-A3:
[0046] Step A1, based on the question statement, performing a first search in the knowledge base to obtain a first search result matching the question;
[0047] Step A2, based on the target document, performing a second search in the knowledge base to obtain a second search result matching the question;
[0048] Step A3: Determine the target knowledge content matching the question based on the first search result and the second search result.
[0049] When retrieving target knowledge content matching the question input by the user from the knowledge base, a composite search method can be used, that is, the question statement and the target file in the question can be used to retrieve the target knowledge content matching the question input by the user in the knowledge base respectively.
[0050] In one example, a question statement can be used to perform a first search in the knowledge base to obtain a first search result that matches the question; a target file can be used to perform a second search in the knowledge base to obtain a second search result that matches the question. Further, a construction strategy for the target knowledge content can be determined based on the first search result and the second search result to construct the target knowledge content.
[0051] Since the question statement is in text mode, the knowledge content matching the question can be retrieved from the description information of the text mode in each knowledge content in the knowledge base. In one implementation, step A1 can be executed as the following steps B1-B2:
[0052] Step B1, extracting key information from the question statement, where the key information knowledge includes entity information and / or attribute information;
[0053] Step B2: Based on the key information, a first search is performed in the description information contained in each piece of knowledge content in the knowledge base, and the knowledge content to which the description information matching the key information belongs is taken as the first search result.
[0054] Among them, key information knowledge includes entity information and / or attribute information. Entity can represent an actual object, such as Figure 3 Passport and Figure 4 The dog in the picture. Attributes refer to the characteristics of an entity, such as whether the passport is authentic, the location and type of the dog, etc. Attribute information can include attribute names and attribute values. Among them, the attribute name can be used to identify the attribute category, and the attribute value can represent the specific content of the attribute. In an example, the attribute name can be "risk" and the attribute value can be "PS synthesis", "AI generation", etc.
[0055] When performing the first search, the question statement can be disassembled and analyzed first to obtain key information, and then the first search can be performed in the knowledge base based on the key information. For example, when the question statement is "Please determine whether this passport has the risk of PS synthesis", the question statement is first disassembled, and the key information "passport" (entity information), "PS synthesis" (attribute information) and "risk" (entity information) are obtained from the disassembled words. Then, the three key information can be used to perform a first search in the description information of each knowledge content in the knowledge base, and the obtained knowledge content is used as the first search result. The first search method can be to use the obtained key information to search simultaneously in the description information contained in each knowledge content in the knowledge base, or to use the entity information to search first, and then search the attribute information and synonyms of the attribute information in the retrieved description information. This specification does not make specific restrictions on this.
[0056] In one example, extended information that expands the attribute information can be obtained based on the attribute information and the entity information to enrich the key information, and the modified key information can be used to determine the first search result to enrich the first search result, thereby obtaining target knowledge content that enables the large language model to learn more fully. For example, when the entity information is "dog" and the attribute information is "type", extended information such as "Samoyed", "Shiba Inu", and "Border Collie" can be obtained. The extended information expands the attribute information "type" and determines a variety of specific dog types, thereby obtaining a more accurate and rich first search result. Similarly, when the entity information is "passport" and the attribute information is "PS synthesis", extended information such as "normal shooting" can be obtained.
[0057] In one example, a matching value can be determined by the similarity between the key information and the description information in each knowledge content in the knowledge base, and then the matching value is compared with the first matching value to obtain the knowledge content corresponding to the vector matching the key information in the knowledge base, thereby realizing the first search. The first matching value can be determined based on experience, etc.
[0058] Since the target file is in a non-textual mode, for the convenience of the second search, the target file and the files in the knowledge base can be converted into vector form. In one implementation, the knowledge content contains the vector corresponding to the file, and step A2 can be executed as follows: steps C1-C2:
[0059] Step C1, encoding the target file to obtain a target vector corresponding to the target file;
[0060] Step C2: Based on the target vector, a second search is performed in the vectors corresponding to the various pieces of knowledge content in the knowledge base, and the knowledge content corresponding to the vector matching the target vector is taken as the second search result.
[0061] Among them, for each modality of file / target file, the encoder corresponding to the modality can be used for encoding to improve the accuracy of the second search. After receiving the question input by the user, the target file in the question can be encoded to obtain the target vector corresponding to the target file, and then the target vector can be matched with the vectors corresponding to each file in the knowledge base to achieve the second search. This process can quickly locate files similar to the target file in the knowledge base through vectorization, reduce the second search time, and improve the efficiency and accuracy of the second search.
[0062] In one example, a matching value can be determined by the similarity between the target vector and the vectors corresponding to each file in the knowledge base, and then the matching value is compared with the second matching value to obtain the knowledge content corresponding to the vector matching the target vector in the knowledge base, thereby realizing the second search. The second matching value can be determined based on experience, etc.
[0063] Since the non-text modalities in the knowledge content may be voice, image, video, etc., in one implementation, step C2 may be performed as follows: steps D1-D2:
[0064] Step D1, determining candidate knowledge content having the modality of the target document from the knowledge base;
[0065] Step D2, based on the target vector, a second search is performed in the vectors corresponding to the candidate knowledge contents, and the knowledge contents corresponding to the vectors matching the target vector are taken as the second search results.
[0066] Specifically, the candidate knowledge content with the same modality as the target file can be first determined from the knowledge base, and then the file matching the target vector of the target file can be determined from the candidate knowledge content to achieve the second search, and then the knowledge content corresponding to the file matching the target vector of the target file can be used as the second search result. This process narrows the scope of the second search by determining the candidate knowledge content, and improves the efficiency of the second search while improving the accuracy of the second search.
[0067] In the embodiment of this specification, the question statement and the target file are used to search in the knowledge base respectively, and then the target knowledge content is determined according to the first search result and the second search result. This process uses a composite search method to determine the target knowledge content, which improves the accuracy and success rate of the target knowledge content, and then can improve the performance of the large language model learned based on the target knowledge content, and improve the accuracy of the answer corresponding to the user input question.
[0068] In one implementation, step A3 may be performed as the following step E1 or steps E2-E3:
[0069] Step E1, when the first search result and the second search result contain the same knowledge content, the same knowledge content is used as the target knowledge content;
[0070] and / or
[0071] Step E2, when the first search result and the second search result do not contain the same knowledge content, determine the first knowledge content from the first search result, and determine the second knowledge content from the second search result;
[0072] Step E3, obtaining target knowledge content according to the first knowledge content and the second knowledge content.
[0073] The first search result and the second search result may contain the same knowledge content, which indicates that the same knowledge content is the knowledge content in the knowledge base that is more closely matched with the question input by the user. Therefore, when the same knowledge content exists in the first search result and the second search result, the same knowledge content can be used as the target knowledge content so that the large language model can use the same knowledge content to better understand the user's intention. Specifically, the files in the same knowledge content can be used as the files in the target knowledge content, and the description information in the same knowledge content can be used as the description information in the target knowledge content.
[0074] In addition to the above-mentioned situation where the first search result and the second search result have the same knowledge content, the first search result and the second search result may not have the same knowledge content. In this case, a portion of each of the first search result and the second search result can be taken as the target search result, so that the large language model can learn multi-dimensional knowledge content and improve the accuracy of the answer output by the large language model for the user input question. In one implementation, step E2 can be executed as the following steps F1-F3:
[0075] Step F1, when the first search result and the second search result do not contain the same knowledge content, determine the first quantity corresponding to the first knowledge content and the second quantity corresponding to the second knowledge content according to the first search result and the second search result;
[0076] Step F2, determining a first quantity of first knowledge contents from the first search results;
[0077] Step F3, determining a second quantity of second knowledge contents from the second search results.
[0078] In the case where the same knowledge content does not exist in the first search result and the second search result, the two can be combined to construct the target knowledge content. Specifically, a portion of the knowledge content can be taken from each of the first search result and the second search result to construct the target knowledge content. In one example, the first quantity of target knowledge content in the first search result and the second quantity of target knowledge content in the second search result can be determined based on the first search result and the second search result. For example, in the case where the description information in each knowledge content included in the first search result is relatively matched with the key information, and the files in each knowledge content in the second search result are not quite matched with the target files, the first quantity can be set to be greater than the second quantity. In one implementation, step F1 can be executed as the following steps G1-G2:
[0079] Step G1, when the first search result and the second search result do not contain the same knowledge content, determine a first matching result between the question statement and the description statement of each knowledge content in the first search result, and determine a second matching result between the target file and the file of each knowledge content in the second search result;
[0080] Step G2: Determine the first quantity and the second quantity according to the first matching result and the second matching result.
[0081] Specifically, the number of target knowledge contents contained in the first search result and the second search result can be set in the strategy for constructing the target knowledge contents. In one example, the first preset number can be determined based on the matching degree between the files of each knowledge content in the first search result and the target file; the second preset number can be determined based on the matching degree between the description text of each knowledge content in the second search result and the target description text.
[0082] Furthermore, the first search result and the second search result may be combined to determine the first preset number and the second preset number. Specifically, after sorting the first search result and the second search result according to the matching degree, a candidate number of first knowledge contents may be selected from the sorted first search result, and a candidate number of second knowledge contents may be selected from the sorted second search result; the matching degrees corresponding to the first knowledge contents may be added to obtain the first matching degree, and the matching degrees corresponding to the second knowledge contents may be added to obtain the second matching degree; the first preset number and the second preset number may be determined based on the first matching degree and the second matching degree. In one example, the ratio of the first preset number and the second preset number may be determined based on the ratio of the first matching degree and the second matching degree, and then the ratio may be multiplied by the candidate number to determine the first preset number. After obtaining the first preset number, the second preset number may be determined based on the ratio and the first preset number.
[0083] After determining the first preset number and the second preset number, the first target knowledge content can be determined according to the first preset number, and the second target knowledge content can be determined according to the second preset number. Finally, the first target knowledge content and the second target knowledge content can be combined to obtain the target knowledge content.
[0084] In the embodiments of the present specification, different target knowledge content construction strategies are set for the specific contents of the first search result and the second search result, so as to obtain target knowledge content that better matches the first search result and the second search result, improve the learning effect of the large language model from the target knowledge content, and further improve the effect of the large language model outputting the answer to the question input by the user, thereby improving the experience of the user who inputs the question.
[0085] It should be noted that the data processing method provided in the embodiments of this specification may be executed by a data processing device or a control module in the data processing device for executing the data processing method. In the embodiments of this specification, the data processing device provided in the embodiments of this specification is described by taking the data processing device executing the data processing method as an example.
[0086] Figure 7 Schematic diagram of the structure of a data processing device according to an embodiment of the present invention. Figure 7 As shown, the data processing device 700 includes:
[0087] The question receiving module 710 is used to receive a question input by a user, including a target file in a non-text mode and a question statement in a text mode, where the question statement is a statement asking a question about the target file;
[0088] The knowledge retrieval module 720 is used to retrieve target knowledge content matching the question from a knowledge base containing multiple pieces of knowledge content, wherein the knowledge content includes files in non-text mode and description information in text mode describing the files;
[0089] Model learning module 730, used to input the target knowledge content into the large language model to obtain the large language model that has learned the association relationship between the files and description information in the target knowledge content;
[0090] The model answering module 740 is used to input the question into the large language model that has learned the association relationship, and use the output of the large language model for the question as the answer corresponding to the question sentence.
[0091] In one embodiment, the knowledge retrieval module 720 includes:
[0092] A first retrieval unit, configured to perform a first search in the knowledge base based on the question statement to obtain a first retrieval result matching the question;
[0093] A second search unit, configured to perform a second search in the knowledge base based on the target document to obtain a second search result matching the question;
[0094] The matching unit is used to determine the target knowledge content matching the question according to the first search result and the second search result.
[0095] In one embodiment, the first retrieval unit is used to:
[0096] Extract key information from the question sentence, where the key information knowledge includes entity information and / or attribute information;
[0097] Based on the key information, a first search is performed in the description information contained in each piece of knowledge content in the knowledge base, and the knowledge content to which the description information matching the key information belongs is taken as the first search result.
[0098] In one embodiment, the knowledge content includes a vector corresponding to the file, and the second retrieval unit is used to:
[0099] Encode the target file to obtain the target vector corresponding to the target file;
[0100] Based on the target vector, a second search is performed in the vectors corresponding to the various pieces of knowledge content in the knowledge base, and the knowledge content corresponding to the vector matching the target vector is taken as the second search result.
[0101] In one embodiment, based on the target vector, a second search is performed in the vectors corresponding to each piece of knowledge content in the knowledge base, and the knowledge content corresponding to the vector matching the target vector is used as the second search result, including:
[0102] From the knowledge base, determining candidate knowledge content having the modality of the target document;
[0103] Based on the target vector, a second search is performed in the vectors corresponding to the candidate knowledge contents, and the knowledge contents corresponding to the vectors matching the target vector are taken as the second search results.
[0104] In one embodiment, the matching unit is used to:
[0105] When the first search result and the second search result contain the same knowledge content, the same knowledge content is used as the target knowledge content;
[0106] and / or
[0107] In the case where the first search result and the second search result do not contain the same knowledge content, the first knowledge content is determined from the first search result, and the second knowledge content is determined from the second search result;
[0108] According to the first knowledge content and the second knowledge content, target knowledge content is obtained.
[0109] In one embodiment, when the first search result and the second search result do not contain the same knowledge content, determining the first knowledge content from the first search result and determining the second knowledge content from the second search result includes:
[0110] In the case that the first search result and the second search result do not contain the same knowledge content, determining a first quantity corresponding to the first knowledge content and a second quantity corresponding to the second knowledge content according to the first search result and the second search result;
[0111] Determining a first number of first knowledge contents from the first search results;
[0112] From the second search results, a second amount of second knowledge content is determined.
[0113] In one embodiment, when the first search result and the second search result do not contain the same knowledge content, determining the first quantity corresponding to the first knowledge content and the second quantity corresponding to the second knowledge content according to the first search result and the second search result includes:
[0114] In the case that the first search result and the second search result do not contain the same knowledge content, determining a first matching result between the question statement and the description statement of each knowledge content in the first search result, and determining a second matching result between the target file and the file of each knowledge content in the second search result;
[0115] A first quantity and a second quantity are determined according to the first matching result and the second matching result.
[0116] It should be noted that the embodiments of the data processing device in this specification and the embodiments of the data processing method in this specification are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the corresponding data processing method mentioned above, and the repeated parts will not be repeated.
[0117] Each module in the above data processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the terminal device or the processor on the server side in the form of hardware, or can be stored in the memory in the terminal device or the memory on the server side in the form of software, so that the processor can call and execute the operations corresponding to each module above.
[0118] Furthermore, corresponding to the data processing method described above, based on the same technical concept, one or more embodiments of this specification also provide an electronic device, which is used to execute the data processing method described above. Figure 8A schematic diagram of the structure of an electronic device provided for one or more embodiments of this specification.
[0119] Based on the same idea, one or more embodiments of this specification also provide an electronic device, such as Figure 8 As shown. The electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 801 and memory 802, and the memory 802 may store one or more storage applications or data. Among them, the memory 802 can be a temporary storage or a permanent storage. The application stored in the memory 802 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 801 can be configured to communicate with the memory 802 to execute a series of computer executable instructions in the memory 802 on the electronic device. The electronic device may also include one or more power supplies 803, one or more wired or wireless network interfaces 804, one or more input and output interfaces 805, and one or more keyboards 806.
[0120] In a specific embodiment, the electronic device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions for the electronic device, and the one or more programs configured to be executed by one or more processors include the following computer executable instructions:
[0121] receiving a question input by a user including a target file in a non-text mode and a question sentence in a text mode, wherein the question sentence is a sentence asking a question about the target file;
[0122] Retrieving target knowledge content matching the question from a knowledge base containing multiple pieces of knowledge content, wherein the knowledge content includes a file in a non-textual mode and description information in a textual mode describing the file;
[0123] Inputting the target knowledge content into a large language model to obtain a large language model that has learned the association between the files and description information in the target knowledge content;
[0124] The question is input into a large language model that has learned the association relationship, and the output of the large language model for the question is used as the answer to the question.
[0125] It should be noted that the embodiments of the electronic device in this specification and the embodiments of the data processing method in this specification are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the corresponding data processing method mentioned above, and the repeated parts will not be repeated.
[0126] Furthermore, corresponding to the data processing method described above, based on the same technical concept, one or more embodiments of this specification further provide a storage medium for storing computer executable instructions. In a specific embodiment, the storage medium may be a USB flash drive, an optical disk, a hard disk, etc. When the computer executable instructions stored in the storage medium are executed by the processor, the following process can be implemented:
[0127] receiving a question input by a user including a target file in a non-text mode and a question sentence in a text mode, wherein the question sentence is a sentence asking a question about the target file;
[0128] Retrieving target knowledge content matching the question from a knowledge base containing multiple pieces of knowledge content, wherein the knowledge content includes a file in a non-textual mode and description information in a textual mode describing the file;
[0129] Inputting the target knowledge content into a large language model to obtain a large language model that has learned the association between the files and description information in the target knowledge content;
[0130] The question is input into a large language model that has learned the association relationship, and the output of the large language model for the question is used as the answer to the question.
[0131] It should be noted that the embodiments of the storage medium in this specification and the data processing method in this specification are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the corresponding data processing method mentioned above, and the repeated parts will not be repeated.
[0132] Furthermore, corresponding to the data processing method described above, based on the same technical concept, one or more embodiments of this specification further provide a computer program product, the computer program product includes a computer program, and when the computer program is executed by a processor, the following process can be implemented:
[0133] receiving a question input by a user including a target file in a non-text mode and a question sentence in a text mode, wherein the question sentence is a sentence asking a question about the target file;
[0134] Retrieving target knowledge content matching the question from a knowledge base containing multiple pieces of knowledge content, wherein the knowledge content includes a file in a non-textual mode and description information in a textual mode describing the file;
[0135] Inputting the target knowledge content into a large language model to obtain a large language model that has learned the association between the files and description information in the target knowledge content;
[0136] The question is input into a large language model that has learned the association relationship, and the output of the large language model for the question is used as the answer to the question.
[0137] It should be noted that the embodiments of the computer program product in this specification and the embodiments of the data processing method in this specification are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the corresponding data processing method mentioned above, and the repeated parts will not be repeated.
[0138] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0139] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.
[0140] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.
[0141] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0142] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0143] It should be understood by those skilled in the art that one or more embodiments of this specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0144] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0145] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0146] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0147] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0148] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0149] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0150] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0151] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0152] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0153] The above description is only an embodiment of this document and is not intended to limit this document. For those skilled in the art, this document may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this document should be included in the scope of the claims of this document.
Claims
1. A data processing method, comprising: receiving a question input by a user including a target file in a non-text mode and a question statement in a text mode, wherein the question statement is a statement asking a question about the target file; Retrieving target knowledge content matching the question from a knowledge base containing multiple pieces of knowledge content, wherein the knowledge content includes a file in a non-textual mode and description information in a textual mode describing the file; Inputting the target knowledge content into a large language model to obtain a large language model that has learned the association relationship between the files and description information in the target knowledge content; The question is input into a large language model that has learned the association relationship, and the output of the large language model for the question is used as an answer corresponding to the question.
2. The method according to claim 1, wherein the step of retrieving target knowledge content matching the question from a knowledge base containing multiple pieces of knowledge content comprises: Based on the question statement, a first search is performed in the knowledge base to obtain a first search result matching the question; Based on the target document, a second search is performed in the knowledge base to obtain a second search result matching the question; According to the first search result and the second search result, target knowledge content matching the question is determined.
3. The method according to claim 2, wherein the first search is performed in the knowledge base based on the question statement to obtain a first search result matching the question, comprising: Extract key information from the question statement, where the key information knowledge includes entity information and / or attribute information; Based on the key information, a first search is performed in the description information included in each piece of the knowledge content in the knowledge base, and the knowledge content to which the description information matching the key information belongs is taken as the first search result.
4. The method according to claim 2, wherein the knowledge content includes a vector corresponding to a file, and the second search is performed in the knowledge base based on the target file to obtain a second search result matching the question, comprising: Performing encoding processing on the target file to obtain a target vector corresponding to the target file; Based on the target vector, a second search is performed in the vectors corresponding to the pieces of knowledge content in the knowledge base, and the knowledge content corresponding to the vector matching the target vector is used as the second search result.
5. The method according to claim 4, wherein based on the target vector, a second search is performed in the vectors corresponding to the pieces of knowledge content in the knowledge base, and the knowledge content corresponding to the vector matching the target vector is used as the second search result, comprising: Determining candidate knowledge content having the modality of the target document from the knowledge base; Based on the target vector, a second search is performed in the vectors corresponding to the candidate knowledge contents, and the knowledge contents corresponding to the vectors matching the target vector are taken as the second search results.
6. The method according to claim 3, wherein determining the target knowledge content matching the question according to the first search result and the second search result comprises: In the case where the first search result and the second search result contain the same knowledge content, taking the same knowledge content as the target knowledge content; and / or In the case where the first search result and the second search result do not contain the same knowledge content, determining the first knowledge content from the first search result and determining the second knowledge content from the second search result; The target knowledge content is obtained according to the first knowledge content and the second knowledge content.
7. The method according to claim 6, wherein when the first search result and the second search result do not contain the same knowledge content, determining the first knowledge content from the first search result and determining the second knowledge content from the second search result comprises: In a case where the first search result and the second search result do not include the same knowledge content, determining a first quantity corresponding to the first knowledge content and a second quantity corresponding to the second knowledge content according to the first search result and the second search result; Determining a first quantity of the first knowledge content from the first search results; A second amount of the second knowledge content is determined from the second search results.
8. The method according to claim 7, wherein when the first search result and the second search result do not contain the same knowledge content, determining the first quantity corresponding to the first knowledge content and the second quantity corresponding to the second knowledge content according to the first search result and the second search result comprises: In the case that the first search result and the second search result do not contain the same knowledge content, determining a first matching result between the question statement and the description statement of each knowledge content in the first search result, and determining a second matching result between the target file and the file of each knowledge content in the second search result; The first quantity and the second quantity are determined according to the first matching result and the second matching result.
9. A data processing device, comprising: A question receiving module, used for receiving a question input by a user, including a target file in a non-text mode and a question statement in a text mode, wherein the question statement is a statement asking a question about the target file; A knowledge retrieval module, used to retrieve target knowledge content matching the question from a knowledge base containing multiple pieces of knowledge content, wherein the knowledge content includes a file in a non-textual mode and description information in a textual mode describing the file; A model learning module, used for inputting the target knowledge content into a large language model to obtain a large language model that has learned the association relationship between the files and description information in the target knowledge content; The model answering module is used to input the question into a large language model that has learned the association relationship, and use the output of the large language model for the question as the answer corresponding to the question statement.
10. An electronic device comprising: processor, and a memory arranged to store computer executable instructions which, when executed, cause the processor to: receiving a question input by a user including a target file in a non-text mode and a question statement in a text mode, wherein the question statement is a statement asking a question about the target file; Retrieving target knowledge content matching the question from a knowledge base containing multiple pieces of knowledge content, wherein the knowledge content includes a file in a non-textual mode and description information in a textual mode describing the file; Inputting the target knowledge content into a large language model to obtain a large language model that has learned the association relationship between the files and description information in the target knowledge content; The question is input into a large language model that has learned the association relationship, and the output of the large language model for the question is used as an answer corresponding to the question statement.
Citation Information
Patent Citations
Knowledge question and answer method, device and equipment and storage medium
CN116561276A
Large model image-text dialogue method and device, equipment and storage medium
CN117633173A
Interactive knowledge interaction system based on knowledge base and large language model
CN118013001A
Cited By
User question response method and system, computer equipment and readable storage medium
CN119202142A
Human-computer interaction method and device based on body-equipped intelligent agent and body-equipped intelligent agent
CN119376549A
