Intelligent question answering method and device, electronic equipment, storage medium and program product

Abstract

The present disclosure relates to an intelligent question answering method and device, an electronic device, a storage medium and a program product. The method comprises: obtaining input question text; based on the question text, searching a target knowledge graph to obtain a search result; the target knowledge graph is a knowledge graph constructed based on multi-modal data, the search result comprises a plurality of search contents related to the question text and relationship information between different search contents, and the search content comprises image information or named entities represented by a target node in the target knowledge graph; and based on the question text and the search result, generating answer information of the question text. The method fully utilizes the context information of the multi-modal data through the target knowledge graph, realizes efficient and accurate search, improves the accuracy of the search, thereby improving the ability to answer complex global questions, and supports the search of multi-modal data, not limited to natural language, and reduces the limitations of searchable data.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to an intelligent question-answering method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] Intelligent question-answering systems are a typical application scenario in the fields of artificial intelligence and natural language processing. These systems ask and answer questions in natural language. Generally, intelligent question-answering systems use deep semantic coding networks to perform semantic embedding calculations on the input natural language question to obtain a vector representation of the question. Then, they calculate the semantic similarity between this vector representation and the vector representations of data in a database. Based on the semantic similarity calculation results, they return the most relevant data from the database to the question, and finally generate the answer based on the returned data.

[0003] Current intelligent question-answering systems cannot fully utilize contextual information for data types such as long documents, are prone to missing search content, have poor search accuracy, struggle to answer complex global questions, and do not support the retrieval of non-natural language data, thus having limitations. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides an intelligent question-answering method, device, electronic device, storage medium, and program product.

[0005] According to a first aspect of the present disclosure, an intelligent question-answering method is provided, the method comprising:

[0006] Get the input question text;

[0007] Based on the question text, a search is performed on the target knowledge graph to obtain search results; the target knowledge graph is a knowledge graph constructed based on multimodal data, and the search results include multiple search contents related to the question text and information on the relationships between different search contents; the search contents include image information or named entities represented by target nodes in the target knowledge graph.

[0008] Based on the question text and the search results, answer information for the question text is generated.

[0009] In some embodiments, the target knowledge graph is managed by a preset database, and the step of retrieving the target knowledge graph based on the question text to obtain retrieval results includes:

[0010] The question text is subjected to intent recognition and slot extraction to obtain intent recognition results and slot extraction results; the intent recognition results are used to characterize the intent of the question text, and the slot extraction results include words extracted from the question text that are associated with the intent recognition results;

[0011] Based on the intent recognition result and the slot extraction result, the question text is converted into a search statement supported by the preset database;

[0012] Based on the search query, the target knowledge graph is searched to obtain the search results.

[0013] In some embodiments, the process of performing intent recognition and slot extraction on the question text to obtain intent recognition results and slot extraction results includes:

[0014] The question text is segmented into words to obtain segmentation information; the segmentation information includes the words in the question text.

[0015] The natural language understanding model is invoked to process the word segmentation information to obtain the intent recognition result and the slot extraction result; the natural language understanding model is used to perform intent recognition and slot extraction on the input information.

[0016] In some embodiments, the natural language understanding model includes an encoder, an intent recognition network, and a slot extraction network. The step of invoking the natural language understanding model to process the word segmentation information and obtain the intent recognition result and the slot extraction result includes:

[0017] The encoder is invoked to encode the word segmentation information, thereby obtaining the encoded features of the word segmentation information;

[0018] The intent recognition network is invoked to perform intent recognition on the encoded features, and the intent recognition result is obtained.

[0019] The slot extraction network is invoked to extract slots from the encoded features, thereby obtaining the slot extraction result.

[0020] In some embodiments, the training process of the natural language understanding model includes:

[0021] Acquire training data, which includes sample information and the sample intent recognition results and sample slot extraction results corresponding to the sample information;

[0022] The sample information is input into the natural language understanding model to be trained, and the predicted intent recognition result and predicted slot extraction result are output.

[0023] The first loss value is determined based on the first loss function, the sample intent recognition result, and the predicted intent recognition result;

[0024] The second loss value is determined based on the second loss function, the sample slot extraction result, and the predicted slot extraction result;

[0025] Based on the first loss value, the second loss value, the first weight corresponding to the first loss value, and the second weight corresponding to the second loss value, the target loss value is determined.

[0026] Based on the target loss value, the model parameters of the natural language understanding model to be trained are adjusted to obtain the natural language understanding model.

[0027] In some embodiments, the target knowledge graph includes a first type of node, which is used to represent named entities;

[0028] The process of retrieving the target knowledge graph based on the retrieval statement to obtain the retrieval results includes:

[0029] For the first type of node, based on the search statement, the first target node, the second target node, and the connection relationship between the first target node and the second target node are retrieved from the target knowledge graph; the similarity between the named entity represented by the first target node and the words contained in the search statement is greater than a first preset threshold, and the second target node is a node connected to the first target node;

[0030] The first named entity represented by the first target node, the second named entity represented by the second target node, and the relationship information represented by the connection relationship between the first target node and the second target node are determined as the retrieval result.

[0031] In some embodiments, the target knowledge graph includes a second type of node, which is used to represent a preset image;

[0032] The process of retrieving the target knowledge graph based on the retrieval statement to obtain the retrieval results includes:

[0033] For the second type of node, multiple candidate nodes are retrieved from the target knowledge graph based on the retrieval statement; the similarity between the label information of the preset image represented by the candidate node and the words contained in the retrieval statement is greater than a second preset threshold.

[0034] Based on the search query, a third target node is determined from the plurality of candidate nodes; the similarity between the vector representation of the preset image represented by the third target node and the vector representation of the search query is greater than a third preset threshold.

[0035] The image information of the preset image represented by the third target node is determined as the retrieval result.

[0036] In some embodiments, the method further includes:

[0037] When the multimodal data includes preset text, obtain information about multiple named entities in the preset text and the relationships between different named entities in the multiple named entities;

[0038] When the multimodal data includes a preset image, image information of the preset image is acquired, and relationship information between the preset image and a target named entity is determined; the target named entity is the named entity associated with the preset image among the plurality of named entities;

[0039] The target knowledge graph is constructed based on multiple named entities in the preset text, the relationship information between the different named entities, the image information of the preset image, and the relationship information between the preset image and the target named entity.

[0040] In some embodiments, obtaining information about multiple named entities in the preset text and the relationships between different named entities includes:

[0041] The preset text is split into multiple sentences to obtain multiple sentences in the preset text;

[0042] Named entity recognition is performed on each of the plurality of statements to obtain the plurality of named entities;

[0043] Based on the multiple statements and the multiple named entities, relationship extraction is performed to determine the relationship information between different named entities among the multiple named entities.

[0044] In some embodiments, constructing the target knowledge graph based on multiple named entities in the preset text, the relationship information between the different named entities, the image information of the preset image, and the relationship information between the preset image and the target named entity includes:

[0045] Create a first type of node to represent the named entity and a second type of node to represent the preset image; the node content of the first type of node is the named entity it represents, and the node content of the second type of node is the image information of the preset image it represents.

[0046] Based on the relationship information between the different named entities, and the relationship information between the preset image and the target named entity, a connection relationship between different nodes is created;

[0047] The target knowledge graph is constructed based on the created first-type nodes, second-type nodes, and the connection relationships between different nodes.

[0048] In some embodiments, generating answer information for the question text based on the question text and the search results includes:

[0049] The search results and the question text are concatenated to obtain contextual information;

[0050] The context information is input into the large language model, and the answer information is output.

[0051] According to a second aspect of the present disclosure, a smart question-answering device is provided, the device comprising:

[0052] The question retrieval module is configured to retrieve the input question text;

[0053] The retrieval module is configured to perform a retrieval on the target knowledge graph based on the question text to obtain retrieval results; the target knowledge graph is a knowledge graph constructed based on multimodal data, and the retrieval results include multiple retrieval contents related to the question text and information on the relationships between different retrieval contents; the retrieval contents include image information or named entities represented by target nodes in the target knowledge graph.

[0054] The answer generation module is configured to generate answer information for the question text based on the question text and the search results.

[0055] In some embodiments, the retrieval module is configured to:

[0056] The question text is subjected to intent recognition and slot extraction to obtain intent recognition results and slot extraction results; the intent recognition results are used to characterize the intent of the question text, and the slot extraction results include words extracted from the question text that are associated with the intent recognition results;

[0057] Based on the intent recognition result and the slot extraction result, the question text is converted into a search statement supported by the preset database;

[0058] Based on the search query, the target knowledge graph is searched to obtain the search results.

[0059] In some embodiments, the retrieval module is configured to:

[0060] The question text is segmented into words to obtain segmentation information; the segmentation information includes the words in the question text.

[0061] The natural language understanding model is invoked to process the word segmentation information to obtain the intent recognition result and the slot extraction result; the natural language understanding model is used to perform intent recognition and slot extraction on the input information.

[0062] In some embodiments, the apparatus further includes a training module, the training module being configured to:

[0063] Acquire training data, which includes sample information and the sample intent recognition results and sample slot extraction results corresponding to the sample information;

[0064] The sample information is input into the natural language understanding model to be trained, and the predicted intent recognition result and predicted slot extraction result are output.

[0065] The first loss value is determined based on the first loss function, the sample intent recognition result, and the predicted intent recognition result;

[0066] The second loss value is determined based on the second loss function, the sample slot extraction result, and the predicted slot extraction result;

[0067] Based on the first loss value, the second loss value, the first weight corresponding to the first loss value, and the second weight corresponding to the second loss value, the target loss value is determined.

[0068] Based on the target loss value, the model parameters of the natural language understanding model to be trained are adjusted to obtain the natural language understanding model.

[0069] In some embodiments, the target knowledge graph includes a second type of node, which is used to represent a preset image; the retrieval module is configured to:

[0070] For the second type of node, multiple candidate nodes are retrieved from the target knowledge graph based on the retrieval statement; the similarity between the label information of the preset image represented by the candidate node and the words contained in the retrieval statement is greater than a second preset threshold.

[0071] Based on the search query, a third target node is determined from the plurality of candidate nodes; the similarity between the vector representation of the preset image represented by the third target node and the vector representation of the search query is greater than a third preset threshold.

[0072] The image information of the preset image represented by the third target node is determined as the retrieval result.

[0073] In some embodiments, the apparatus further includes a knowledge graph construction module, which is configured to:

[0074] When the multimodal data includes preset text, obtain information about multiple named entities in the preset text and the relationships between different named entities in the multiple named entities;

[0075] When the multimodal data includes a preset image, image information of the preset image is acquired, and relationship information between the preset image and a target named entity is determined; the target named entity is the named entity associated with the preset image among the plurality of named entities;

[0076] The target knowledge graph is constructed based on multiple named entities in the preset text, the relationship information between the different named entities, the image information of the preset image, and the relationship information between the preset image and the target named entity.

[0077] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0078] processor;

[0079] Memory used to store processor-executable instructions;

[0080] The processor is configured to execute the intelligent question-answering method as described in the first aspect of the embodiments of this disclosure.

[0081] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the intelligent question-answering method as described in the first aspect of the present disclosure.

[0082] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the intelligent question-answering method as described in the first aspect of the present disclosure.

[0083] The method described above, as disclosed in this invention, has the following beneficial effects:

[0084] The method provided in this disclosure constructs a target knowledge graph based on multimodal data. The target knowledge graph more comprehensively represents the relationships between various data. During the retrieval process, the target knowledge graph is retrieved based on the input question text. The contextual information of the multimodal data is fully utilized to achieve efficient and accurate retrieval, thereby improving the accuracy of the retrieval. Then, answer information is generated based on the question text and the retrieved search results, thereby improving the ability to answer complex global questions. Furthermore, it can support the retrieval of multimodal data, not limited to natural language, reducing the limitation of searchable data.

[0085] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0086] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0087] Figure 1 This is a flowchart illustrating an intelligent question-answering method according to an exemplary embodiment.

[0088] Figure 2 This is a flowchart illustrating a knowledge graph construction process according to an exemplary embodiment.

[0089] Figure 3 This is a schematic diagram illustrating a portion of the nodes and connections in a target knowledge graph, according to an exemplary embodiment.

[0090] Figure 4 This is a flowchart illustrating an intelligent question-answering method according to an exemplary embodiment.

[0091] Figure 5 This is a flowchart illustrating a model training process according to an exemplary embodiment.

[0092] Figure 6 This is a flowchart illustrating an intelligent question-answering method according to an exemplary embodiment.

[0093] Figure 7 This is a schematic diagram of an intelligent question-answering system according to an exemplary embodiment.

[0094] Figure 8 This is a schematic diagram illustrating an application scenario according to an exemplary embodiment.

[0095] Figure 9 This is a block diagram illustrating an intelligent question-answering device according to an exemplary embodiment.

[0096] Figure 10 This is a block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0097] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0098] In related technologies, intelligent question-answering systems generally employ deep semantic coding networks to perform semantic embedding calculations on the input natural language questions to obtain a vector representation of the question. Then, they calculate the semantic similarity between this question's vector representation and the vector representations of data in the database. Based on the semantic similarity calculation results, they return the most relevant data from the database to the question, and finally generate the answer based on the returned data. Current intelligent question-answering systems, however, cannot fully utilize contextual information for long documents and similar data types, easily missing retrieved content, exhibiting poor retrieval accuracy, difficulty in answering complex global questions, and lack support for retrieving non-natural language data, thus possessing limitations.

[0099] The method provided in this disclosure involves: acquiring input question text; searching a target knowledge graph based on the question text to obtain search results; the target knowledge graph being a knowledge graph constructed based on multimodal data; and the search results including multiple search contents related to the question text and information about the relationships between different search contents, with search contents including image information or named entities; and generating answer information for the question text based on the question text and the search results. During the search process, the method fully utilizes the contextual information of the multimodal data based on the constructed target knowledge graph to achieve efficient and accurate retrieval, improving search accuracy and thus enhancing the ability to answer complex global questions. Furthermore, it supports the retrieval of multimodal data, not limited to natural language, reducing the limitation of searchable data.

[0100] The method provided in this disclosure is executed by an electronic device, which may be a mobile phone, tablet computer, laptop computer, wearable device, vehicle terminal, smart home device, etc.

[0101] Figure 1 This is a flowchart illustrating an intelligent question-answering method according to an exemplary embodiment, executed by an electronic device. See also... Figure 1 The method includes the following steps:

[0102] Step S101: Obtain the input question text.

[0103] The question text is a question in natural language input by the user, which may include text and punctuation marks.

[0104] In some embodiments, the question text can be of two types. One type is generative question text, which requests the generation of content related to the question text. The searched content needs to be summarized and displayed to the user. For example, the question text could be "Please help me generate a summary based on the meeting-related documents I received yesterday." The other type is searchable question text, which requests the search for content related to the question text. The searched content can be directly displayed to the user as an answer. For example, the question text could be "Where does Xiaoming live?".

[0105] In some embodiments, the question text is the text entered by the user in the search box, and the text entered by the user in the search box is retrieved in response to a detected search operation.

[0106] Step S102: Based on the question text, a search is performed on the target knowledge graph to obtain the search results. The target knowledge graph is a knowledge graph constructed based on multimodal data. The search results include multiple search contents related to the question text and the relationship information between different search contents. The search contents include image information or named entities represented by the target nodes in the target knowledge graph.

[0107] The multimodal data can include documents, images, audio, video, and other data. For example, an image can be any image, and may or may not contain text; a document can be any document, and may include text, images, tables, etc. Of course, multimodal data can also be other types of data, and this disclosure does not limit this.

[0108] Optionally, for documents, named entities can be extracted from the text, and nodes can be created based on these entities; for images and videos containing multiple video frames (images), nodes can be created based on the images; for audio, the audio can first be recognized to obtain the corresponding text, and then the named entities in the corresponding text can be extracted, and nodes can be created based on these named entities. Furthermore, the connections between different nodes can be determined based on the relationships between different named entities and the relationships between each named entity and an image, thereby creating a corresponding target knowledge graph for multimodal data.

[0109] During the retrieval process, because the information contained in the multimodal data is integrated into the target knowledge graph in the form of nodes and connections, and because the relationships between named entities and images in the multimodal data have been analyzed during the construction of the knowledge graph, the contextual information of the multimodal data can be fully utilized to achieve efficient and accurate retrieval based on the question text, thus improving retrieval accuracy. The retrieval results include multiple search terms related to the question text and the relationship information between different search terms. The search terms include image information or named entities represented by target nodes in the target knowledge graph, and the relationship information between different search terms is the connection relationship between different target nodes.

[0110] Step S103: Based on the question text and the search results, generate the answer information for the question text.

[0111] Since search results may not be a complete sentence or may be a mixture of many search terms, directly presenting them to the user may be difficult for them to understand or result in poor presentation. Therefore, after obtaining the search results, or further based on the question text and search results, an answer that is easy for the user to understand and view can be generated. This answer can include various forms of information such as text, documents, and images. This embodiment of the disclosure does not limit the form of the answer. For example, if the question text is "Please help me generate a summary based on the meeting-related documents I received yesterday," and the search results include "Participant A said ×××, Participant B said ×××, Participant C said ×××," the search results only provide the statements of each participant. However, the user wants a summary of the meeting content. Therefore, the answer can be generated based on the question text and search results as "The meeting content was ×××, and the meeting focus was ×××."

[0112] The method provided in this disclosure constructs a target knowledge graph based on multimodal data. The target knowledge graph more comprehensively represents the relationships between various data. During the retrieval process, the target knowledge graph is retrieved based on the input question text. The contextual information of the multimodal data is fully utilized to achieve efficient and accurate retrieval, thereby improving the accuracy of the retrieval. Then, answer information is generated based on the question text and the retrieved search results, thereby improving the ability to answer complex global questions. Furthermore, it can support the retrieval of multimodal data, not limited to natural language, reducing the limitation of searchable data.

[0113] In this embodiment of the disclosure, a target knowledge graph needs to be constructed before starting the retrieval. The following describes... Figure 2 The illustrated example demonstrates the process of constructing a target knowledge graph.

[0114] Figure 2 This is a flowchart illustrating a knowledge graph construction process according to an exemplary embodiment, performed by an electronic device, see [link to flowchart]. Figure 2 The method includes the following steps:

[0115] Step S201: When the multimodal data includes preset text, obtain information about multiple named entities in the preset text and the relationships between different named entities among the multiple named entities.

[0116] The preset text can be text contained in a document, document title, text converted from audio, etc. The named entities in the preset text can be key entities such as names of people, places, times, and events. The relationship information between different named entities is words that represent the relationship between different named entities. For example, for the named entities "Xiaoming" and "City A", the relationship information between "Xiaoming" and "City A" can be words such as "born in" and "residing in" that indicate the relationship between "Xiaoming" and "City A".

[0117] In some embodiments, the preset text is split into sentences to obtain multiple sentences; named entity recognition is performed on each sentence to obtain multiple named entities in each sentence; named entity recognition is performed on each of the multiple sentences to obtain multiple named entities; and relationship extraction is performed based on the multiple sentences and multiple named entities to determine the relationship information between different named entities. That is, when the preset text is long and contains a lot of content, by identifying multiple named entities from the preset text and extracting the relationship information between different named entities, some useless words in the preset text can be removed, reducing the amount of subsequent processing, and allowing for a more thorough analysis of the preset text content.

[0118] Optionally, a sentence segmentation tool for natural language processing can be used to segment the preset text to obtain multiple sentences.

[0119] Optionally, a named entity recognition model is used to perform named entity recognition on each statement to obtain multiple named entities in each statement. The named entity recognition model is used to identify named entities in the statement, and this model can be a BERT model (Bidirectional Encoder Representations from Transformers, a pre-trained language model based on Transformers), a large language model, or other models.

[0120] In some embodiments, the acquired multiple named entities and the relationship information between different named entities are stored in the form of triples. The triples are in the form of (named entity 1, relation, named entity 2). For example, for the text "Xiaoming's home is in city A", the named entities "Xiaoming" and "city A" and the relationship information "resides in" can be extracted. The triple can be represented as (Xiaoming, resides in, city A). Of course, other forms can also be used to store the acquired multiple named entities and the relationship information between different entities. This disclosure does not limit this.

[0121] Step S202: When the multimodal data includes a preset image, obtain the image information of the preset image and determine the relationship information between the preset image and the target named entity.

[0122] The preset image can be an image stored in an electronic device, a video frame from a video, an image from a document, etc. The image information includes at least one of the following: label information, description information, vector representation, and text within the preset image. The label information indicates the object contained in the preset image; the description information describes the content of the preset image; the vector representation is the feature obtained after feature extraction of the preset image; and the text is the text obtained after OCR (Optical Character Recognition) recognition of the preset image. Text is not mandatory; the preset image may or may not contain text. For example, an exemplary preset image might have the label information "dog" and "grass," meaning the preset image contains a dog and grass; the description information is "a dog lying on the grass"; the vector representation is [0.321, -0.201, 0.231, ...]; and the text is none, meaning the preset image contains no text.

[0123] The target named entity is the named entity associated with the preset image among multiple named entities. Optionally, the target named entity can be a named entity related to the label information of the preset image. For example, if the label information of the preset image is "dog," then the target named entity can be a named entity related to dogs, such as dog, pet, or animal. Optionally, the target named entity can be a named entity related to the description information of the preset image. For example, if the label information of the preset image is "a dog lying on the grass," then the target named entity can be the location "grass." Of course, other methods can also be used to determine the named entity associated with the preset image.

[0124] After identifying the target named entities, it is necessary to further determine the relationship information between the preset image and the target named entities. The relationship information between the preset image and different target named entities can be the same or different. Based on the above example, the relationship information between the preset image and the target named entity "pet" can be represented as "RELATED_TO", which means that the preset image is related to a pet. The relationship information between the preset image and the target named entity "grass" can be represented as "LOCATED_IN", which means that the dog is lying on the grass.

[0125] In some embodiments, to align text and image data, a CLIP (Contrastive Language-Image Pre-Training) encoder can be used to encode a preset image to obtain a vector representation of the preset image. This vector representation can be 512-dimensional, 1024-dimensional, or other dimensional; this embodiment of the present disclosure does not impose any limitations on this. Of course, other encoders can also be used to encode the preset image; this embodiment of the present disclosure does not impose any limitations on this either.

[0126] In this embodiment, the tag information and description information included in the image information are helpful in identifying named entities associated with the preset image. The vector representation included in the image information represents richer content and can more comprehensively describe the content of the preset image compared to tag information and description information. For example, if the preset image is a photo of a golden retriever, but the tag information and description information only provide the description of a dog, and the user's question text is "golden retriever," the preset image may not be retrieved based solely on the tag information and description information, but it can be retrieved based on the vector representation. The text in the preset image included in the image information can be used as the retrieved text, providing more and more accurate contextual information for subsequently generating answer information.

[0127] Step S203: Construct a target knowledge graph based on multiple named entities in the preset text, the relationship information between different named entities, the image information of the preset image, and the relationship information between the preset image and the target named entity.

[0128] When constructing a target knowledge graph, it is necessary to create nodes and the connection relationships between different nodes. In this embodiment of the disclosure, a corresponding node is created for each named entity and preset image.

[0129] In some embodiments, a first type of node is created to represent named entities and a second type of node is created to represent a preset image. The node content of the first type of node is the named entity it represents, and the node content of the second type of node is the image information of the preset image it represents. Based on the relationship information between different named entities and the relationship information between the preset image and the target named entity, connection relationships between different nodes are created. Based on the created first type of node, second type of node, and connection relationships between different nodes, a target knowledge graph is constructed. The connection relationships between different nodes can also be referred to as edges.

[0130] See one example. Figure 3 The diagram shown illustrates some nodes and connections in the target knowledge graph. Node1 is created for a preset image. The preset image's labels are "dog" and "grass," indicating that the image contains a dog and grass. The description is "a dog lying on the grass," represented by a vector [0.321, -0.201, 0.231, ...]. The text is empty. Node2 is created for the named entity "pet," with a connection between node1 and node2 of "RELATED_TO." Node3 is created for the named entity "grass," with a connection between node1 and node3 of "LOCATED_IN."

[0131] It should be noted that after the target knowledge graph is constructed, since multimodal data may be updated, electronic devices can update the target knowledge graph during non-retrieval periods to ensure that the latest data can be retrieved in the next retrieval process. The method of creating nodes and connections when updating the target knowledge graph is similar to the implementation method described above.

[0132] In some embodiments, after the target knowledge graph is constructed, a preset database is used to store and manage the target knowledge graph. Optionally, considering that data updates in electronic devices may be rapid, the preset database can be a graph database or a Neo4j distributed graph database to support rapid incremental updates of nodes and connections. Of course, other types of databases can also be used to store and manage the target knowledge graph.

[0133] In this embodiment of the disclosure, nodes corresponding to preset text and preset images are created for multimodal data, and connection relationships between different nodes are created to construct a target knowledge graph. This ensures that the content of multimodal data is fully explored. By constructing a target knowledge graph, the hierarchical relationships and background information between the content of multimodal data are enriched. Compared with directly calculating the semantic similarity between multimodal data and question text, using a target knowledge graph can achieve more accurate retrieval during the retrieval process.

[0134] In some embodiments, after obtaining the input question text, in order to fully understand the user's question and intent during subsequent retrieval, the question text can be processed before searching the target knowledge graph. The following describes how... Figure 4 The illustrated embodiments illustrate this.

[0135] Figure 4 This is a flowchart illustrating an intelligent question-answering method according to an exemplary embodiment, executed by an electronic device. See also... Figure 4 The method includes the following steps:

[0136] Step S401: Perform intent recognition and slot extraction on the question text to obtain intent recognition results and slot extraction results.

[0137] The intent recognition result is used to characterize the intent of the question text, which may include generating an answer corresponding to the question text, or retrieving information related to the question text from multimodal data. The slot extraction result includes words extracted from the question text that are associated with the intent recognition result.

[0138] In some embodiments, the question text is segmented to obtain segmentation information; a natural language understanding model is invoked to process the segmentation information to obtain intent recognition results and slot extraction results; wherein, the segmentation information includes words in the question text, and the natural language understanding model is used to perform intent recognition and slot extraction on the input information.

[0139] Optionally, a preset word segmentation algorithm can be used to segment the problem text to obtain word segmentation information.

[0140] Optionally, if the question text includes punctuation marks, the word segmentation information may or may not include punctuation marks. You can choose whether to retain punctuation marks based on the actual situation.

[0141] In some embodiments, the natural language understanding model includes an encoder, an intent recognition network, and a slot extraction network. Calling the natural language understanding model to process segmented information and obtain intent recognition and slot extraction results includes: calling the encoder to encode the segmented information to obtain encoded features of the segmented information; calling the intent recognition network to perform intent recognition on the encoded features to obtain intent recognition results; and calling the slot extraction network to extract slots from the encoded features to obtain slot extraction results.

[0142] The training process for natural language understanding models is described below. Figure 5 The embodiments shown will not be described in detail here.

[0143] Step S402: Based on the intent recognition results and slot extraction results, the question text is converted into a search statement supported by a preset database.

[0144] Since the target knowledge graph is managed by a pre-defined database, the question text needs to be converted into a search statement supported by the pre-defined database in order to achieve efficient retrieval of the target knowledge graph within the pre-defined database.

[0145] Step S403: Based on the search query, search the target knowledge graph to obtain the search results.

[0146] In this embodiment of the disclosure, a first type of node and a second type of node are involved. Since the contents of the first type of node and the second type of node are different, different methods can be used to search for the first type of node and the second type of node in order to improve the accuracy of the search.

[0147] In some embodiments, the target knowledge graph includes a first type of node, which represents named entities. For the first type of node, based on the search query, a first target node, a second target node, and the connection relationship between the first target node and the second target node are retrieved from the target knowledge graph; the relationship information represented by the first named entity represented by the first target node, the second named entity represented by the second target node, and the connection relationship between the first target node and the second target node is determined as the search result. Wherein, the similarity between the named entity represented by the first target node and the words contained in the search query is greater than a first preset threshold, the first preset threshold being a pre-set value greater than 0 and less than 1, and the second target node is a node connected to the first target node.

[0148] In other words, when searching for named entities, the search is performed directly based on the similarity between the named entity and the words contained in the search query.

[0149] In some embodiments, the target knowledge graph includes a second type of node, which represents a preset image. For the second type of node, multiple candidate nodes are retrieved from the target knowledge graph based on a search query. The similarity between the tag information of the preset image represented by the candidate node and the words contained in the search query is greater than a second preset threshold. Then, based on the search query, a third target node is determined from the multiple candidate nodes. The similarity between the vector representation of the preset image represented by the third target node and the vector representation of the search query is greater than a third preset threshold. Finally, the image information of the preset image represented by the third target node is determined as the search result. The second and third preset thresholds are pre-set values ​​greater than 0 and less than 1.

[0150] In other words, when retrieving a preset image, a hybrid retrieval method is used. First, a coarse search is performed based on the preset image's label information to retrieve potentially relevant preset images (i.e., candidate nodes). Then, a more precise search is performed based on the vector representation of the preset image. This retrieval method ensures accuracy while requiring minimal computation.

[0151] In this embodiment, by performing intent recognition and slot extraction on the question text, the user's intent can be understood more accurately, improving the accuracy of subsequent searches. Furthermore, during the search process, different search methods are adopted based on the characteristics of text and images, ensuring search accuracy while requiring minimal computation, thus simultaneously meeting the requirements of accuracy and performance.

[0152] In some embodiments, the natural language understanding model needs to be trained before it can be used. The following describes how... Figure 5 The illustrated example demonstrates the training process of a natural language understanding model.

[0153] Figure 5 This is a flowchart illustrating a model training process according to an exemplary embodiment, performed by an electronic device, see [link to flowchart]. Figure 5 The method includes the following steps:

[0154] Step S501: Obtain training data, which includes sample information and the sample intent recognition results and sample slot extraction results corresponding to the sample information.

[0155] Step S502: Input the sample information into the natural language understanding model to be trained, and output the predicted intent recognition result and the predicted slot extraction result.

[0156] Step S503: Determine the first loss value based on the first loss function, the sample intent recognition result, and the predicted intent recognition result.

[0157] The first loss function can be the standard cross-entropy loss function, or other loss functions.

[0158] Step S504: Determine the second loss value based on the second loss function, the sample slot extraction result, and the predicted slot extraction result.

[0159] The second loss function can be the cross-entropy loss function, or other loss functions. The first and second loss functions can be the same or different.

[0160] In one example, the second loss function is shown in the following formula:

[0161]

[0162] Among them, L slot The second loss value is represented by T, the length of the question text is represented by i, and the i-th character in the question text is represented by y. i Let C represent the predicted label corresponding to the i-th character, and let C represent the number of sample labels. This represents the probability that the label corresponding to the i-th character is the sample label c.

[0163] It should be noted that the sample slot extraction results and the predicted slot extraction results are characterized by the label of each predicted character to indicate whether the character is the one that needs to be extracted.

[0164] It should be noted that the embodiments disclosed herein are only illustrated by taking the example of executing step S503 first and then step S504. In another embodiment, step S504 may be executed first and then step S503 may be executed, or steps S503 and S504 may be executed simultaneously.

[0165] Step S505: Determine the target loss value based on the first loss value, the second loss value, the first weight corresponding to the first loss value, and the second weight corresponding to the second loss value.

[0166] In this embodiment of the disclosure, the intent recognition network and the slot extraction network are jointly trained. However, the intent recognition network and the slot extraction network share a single encoder. In order to train better and achieve better results, dynamic loss weighting is used to determine the first weight corresponding to the intent recognition network and the second weight corresponding to the slot extraction network. Then, based on the first loss value, the second loss value, the first weight corresponding to the first loss value, and the second weight corresponding to the second loss value, the target loss value is determined.

[0167] In some embodiments, the first weight and the second weight can be preset fixed values, or they can be adjusted in real time according to the training process.

[0168] Optionally, in joint training, the intent recognition network and the slot extraction network are trained with different levels of difficulty. The first and second weights can be dynamically adjusted based on the performance of the intent recognition network and the slot extraction network during training. In one example, the weights are adjusted based on the gradient using the following formula:

[0169]

[0170] Where, α k This represents the weight corresponding to task k, where the task includes the intent recognition network and the slot extraction network. This represents the norm of the gradient of the model parameter gradient θ with respect to task k.

[0171] The weights are adjusted based on the gradient magnitudes of the intent recognition network and the slot extraction network; the larger the gradient, the higher the weight.

[0172] Step S506: Based on the target loss value, adjust the model parameters of the natural language understanding model to be trained to obtain the natural language understanding model.

[0173] It should be noted that the embodiments disclosed herein are illustrated using the example of training a natural language understanding model on an electronic device. In another embodiment, a natural language understanding model can also be trained by other devices using the above-described implementation method, and then the natural language understanding model can be deployed in an electronic device.

[0174] In this embodiment of the disclosure, in order to better train the natural language understanding model and achieve better results, different weights are set for the training of the intent recognition network and the slot extraction network. The training status of the intent recognition network and the slot extraction network are comprehensively considered to adjust the model parameters of the entire natural language understanding model, thereby improving the accuracy of the natural language understanding model.

[0175] In some embodiments, after retrieving the search results, the search results and the question text are concatenated to obtain contextual information; this contextual information is then input into the large language model to output the answer information. In other words, the retrieved search results are used to construct contextual information, providing rich background knowledge for the large language model's processing. This avoids the large language model relying solely on its internal preferred data for answering, allowing it to generate content using an autoregressive generation strategy based on a large amount of contextual information, and combining this with an attention mechanism to optimize the relevance between the generated answer information and the question text.

[0176] Optionally, for the response information output by the large language model, consistency can be ensured by comparing it with information in a preset knowledge base or target knowledge graph. If the response information is inconsistent with the information in the preset knowledge base or target knowledge graph, the response information is corrected based on the information in the preset knowledge base or target knowledge graph, and the corrected response information is taken as the final response information.

[0177] Figure 6 This is a flowchart illustrating an intelligent question-answering method according to an exemplary embodiment, executed by an electronic device. See also... Figure 6 The method includes the following steps:

[0178] Step S601: Construct a target knowledge graph based on multimodal data.

[0179] Step S602: Obtain the input question text.

[0180] Step S603: Perform intent recognition and slot extraction on the question text to obtain intent recognition results and slot extraction results.

[0181] Step S604: Based on the intent recognition results and slot extraction results, the question text is converted into a search statement supported by a preset database.

[0182] Step S605: Based on the search query, perform a search on the target knowledge graph to obtain the search results.

[0183] Step S606: Concatenate the search results and the question text to obtain contextual information.

[0184] Step S607: Input the context information into the large language model and output the answer information.

[0185] The implementation methods for steps S601-S607 are the same as those in the above embodiments, and will not be repeated here.

[0186] In some embodiments, the intelligent question-answering method shown in the above embodiments is implemented based on an intelligent question-answering system. In one example, see [link to example]. Figure 7 The diagram shows the architecture of an intelligent question-answering system. This system comprises three parts: The first part performs vectorized embedding and relation extraction on multimodal data to construct a target knowledge graph; the second part acquires the user-input question text, performs NLU (Natural Language Understanding) on ​​the question text (i.e., intent recognition and slot extraction), and retrieves the target knowledge graph based on the question text, intent recognition results, and slot extraction results to obtain the retrieval results. Then, based on the question text and retrieval results, it generates contextual information; the third part inputs the contextual information into a large language model and outputs the answer information.

[0187] The intelligent question-answering method provided in this disclosure can be applied to various question-answering scenarios, such as scenarios where intelligent question-answering systems can be used, such as scenarios where data is retrieved locally on an electronic device or where data is retrieved through a browser.

[0188] In one example, Figure 8This is a schematic diagram illustrating an application scenario based on an exemplary embodiment. Taking the scenario of retrieving local data on an electronic device as an example, the user inputs the question text "Please help me generate a summary based on the meeting-related documents received yesterday." The NLU model (Natural Language Understanding model) is invoked to extract information such as "Time: Yesterday; Data Domain: Documents; Content: Meeting" from the question text, and the intent of the question text is determined to be to generate a summary. Then, relevant document content is retrieved from the knowledge graph. Finally, the large language model is invoked to generate answer information (i.e., generate a meeting summary) based on the question text and the retrieved relevant document content, and the meeting summary is displayed to the user: "The following is a summary generated for yesterday's meeting-related documents. Meeting content: 1…".

[0189] Of course, it can also be applied to other scenarios, which will not be described in detail here.

[0190] Figure 9 This is a block diagram illustrating an intelligent question-answering device according to an exemplary embodiment, configured in an electronic device. See also... Figure 9 The device includes:

[0191] Problem retrieval module 901 is configured to retrieve the input problem text;

[0192] The retrieval module 902 is configured to perform a retrieval on the target knowledge graph based on the question text and obtain the retrieval results. The target knowledge graph is a knowledge graph constructed based on multimodal data. The retrieval results include multiple retrieval contents related to the question text and the relationship information between different retrieval contents. The retrieval contents include image information or named entities represented by the target nodes in the target knowledge graph.

[0193] The answer generation module 903 is configured to generate answer information based on the question text and the search results.

[0194] In some embodiments, the target knowledge graph is managed by a preset database, and the retrieval module 902 is configured as follows:

[0195] The question text is subjected to intent recognition and slot extraction to obtain intent recognition results and slot extraction results. The intent recognition results are used to characterize the intent of the question text, and the slot extraction results include words extracted from the question text that are associated with the intent recognition results.

[0196] Based on the intent recognition results and slot extraction results, the question text is converted into a search statement supported by the preset database;

[0197] Based on the search query, the target knowledge graph is searched to obtain the search results.

[0198] In some embodiments, the retrieval module 902 is configured to:

[0199] The question text is segmented into words to obtain segmentation information; the segmentation information includes the words in the question text.

[0200] The natural language understanding model is invoked to process the word segmentation information and obtain the intent recognition result and slot extraction result; the natural language understanding model is used to perform intent recognition and slot extraction on the input information.

[0201] In some embodiments, the natural language understanding model includes an encoder, an intent recognition network, and a slot extraction network, and the retrieval module 902 is configured to:

[0202] The encoder is invoked to encode the word segmentation information, thereby obtaining the encoded features of the word segmentation information;

[0203] The intent recognition network is invoked to perform intent recognition on the encoded features, and the intent recognition result is obtained.

[0204] The slot extraction network is invoked to extract slots from the encoded features, and the slot extraction results are obtained.

[0205] In some embodiments, the apparatus further includes a training module, which is configured to:

[0206] Acquire training data, which includes sample information and the corresponding sample intent recognition results and sample slot extraction results.

[0207] The sample information is input into the natural language understanding model to be trained, and the output is the predicted intent recognition result and the predicted slot extraction result.

[0208] The first loss value is determined based on the first loss function, the sample intent recognition result, and the predicted intent recognition result;

[0209] The second loss value is determined based on the second loss function, the sample slot extraction results, and the predicted slot extraction results;

[0210] The target loss value is determined based on the first loss value, the second loss value, the first weight corresponding to the first loss value, and the second weight corresponding to the second loss value.

[0211] Based on the target loss value, the model parameters of the natural language understanding model to be trained are adjusted to obtain the natural language understanding model.

[0212] In some embodiments, the target knowledge graph includes a first type of node, which is used to represent named entities; the retrieval module 902 is configured to:

[0213] For the first type of node, based on the search statement, the first target node, the second target node, and the connection relationship between the first target node and the second target node are retrieved from the target knowledge graph; the similarity between the named entity represented by the first target node and the words contained in the search statement is greater than a first preset threshold, and the second target node is a node connected to the first target node;

[0214] The first named entity represented by the first target node, the second named entity represented by the second target node, and the relationship information represented by the connection relationship between the first target node and the second target node are determined as the retrieval results.

[0215] In some embodiments, the target knowledge graph includes a second type of node, which is used to represent a preset image; the retrieval module 902 is configured to:

[0216] For the second type of node, multiple candidate nodes are retrieved from the target knowledge graph based on the retrieval statement; the similarity between the label information of the preset image represented by the candidate node and the words contained in the retrieval statement is greater than the second preset threshold.

[0217] Based on the search query, a third target node is determined from multiple candidate nodes; the similarity between the vector representation of the preset image represented by the third target node and the vector representation of the search query is greater than a third preset threshold.

[0218] The image information of the preset image represented by the third target node is determined as the retrieval result.

[0219] In some embodiments, the apparatus further includes a knowledge graph construction module, which is configured to:

[0220] When multimodal data includes preset text, obtain information about multiple named entities in the preset text and the relationships between different named entities within those entities;

[0221] When the multimodal data includes a preset image, the image information of the preset image is obtained, and the relationship information between the preset image and the target named entity is determined; the target named entity is the named entity associated with the preset image among multiple named entities;

[0222] A target knowledge graph is constructed based on multiple named entities in the preset text, the relationship information between different named entities, the image information of the preset image, and the relationship information between the preset image and the target named entity.

[0223] In some embodiments, the knowledge graph construction module is configured as follows:

[0224] The preset text is split into multiple sentences to obtain the preset text.

[0225] Named entity recognition is performed on each of the multiple statements to obtain multiple named entities;

[0226] Relationship extraction is performed based on multiple statements and multiple named entities to determine the relationship information between different named entities.

[0227] In some embodiments, the knowledge graph construction module is configured as follows:

[0228] Create a first type of node to represent named entities and a second type of node to represent preset images; the node content of the first type of node is the named entity it represents, and the node content of the second type of node is the image information of the preset image it represents.

[0229] Based on the relationship information between different named entities, and the relationship information between the preset image and the target named entity, a connection relationship is created between different nodes;

[0230] Based on the created first-type nodes, second-type nodes, and the connection relationships between different nodes, a target knowledge graph is constructed.

[0231] In some embodiments, the answer generation module 903 is configured to:

[0232] By concatenating the search results and the question text, contextual information can be obtained.

[0233] The contextual information is input into the large language model, and the response information is output.

[0234] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0235] This disclosure also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the intelligent question-answering method in the above embodiments.

[0236] Figure 10 This is a block diagram of an electronic device 1000 according to an exemplary embodiment.

[0237] Reference Figure 10 The electronic device 1000 may include one or more of the following components: a processing component 1002, a memory 1004, a power supply component 1006, a multimedia component 1008, an audio component 1010, an input / output (I / O) interface 1012, a sensor component 1014, and a communication component 1016.

[0238] Processing component 1002 typically controls the overall operation of electronic device 1000, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1002 may include one or more processors 1020 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1002 may include one or more modules to facilitate interaction between processing component 1002 and other components. For example, processing component 1002 may include a multimedia module to facilitate interaction between multimedia component 1008 and processing component 1002.

[0239] Memory 1004 is configured to store various types of data to support the operation of electronic device 1000. Examples of this data include instructions for any application or method operating on electronic device 1000, contact data, phonebook data, messages, pictures, videos, etc. Memory 1004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0240] Power supply component 1006 provides power to various components of electronic device 1000. Power supply component 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 1000.

[0241] Multimedia component 1008 includes a screen that provides an output interface between the electronic device 1000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1008 includes a front-facing camera and / or a rear-facing camera. When the electronic device 1000 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0242] Audio component 1010 is configured to output and / or input audio signals. For example, audio component 1010 includes a microphone (MIC) configured to receive external audio signals when electronic device 1000 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1004 or transmitted via communication component 1016. In some embodiments, audio component 1010 also includes a speaker for outputting audio signals.

[0243] I / O interface 1012 provides an interface between processing component 1002 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0244] Sensor assembly 1014 includes one or more sensors for providing state assessments of various aspects of electronic device 1000. For example, sensor assembly 1014 may detect the on / off state of electronic device 1000, the relative positioning of components such as the display and keypad of electronic device 1000, changes in position of electronic device 1000 or a component of electronic device 1000, the presence or absence of user contact with electronic device 1000, the orientation or acceleration / deceleration of electronic device 1000, and temperature changes of electronic device 1000. Sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1014 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1014 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0245] Communication component 1016 is configured to facilitate wired or wireless communication between electronic device 1000 and other devices. Electronic device 1000 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 1016 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1016 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0246] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0247] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1004 including instructions, which can be executed by a processor 1020 of an electronic device 1000 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0248] This disclosure also provides a non-transitory computer-readable storage medium, wherein when the instructions in the storage medium are executed by the processor of an electronic device, the electronic device is able to perform the intelligent question-answering method described in the above embodiments.

[0249] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the intelligent question-answering method described in the above embodiments.

[0250] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0251] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. An intelligent question-answering method, characterized in that, The method includes: Get the input question text; Based on the question text, a search is performed on the target knowledge graph to obtain search results; the target knowledge graph is a knowledge graph constructed based on multimodal data, and the search results include multiple search contents related to the question text and information on the relationships between different search contents; the search contents include image information or named entities represented by target nodes in the target knowledge graph. Based on the question text and the search results, answer information for the question text is generated.

2. The intelligent question-answering method according to claim 1, characterized in that, The target knowledge graph is managed by a preset database. The search of the target knowledge graph based on the question text yields search results, including: The question text is subjected to intent recognition and slot extraction to obtain intent recognition results and slot extraction results; the intent recognition results are used to characterize the intent of the question text, and the slot extraction results include words extracted from the question text that are associated with the intent recognition results; Based on the intent recognition result and the slot extraction result, the question text is converted into a search statement supported by the preset database; Based on the search query, the target knowledge graph is searched to obtain the search results.

3. The intelligent question-answering method according to claim 2, characterized in that, The process of performing intent recognition and slot extraction on the question text to obtain intent recognition results and slot extraction results includes: The question text is segmented into words to obtain segmentation information; the segmentation information includes the words in the question text. The natural language understanding model is invoked to process the word segmentation information to obtain the intent recognition result and the slot extraction result; the natural language understanding model is used to perform intent recognition and slot extraction on the input information.

4. The intelligent question-answering method according to claim 3, characterized in that, The natural language understanding model includes an encoder, an intent recognition network, and a slot extraction network. The step of calling the natural language understanding model to process the word segmentation information and obtain the intent recognition result and the slot extraction result includes: The encoder is invoked to encode the word segmentation information, thereby obtaining the encoded features of the word segmentation information; The intent recognition network is invoked to perform intent recognition on the encoded features, and the intent recognition result is obtained. The slot extraction network is invoked to extract slots from the encoded features, thereby obtaining the slot extraction result.

5. The intelligent question-answering method according to claim 3, characterized in that, The training process of the natural language understanding model includes: Acquire training data, which includes sample information and the sample intent recognition results and sample slot extraction results corresponding to the sample information; The sample information is input into the natural language understanding model to be trained, and the predicted intent recognition result and predicted slot extraction result are output. The first loss value is determined based on the first loss function, the sample intent recognition result, and the predicted intent recognition result; The second loss value is determined based on the second loss function, the sample slot extraction result, and the predicted slot extraction result; Based on the first loss value, the second loss value, the first weight corresponding to the first loss value, and the second weight corresponding to the second loss value, the target loss value is determined. Based on the target loss value, the model parameters of the natural language understanding model to be trained are adjusted to obtain the natural language understanding model.

6. The intelligent question-answering method according to claim 2, characterized in that, The target knowledge graph includes a first type of node, which is used to represent named entities; The process of retrieving the target knowledge graph based on the retrieval statement to obtain the retrieval results includes: For the first type of node, based on the search statement, the first target node, the second target node, and the connection relationship between the first target node and the second target node are retrieved from the target knowledge graph; the similarity between the named entity represented by the first target node and the words contained in the search statement is greater than a first preset threshold, and the second target node is a node connected to the first target node; The first named entity represented by the first target node, the second named entity represented by the second target node, and the relationship information represented by the connection relationship between the first target node and the second target node are determined as the retrieval result.

7. The intelligent question-answering method according to claim 2, characterized in that, The target knowledge graph includes a second type of node, which is used to represent a preset image; The process of retrieving the target knowledge graph based on the retrieval statement to obtain the retrieval results includes: For the second type of node, multiple candidate nodes are retrieved from the target knowledge graph based on the retrieval statement; the similarity between the label information of the preset image represented by the candidate node and the words contained in the retrieval statement is greater than a second preset threshold. Based on the search query, a third target node is determined from the plurality of candidate nodes; the similarity between the vector representation of the preset image represented by the third target node and the vector representation of the search query is greater than a third preset threshold. The image information of the preset image represented by the third target node is determined as the retrieval result.

8. The intelligent question-answering method according to claim 1, characterized in that, The method further includes: When the multimodal data includes preset text, obtain information about multiple named entities in the preset text and the relationships between different named entities in the multiple named entities; When the multimodal data includes a preset image, image information of the preset image is acquired, and relationship information between the preset image and a target named entity is determined; the target named entity is the named entity associated with the preset image among the plurality of named entities; The target knowledge graph is constructed based on multiple named entities in the preset text, the relationship information between the different named entities, the image information of the preset image, and the relationship information between the preset image and the target named entity.

9. The intelligent question-answering method according to claim 8, characterized in that, The step of obtaining multiple named entities in the preset text and the relationship information between different named entities includes: The preset text is split into multiple sentences to obtain multiple sentences in the preset text; Named entity recognition is performed on each of the plurality of statements to obtain the plurality of named entities; Based on the multiple statements and the multiple named entities, relationship extraction is performed to determine the relationship information between different named entities among the multiple named entities.

10. The intelligent question-answering method according to claim 8, characterized in that, The construction of the target knowledge graph based on multiple named entities in the preset text, the relationship information between the different named entities, the image information of the preset image, and the relationship information between the preset image and the target named entity includes: Create a first type of node to represent the named entity and a second type of node to represent the preset image; the node content of the first type of node is the named entity it represents, and the node content of the second type of node is the image information of the preset image it represents. Based on the relationship information between the different named entities, and the relationship information between the preset image and the target named entity, a connection relationship between different nodes is created; The target knowledge graph is constructed based on the created first-type nodes, second-type nodes, and the connection relationships between different nodes.

11. The intelligent question-answering method according to any one of claims 1-10, characterized in that, The step of generating answer information for the question text based on the question text and the search results includes: The search results and the question text are concatenated to obtain contextual information; The context information is input into the large language model, and the answer information is output.

12. An intelligent question-and-answer device, characterized in that, The device includes: The question retrieval module is configured to retrieve the input question text; The retrieval module is configured to perform a retrieval on the target knowledge graph based on the question text to obtain retrieval results; the target knowledge graph is a knowledge graph constructed based on multimodal data, and the retrieval results include multiple retrieval contents related to the question text and information on the relationships between different retrieval contents; the retrieval contents include image information or named entities represented by target nodes in the target knowledge graph. The answer generation module is configured to generate answer information for the question text based on the question text and the search results.

13. The intelligent question-answering device according to claim 12, characterized in that, The retrieval module is configured as follows: The question text is subjected to intent recognition and slot extraction to obtain intent recognition results and slot extraction results; the intent recognition results are used to characterize the intent of the question text, and the slot extraction results include words extracted from the question text that are associated with the intent recognition results; Based on the intent recognition result and the slot extraction result, the question text is converted into a search statement supported by the preset database; Based on the search query, the target knowledge graph is searched to obtain the search results.

14. The intelligent question-answering device according to claim 13, characterized in that, The retrieval module is configured as follows: The question text is segmented into words to obtain segmentation information; the segmentation information includes the words in the question text. A natural language understanding model is invoked to process the word segmentation information, thereby obtaining the intent recognition result and the slot extraction result; The natural language understanding model is used to perform intent recognition and slot extraction on the input information.

15. The intelligent question-answering device according to claim 14, characterized in that, The device further includes a training module, which is configured to: Acquire training data, which includes sample information and the sample intent recognition results and sample slot extraction results corresponding to the sample information; The sample information is input into the natural language understanding model to be trained, and the predicted intent recognition result and predicted slot extraction result are output. The first loss value is determined based on the first loss function, the sample intent recognition result, and the predicted intent recognition result; The second loss value is determined based on the second loss function, the sample slot extraction result, and the predicted slot extraction result; Based on the first loss value, the second loss value, the first weight corresponding to the first loss value, and the second weight corresponding to the second loss value, the target loss value is determined. Based on the target loss value, the model parameters of the natural language understanding model to be trained are adjusted to obtain the natural language understanding model.

16. The intelligent question-answering device according to claim 13, characterized in that, The target knowledge graph includes a second type of node, which is used to represent a preset image; the retrieval module is configured as follows: For the second type of node, multiple candidate nodes are retrieved from the target knowledge graph based on the retrieval statement; the similarity between the label information of the preset image represented by the candidate node and the words contained in the retrieval statement is greater than a second preset threshold. Based on the search query, a third target node is determined from the plurality of candidate nodes; the similarity between the vector representation of the preset image represented by the third target node and the vector representation of the search query is greater than a third preset threshold. The image information of the preset image represented by the third target node is determined as the retrieval result.

17. The intelligent question-answering device according to claim 12, characterized in that, The device further includes a knowledge graph construction module, which is configured to: When the multimodal data includes preset text, obtain information about multiple named entities in the preset text and the relationships between different named entities in the multiple named entities; When the multimodal data includes a preset image, the image information of the preset image is acquired, and the relationship information between the preset image and the target named entity is determined; The target named entity is the named entity associated with the preset image among the plurality of named entities; The target knowledge graph is constructed based on multiple named entities in the preset text, the relationship information between the different named entities, the image information of the preset image, and the relationship information between the preset image and the target named entity.

18. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to execute the intelligent question-answering method as described in any one of claims 1-11.

19. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the intelligent question-answering method as described in any one of claims 1-11.

20. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the intelligent question-answering method as described in any one of claims 1-11.

Images