Question and answer method and device and storage medium

Abstract

The invention discloses a question and answer method and device and a storage medium. The question and answer method comprises the steps that a plurality of candidate knowledge matched with a user question is retrieved; extracting target attention information of the plurality of candidate knowledge; and inputting the plurality of candidate knowledge and the target attention information into the question and answer large model to obtain a target answer result about the user question output by the question and answer large model. In this way, the answering accuracy of the question and answer large model can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a question-answering method, device, and storage medium. Background Art

[0002] The big question-answering model is an important application in the field of artificial intelligence, which aims to enable computers to understand and answer questions posed in natural language. At present, the big question-answering model can be applied to various fields such as intelligent customer service, online education, clinical diagnosis, social media, finance, etc., and can assist in decision-making and problem solving, and improve work efficiency and user experience.

[0003] However, due to the wide variety of question types that the big question-answering model needs to handle and the limitations of the model architecture and algorithms, the accuracy of the answers given by the current big question-answering model is not high and still needs further optimization. Summary of the invention

[0004] The main technical problem solved by this application is to provide a question-answering method, device and computer-readable storage medium that can improve the accuracy of answers given by a large question-answering model.

[0005] In order to solve the above technical problems, a technical solution adopted in this application is: to provide a question-answering method, the method comprising: retrieving a number of candidate knowledge matching the user's question; extracting target focus information of the number of candidate knowledge; inputting the number of candidate knowledge and the target focus information into a question-answering big model, and obtaining the target answer result about the user's question output by the question-answering big model.

[0006] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, comprising a memory and a processor coupled to each other, the memory storing program instructions; the processor is used to execute the program instructions stored in the memory to implement the above question-and-answer method.

[0007] In order to solve the above technical problems, another technical solution adopted in the present application is: providing a computer-readable storage medium, which is used to store program instructions, and the program instructions can be executed by a processor to implement the above question-and-answer method.

[0008] In the above scheme, after retrieving several candidate knowledge matching the user's question, the target attention information of several candidate knowledge is further extracted, and the candidate knowledge and the target attention information are input into the question-answering big model together, and the target answer result of the user's question is generated by the question-answering big model. In this way, the question-answering big model can pay attention to the target attention information during the real-time answering process, assist the question-answering big model in understanding and integrating several candidate knowledge, reduce the difficulty of understanding of the question-answering big model, and thus improve the accuracy of the question-answering big model's answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a flowchart of an embodiment of a question-and-answer method provided by the present application;

[0010] Figure 2 It is a flowchart of an embodiment of a method for constructing a target knowledge vector library provided by the present application;

[0011] Figure 3 It is a schematic diagram of the principle of semantic segmentation using a semantic segmentation large model provided by the present application;

[0012] Figure 4 is a schematic diagram of the model structure of the multimodal vector representation model provided by this application;

[0013] Figure 5 is a schematic diagram of the target knowledge vector library provided by this application;

[0014] Figure 6 It is a flowchart of another embodiment of the question-answering method provided by the present application;

[0015] Figure 7 It is a schematic diagram of the integrated text knowledge provided by this application;

[0016] Figure 8 is a schematic diagram of the second candidate knowledge set and the attention information encoding features of the second sub-attention information provided by the present application;

[0017] Fig. 9 is a schematic diagram of the third candidate knowledge set and the attention information encoding features of the third sub-attention information provided by the present application;

[0018] Fig.10 It is a schematic diagram of the model structure of the question-answering model provided by this application;

[0019] Fig.11 It is a schematic diagram of the framework of an embodiment of a question-answering device provided by the present application;

[0020] Fig.12 It is a schematic diagram of a framework of an embodiment of an electronic device provided by the present application;

[0021] Fig.13 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail below with reference to the accompanying drawings and examples.

[0023] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0024] It should be noted that the term "and / or" in this article is only an association information describing associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "multiple" means two or more than two. The term "at least one" means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean any one or more elements selected from the set consisting of A, B and C. The term "several" means at least one. The terms "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0025] See also Figure 1 , Figure 1 is a flowchart of an embodiment of the question-answering method provided by the present application. The method can be executed by a terminal device, which can be a mobile phone, a laptop, a tablet computer, a desktop computer, a vehicle terminal, a smart wearable device, etc., and this embodiment does not specifically limit this. It should be noted that if there are substantially the same results, the method of the present application is not limited to Figure 1 The process sequence shown is limited. Figure 1 As shown, the method comprises the following steps:

[0026] S11: Retrieve several candidate knowledge matching the user question.

[0027] In this embodiment, the user question is converted into at least one retrieval vector, and each retrieval vector is matched with each knowledge vector of the pre-constructed target knowledge vector library to obtain a number of candidate knowledge vectors that match each retrieval vector, and the original knowledge corresponding to each candidate knowledge vector is used as a number of candidate knowledge that matches the user question.

[0028] The target knowledge vector library includes several knowledge vectors generated based on several original knowledge. The several original knowledge used to construct the target knowledge vector library can be knowledge in a certain vertical field, for example, relevant knowledge in the field of education, relevant knowledge in the field of medicine, or relevant knowledge in the field of finance. Alternatively, the several original knowledge used to construct the target knowledge vector library can include relevant knowledge in several vertical fields.

[0029] The target knowledge vector library may include at least one of a private domain knowledge vector library and an Internet domain knowledge vector library. That is, the target knowledge vector library may include only a private domain knowledge vector library, or only an Internet domain knowledge vector library, or both a private domain knowledge vector library and an Internet domain knowledge vector library. The knowledge domain of the several original knowledge used to construct the target knowledge vector library may include at least one of a private domain and an Internet domain, that is, the several original knowledge may further include at least one of several original private domain knowledge and several original Internet domain knowledge, and the private domain knowledge vector library may be constructed using several original private domain knowledge, and the Internet domain knowledge vector library may be constructed using several original Internet domain knowledge. It is understandable that the channels for acquiring private domain knowledge and Internet domain knowledge are different. Private domain knowledge refers to the knowledge accumulated by a specific organization, group or individual, such as internal enterprise data, expert databases, etc. Internet domain knowledge refers to knowledge that is widely disseminated and shared through Internet platforms (such as search engines, online courses, social media, etc.). For the specific construction process of the target knowledge vector library, please refer to the relevant statements in the following text.

[0030] Exemplarily, the target knowledge vector library includes a private domain knowledge vector library and an Internet domain knowledge vector library, and each search vector corresponding to the user question is matched with the private domain knowledge vector library and the Internet domain knowledge vector library respectively to obtain a number of candidate private domain knowledge vectors and a number of candidate Internet domain knowledge vectors, and the original private domain knowledge corresponding to each candidate private domain knowledge vector is used as a number of candidate private domain knowledge matching the user question, and the original Internet domain knowledge corresponding to each candidate Internet domain knowledge vector is used as a number of candidate Internet domain knowledge matching the user question. The number of candidate private domain knowledge and the number of candidate Internet domain knowledge constitute a number of candidate knowledge.

[0031] S12: Extract target attention information of several candidate knowledge.

[0032] The extracted target attention information is used to assist the question-answering large model in understanding and integrating several candidate knowledge, so that the question-answering large model can select one or more appropriate candidate knowledge from several candidate knowledge for knowledge understanding and answering.

[0033] In this embodiment, the target focus information may include at least one of the following information: the degree of matching between each candidate knowledge and the user question (or the confidence of each candidate knowledge), the knowledge domain to which each candidate knowledge belongs, the priority order of each candidate knowledge, and the association information between the candidate private domain knowledge and the candidate Internet domain knowledge.

[0034] The matching degree between candidate knowledge and user questions refers to the similarity between the knowledge vector corresponding to the candidate knowledge and the search vector corresponding to the user question. By obtaining the matching degree between each candidate knowledge and the user question, it helps the question-answering model to select the candidate knowledge most closely related to the user question for knowledge understanding and answering, thereby improving the accuracy of the answer.

[0035] The knowledge domain to which the candidate knowledge belongs is one of the private domain and the Internet domain. When the candidate knowledge is candidate private domain knowledge, the knowledge domain to which the candidate knowledge belongs is the private domain; when the candidate knowledge is candidate Internet domain knowledge, the knowledge domain to which the candidate knowledge belongs is the Internet domain. It is understandable that since knowledge in different knowledge domains has different authority and applicability, such as the authority and accuracy of private domain knowledge is usually higher than that of Internet domain knowledge, by obtaining the knowledge domain to which each candidate knowledge belongs, it helps the question-answering model to identify the knowledge domain to which each candidate knowledge belongs, and give priority to screening candidate knowledge belonging to the private domain for knowledge understanding and answering, thereby improving the accuracy of the answer.

[0036] The association information between the candidate private domain knowledge and the candidate Internet domain knowledge is used to characterize whether the candidate private domain knowledge has associated candidate Internet domain knowledge, and the degree of correlation between each candidate Internet domain knowledge associated with the candidate private domain knowledge and the corresponding candidate private domain knowledge. Since there is knowledge complementarity between the candidate private domain knowledge and the candidate Internet domain knowledge associated with it, by obtaining the association relationship between the candidate private domain knowledge and the candidate Internet domain knowledge, it helps the question-answering model to pay more attention to the candidate Internet domain knowledge with a higher degree of correlation with the candidate private domain knowledge, and to generate a more comprehensive and accurate answer by combining the candidate private domain knowledge and the candidate Internet domain knowledge associated with it.

[0037] The priority order of each candidate knowledge can be obtained by combining the matching degree of each candidate knowledge with the user's question and the preset priority rule. Among them, the preset priority rule is: the priority order of candidate private domain knowledge is higher than the priority order of candidate Internet domain knowledge. Since private domain knowledge is more accurate and reliable than Internet domain knowledge, by obtaining the priority order of each candidate knowledge, it is helpful for the question-answering big model to form an understanding priority, so that the question-answering big model gives priority to understanding the candidate private domain knowledge. When there is a conflict between the candidate private domain knowledge and the candidate Internet domain knowledge, the candidate private domain knowledge is given priority. When the candidate private domain knowledge cannot answer the user's question, the candidate Internet domain knowledge is used to answer, thereby improving the accuracy of the answer.

[0038] S13: Input several candidate knowledge and target attention information into the question-answering big model, and obtain the target answer result about the user's question output by the question-answering big model.

[0039] After obtaining a number of candidate knowledge and target attention information, the candidate knowledge and target attention information are input into the question-answering big model together, and the understanding and reasoning ability of the question-answering big model is used to generate the target answer result corresponding to the user's question.

[0040] In this embodiment, after retrieving several candidate knowledge matching the user's question, target attention information of several candidate knowledge is further extracted, and several candidate knowledge and target attention information are input into the question-answering big model together, and the question-answering big model is used to generate the target answer result of the user's question. In this way, the question-answering big model can pay attention to the target attention information during the real-time answering process, assist the question-answering big model in understanding and integrating several candidate knowledge, reduce the difficulty of understanding of the question-answering big model, and thus improve the accuracy of the question-answering big model's answer.

[0041] See also Figure 2 , Figure 2 1 is a flow chart of an embodiment of a method for constructing a target knowledge vector library provided by the present application. The target knowledge vector library in this embodiment includes a private domain knowledge vector library and an Internet domain knowledge vector library. Figure 2 As shown, the method comprises the following steps:

[0042] S21: performing knowledge segmentation on a number of original private domain knowledge to obtain a number of private domain knowledge units; and performing knowledge segmentation on a number of original Internet domain knowledge to obtain a number of Internet domain knowledge units.

[0043] In one implementation, a corresponding access permission may be applied for to obtain some original private domain knowledge owned by a specific organization, group or individual.

[0044] In one implementation, some original Internet domain knowledge can be directly acquired from major Internet platforms.

[0045] In another embodiment, considering that the Internet domain knowledge is numerous and difficult to obtain, in order to improve the efficiency of constructing the target knowledge vector library, only the original Internet domain knowledge associated with each original private domain knowledge can be obtained. In this embodiment, the step of obtaining a number of original Internet domain knowledge includes: for each private domain knowledge unit, generating a search request corresponding to the private domain knowledge unit, and calling the Internet search interface to search for multiple original Internet domain knowledge associated with the private domain knowledge unit based on the search request. Among them, the Internet search interface can be a search interface of a general search engine. For the relevant content of obtaining the private domain knowledge unit, please refer to the relevant statements in the following text.

[0046] Specifically, a search request formed by a private domain knowledge unit is input into a search interface of a general search engine, and search results of the general search engine regarding the search request are obtained, and multiple original Internet domain knowledge associated with the private domain knowledge unit is obtained based on the search results of the search engine.

[0047] In one example, the multiple original Internet domain knowledge included in the search results may be directly used as the multiple original Internet domain knowledge associated with the private domain knowledge unit.

[0048] In another example, the multiple original Internet domain knowledge included in the search results are input into the large language model, the large language model is used to perform preset processing on the multiple original Internet domain knowledge in the search results, and the multiple original Internet domain knowledge obtained after the preset processing are used as the multiple original Internet domain knowledge associated with the private domain knowledge unit.

[0049] Among them, the preset processing may include at least one of the following processing: deleting repeated original Internet domain knowledge in the search results; deleting erroneous original Internet domain knowledge in the search results; sorting the original Internet domain knowledge in the search results according to the relevance to the private domain knowledge unit, and deleting the original Internet domain knowledge that does not meet the relevance sorting requirements. Exemplarily, the large language model can calculate the relevance score between each original Internet domain knowledge and the corresponding private domain knowledge unit, and sort the original Internet domain knowledge in the search results according to the order of the relevance score of each original Internet domain knowledge, and retain the original Internet domain knowledge ranked before the first preset number of digits. The first preset number of digits can be set according to actual needs. For example, the first preset number of digits is 10, which means that the original Internet domain knowledge ranked in the top 10 is retained.

[0050] Optionally, in this example, the large language model can also generate corresponding relevant representation information for each original Internet domain knowledge associated with the private domain knowledge unit, and the relevant representation information of the original Internet domain knowledge is used to indicate the degree of correlation between the original Internet domain knowledge and the corresponding private domain knowledge unit (or the corresponding original private domain knowledge). Exemplarily, the relevant representation information can be a correlation score, or the relevant representation information can also be a character identifier, such as "top1", "top2", "top3", etc. Among them, the correlation degree of "top1" is higher than the correlation degree of "top2", the correlation degree of "top2" is higher than the correlation degree of "top3", and so on.

[0051] By generating corresponding relevant representation information for each original Internet domain knowledge associated with the private domain knowledge unit, the question-answering model can pay more attention to the original Internet domain knowledge that is more relevant to the original private domain knowledge in the subsequent real-time understanding link.

[0052] It should be noted that, since each original private domain knowledge in this embodiment is divided into one or more private domain knowledge units, if a certain original Internet domain knowledge is associated with a certain private domain knowledge unit, it can be considered that the original Internet domain knowledge is associated with the original private domain knowledge corresponding to the private domain knowledge unit.

[0053] In this embodiment, after obtaining a number of original private domain knowledge and a number of original Internet domain knowledge, the number of original private domain knowledge and the number of original Internet domain knowledge are further segmented to obtain a number of private domain knowledge units corresponding to the number of original private domain knowledge and a number of Internet domain knowledge units corresponding to the number of original Internet domain knowledge. The modalities of the number of original private domain knowledge and the modalities of the number of original Internet domain knowledge both include at least one of text modality, audio modality and video modality. For example, the modalities of the number of original private domain knowledge and the modalities of the number of original Internet domain knowledge both include original documents of several text modalities, original audios of several audio modalities, and original videos of several video modalities. Knowledge of different modalities can be segmented using corresponding segmentation methods.

[0054] In the case where a plurality of original private domain knowledge and a plurality of original Internet domain knowledge both include a plurality of original videos, knowledge segmentation can be performed on each original video by the following steps:

[0055] Step 1: Divide the original video into several video frames.

[0056] Specifically, the original video can be divided into several video frames by using relevant video processing software. For the original video with sound, each video frame includes a corresponding frame of image and voice information, or each video frame only includes a corresponding frame of image. For the original video without sound, each video frame only includes a corresponding frame of image.

[0057] Step 2: divide a number of video frames into a number of video frame clusters.

[0058] A video frame cluster includes multiple similar video frames.

[0059] In one example, each video frame includes a corresponding frame of image and voice information, and dividing a plurality of video frames into a plurality of video frame clusters may include the following sub-steps:

[0060] Sub-step 1: extracting image features from the images of each video frame to obtain image extraction features of each video frame.

[0061] The image feature extraction network is used to extract image features from the images of each video frame to obtain image extraction features of each video frame. Exemplarily, the image feature extraction network can be a network such as CNN.

[0062] Sub-step two: converting the voice information of each video frame into first text information, and performing text feature extraction on the first text information of each video frame to obtain text extraction features of each video frame.

[0063] Speech recognition is performed on the speech information of each video frame to obtain the first text information of each video frame. Then, a text feature extraction network is used to extract text features from the first text information of each video frame to obtain text extraction features of each video frame. Exemplarily, the text feature extraction network can be a language model such as Bert.

[0064] In sub-step three, attention processing is performed on the image extraction features and text extraction features of each video frame to obtain the fusion features corresponding to each video frame.

[0065] In one embodiment, the image extraction features and text extraction features of each video frame may be cross-attention processed to obtain fusion features corresponding to each video frame. For example, for each video frame, the text extraction features of the video frame are used as the query features of the video frame, and the image extraction features of the video frame are used as the key features and value features of the video frame; based on the query features, key features and value features of the video frame, the fusion features of the video frame are determined.

[0066] Specifically, the step of determining the fusion feature of the video frame based on the query feature, key feature and value feature of the video frame includes: multiplying the query feature and the key feature of the video frame, and normalizing (softmax) the result of the multiplication to obtain the attention weight feature of the video frame; and taking the product of the value feature of the video frame and the attention weight feature of the video frame as the fusion feature of the video frame.

[0067] Sub-step four: dividing the video frames into a number of video frame clusters based on the similarity between the fusion features of each video frame.

[0068] In one embodiment, the plurality of video frames are divided into a plurality of video frame clusters by the following sub-steps: the first video frame of the plurality of video frames is used as the first video frame, and the second video frame of the plurality of video frames is used as the second video frame; the cosine similarity between the fusion feature of the first video frame and the fusion feature of the second video frame is calculated; whether the cosine similarity between the first video frame and the second video frame is greater than a preset threshold value; if the cosine similarity between the first video frame and the second video frame is greater than the preset threshold value, the second video frame is divided into the video frame cluster where the first video frame is located, and the current second video frame is used as a new first video frame; and The next video frame of the current second video frame is used as a new second video frame, and then the cosine similarity between the fusion features of the first video frame and the fusion features of the second video frame and the subsequent steps are calculated again; if the cosine similarity between the first video frame and the second video frame is less than or equal to the preset threshold, the second video frame is divided into another video frame cluster, and the current second video frame is used as a new first video frame, and the next video frame of the current second video frame is used as a new second video frame, and then the cosine similarity between the fusion features of the first video frame and the fusion features of the second video frame and the subsequent steps are calculated again. In this way, similar video frames can be divided into the same video frame cluster, and dissimilar video frames can be divided into different video frame clusters, thereby obtaining several video frame clusters. Exemplarily, the preset threshold can be set according to actual needs.

[0069] In this implementation, the image information and voice information of each video frame are fused respectively to obtain the fusion features corresponding to each video frame, and the plurality of video frames are divided into a plurality of video frame clusters based on the fusion features of each video frame. Compared with the method of dividing the video frame clusters based on a single information, the accuracy of the video frame cluster division can be improved.

[0070] Optionally, in other embodiments, each video frame includes only a corresponding frame of image, then dividing a number of video frames into a number of video frame clusters may include the following sub-steps: performing image feature extraction on the image of each video frame to obtain image extraction features of each video frame; based on the similarity between the image extraction features of each video frame, dividing the number of video frames into a number of video frame clusters.

[0071] Step three, for each video frame cluster, obtain text description information of the video frame cluster, multiple frame images corresponding to the video frame cluster, and audio information corresponding to the video frame cluster as the video knowledge unit corresponding to the video frame cluster.

[0072] Specifically, the text description information of a video frame cluster, multiple frames of images and audio information corresponding to the video frame cluster are taken as a video knowledge unit. Alternatively, only the text description information of a video frame cluster and multiple frames of images are taken as a video knowledge unit.

[0073] Specifically, the step of obtaining text description information of a video frame cluster includes: converting the audio information of the video frame cluster into second text information; inputting multiple frame images and the second text information of the video frame cluster into a multimodal large model to obtain text description information of the video frame cluster.

[0074] The audio information of the video frame cluster includes an audio segment corresponding to the video frame cluster. The second text information of the video frame cluster can be obtained by performing speech recognition on the audio information of the video frame cluster.

[0075] The text description information of the video frame cluster includes image description information and subject information of the video frame cluster.

[0076] The multimodal big model refers to an artificial intelligence model that can process data in multiple modalities (such as text modality, image modality, audio modality, video modality, etc.). The multimodal big model provides more comprehensive and accurate understanding and analysis by fusing data in different modalities. The multi-frame images and the second text information of the video frame cluster are input into the multimodal big model, and the natural language description ability of the multimodal big model is used to understand the multi-frame images and the second text information of the video frame cluster, so as to generate image description information of the video frame cluster and extract the theme information of the video frame cluster.

[0077] In the case where a number of original private domain knowledge and a number of original Internet domain knowledge both include a number of original documents, one or more original documents may be regarded as an original text knowledge, and knowledge segmentation is performed on each original text knowledge through the following steps:

[0078] Step 1: Extract text-related information from original text knowledge.

[0079] The original document corresponding to the original text knowledge may be in the format of word, excel, pdf, txt, etc. The document content of the original text knowledge can be parsed through open source document parsing tools to extract text-related information of the original text knowledge.

[0080] Exemplarily, the text-related information may include specific text content, spatial layout information of text / table / picture, etc. (such as which line it is located in, centered, left-aligned, etc.), font size, font type, font color, etc.

[0081] Step 2: Input the text-related information of the original text knowledge into the semantic segmentation model to obtain the semantic segmentation result of the original text knowledge.

[0082] After extracting the text-related information of the original text knowledge, a prompt text corresponding to the text-related information can be generated based on a preset prompt information template, and the prompt text includes role definition information, task requirement information, and the aforementioned extracted text-related information. Exemplarily, the role definition information is "You are a document analysis assistant", and the task requirement information is "Please perform hierarchical title recognition and semantic paragraph recognition on the following text." Further, the generated prompt text is input into the semantic segmentation large model to obtain the semantic segmentation result corresponding to the original text knowledge. Among them, the semantic segmentation result includes hierarchical title recognition results (such as first-level titles, second-level titles, etc.) and semantic paragraph recognition results. Exemplarily, the semantic segmentation large model can be a large language model of the GPT series.

[0083] Figure 3 This is a schematic diagram of the principle of semantic segmentation using the semantic segmentation model provided by this application, which can generate Figure 3 The prompt text input semantic segmentation model shown in the figure is Figure 3 The semantic segmentation results are shown.

[0084] Step three, based on the semantic segmentation results of the original text knowledge, divide the original text knowledge into several text knowledge units.

[0085] Specifically, the original text knowledge can be segmented into several text knowledge units according to a predetermined segmentation rule based on the hierarchical title video results and semantic paragraph recognition results of the original text knowledge. For example, different semantic segmentation paragraphs recognized by the semantic segmentation large model can be respectively regarded as a text knowledge unit.

[0086] In the case where a number of original private domain knowledge and a number of original Internet domain knowledge include a number of original audios, speech recognition is performed on each original audio to obtain a recognition text corresponding to each original audio, and the recognition text of one or more original audios is used as an original text knowledge, and then the original text knowledge segmentation method in the previous implementation is used to perform knowledge segmentation.

[0087] Through step S21, the original private domain knowledge of each modality can be divided into at least one private domain knowledge unit, and the original Internet domain knowledge of each modality can be divided into at least one Internet domain knowledge unit. The private domain knowledge unit can be a video knowledge unit belonging to the private domain or a text knowledge unit belonging to the private domain, and the Internet domain knowledge unit can be a video knowledge unit belonging to the Internet domain or a text knowledge unit belonging to the Internet domain.

[0088] S22: performing vectorization processing on each private domain knowledge unit and each Internet domain knowledge unit respectively, to obtain a private domain knowledge vector corresponding to each private domain knowledge unit and an Internet domain knowledge vector corresponding to each Internet domain knowledge unit respectively.

[0089] In the case where the segmented private domain knowledge units and the Internet domain knowledge units include video knowledge units, the step of vectorizing each video knowledge unit includes: inputting the text description information, multi-frame images and audio information of the video knowledge unit into a multimodal vector representation model to obtain a multimodal feature vector of the video knowledge unit; at least using the multimodal feature vector of the video knowledge unit as the vector after the video knowledge unit is vectorized. The multimodal vector representation model can capture the relationship between the text description information, multi-frame images and audio information of the video knowledge unit, so that the multimodal feature vector obtained by the multimodal vector representation model can better represent the corresponding video knowledge unit.

[0090] Figure 4 is a schematic diagram of the model structure of the multimodal vector representation model provided in this application. Figure 4 As shown, the multimodal vector representation model includes a first text vector representation network, a first image vector representation network, an audio vector representation network, and an attention processing network. The first text vector representation network, the first image vector representation network, and the audio vector representation network can respectively adopt corresponding general networks, and this embodiment will not be listed one by one.

[0091] Specifically, the multimodal feature vector of the video knowledge unit can be obtained by the following steps:

[0092] Step 1: Input the text description information, multi-frame images and audio information of the video knowledge unit into the first text vector representation network, the first image vector representation network and the audio vector representation network of the multimodal vector representation model respectively.

[0093] Step two, use the first text vector representation network to process the text description information of the video knowledge unit to obtain the first text feature vector of the video knowledge unit; use the first image vector representation network to process multiple frames of images of the video knowledge unit to obtain the first image feature vector of the video knowledge unit; and use the audio vector representation network to process the audio information of the video knowledge unit to obtain the audio feature vector of the video knowledge unit.

[0094] Step three, performing modal weighting processing on the first text feature vector, the first image feature vector and the audio feature vector of the video knowledge unit.

[0095] Specifically, the first text feature vector, the first image feature vector and the audio feature vector are concatenated according to feature dimensions; the concatenated feature vector is multiplied by the target weight feature vector to obtain a feature vector after modal weighting processing.

[0096] Among them, the target weight feature vector includes the weighted weights corresponding to the first text feature vector, the first image feature vector and the audio feature vector respectively, and the dimension of the target weight feature vector is 3*hidden_size, and hidden_size is the same as the dimension of the first text feature vector, the first image feature vector and the audio feature vector. The parameters of the target weight feature vector can be randomly initialized first, and then in the process of training the multimodal vector representation model, the parameters of the target weight feature vector are continuously adjusted until the multimodal vector representation model training is completed. Exemplarily, the multimodal vector representation model can be trained by the training method of infonce, and the specific training related content is not described in detail in this article.

[0097] Step 4: Use the attention processing network to perform attention processing on the feature vector after modality weighting processing to obtain the multimodal feature vector of the video knowledge unit.

[0098] Exemplarily, the attention processing network is a cross attention processing network, and the feature vector after modality weighting processing includes weighted feature vectors corresponding to the first text feature vector, the first image feature vector, and the audio feature vector of the video knowledge unit. The weighted feature vector corresponding to the first text feature vector of the video knowledge unit, the weighted feature vector corresponding to the first image feature vector, and the weighted feature vector corresponding to the audio feature vector are used as the query feature vector, key feature vector, and value feature vector of the video knowledge unit, respectively; based on the query feature vector, key feature vector, and value feature vector of the video knowledge unit, the multimodal feature vector of the video knowledge unit is determined.

[0099] Specifically, based on the query feature vector, key feature vector and value feature vector of the video knowledge unit, determining the multimodal feature vector of the video knowledge unit includes: multiplying the query feature vector and the key feature vector, and normalizing (softmax) the result of the multiplication to obtain the attention weight feature vector of the video knowledge unit; and taking the product of the value feature vector and the attention weight feature vector as the multimodal feature vector of the video knowledge unit.

[0100] The modality weighting processing and attention processing in step 3 and step 4 can be expressed by formula (1) and formula (2) respectively:

[0101] Emb~=concat(Emb txt +Emb cv +Emb asr )*Q (1)

[0102] Emb=cross_attention(Emb~) (2)

[0103] Among them, Embtxt 、Emb cv 、Emb asr and Q represent the first text feature vector, the first image feature vector, the audio feature vector and the target weight feature vector of the video knowledge unit respectively.

[0104] Optionally, the multimodal vector representation model may also only include a first text vector representation network, a first image vector representation network and an attention processing network, and only the text description information and multi-frame images of the video knowledge unit may be input into the multimodal vector representation model to obtain the multimodal feature vector of the video unit.

[0105] In step 3 and step 4, since the information richness of different modalities is different and the proportion of information of different modalities is not uniform, by performing modal weighting processing on the first text feature vector, the first image feature vector and the audio feature vector of the video knowledge unit, the attention weight of the multimodal vector representation model on the information of different modalities can be improved, and the information imbalance problem between different modalities can be neutralized, so that the multimodal vector representation model can pay more attention to the core information in the process of extracting multimodal feature vectors. In addition, by performing attention processing on the feature vectors after modal weighting processing, the direct information interaction between different modalities can be strengthened, information sharing between different modalities can be realized, and the representation ability of multimodal feature vectors can be further improved.

[0106] In one implementation, only the multimodal feature vector of the video knowledge unit may be used as the vector after the video knowledge unit is vectorized.

[0107] In another embodiment, the text description information of the video knowledge unit is input into the second text vector representation network to obtain the second text feature vector of the video knowledge unit; multiple frames of images of the video knowledge unit are input into the second image vector representation network to obtain the second image feature vector of the video knowledge unit; at least one of the second text feature vector and the second image feature vector of the video knowledge unit, and the multimodal feature vector of the video knowledge unit are used as the vector of the video knowledge unit after vectorization. The second text vector representation network and the second image vector representation network can respectively adopt corresponding general networks, which will not be listed one by one in this embodiment.

[0108] In one example, the second text feature vector, the second image feature vector, and the multimodal feature vector of the video knowledge unit may be used as the vector after the video knowledge unit is vectorized.

[0109] In another example, the second text feature vector and the multimodal feature vector of the video knowledge unit may be used as the vectorized vector of the video knowledge unit.

[0110] In yet another example, the second image feature vector and the multimodal feature vector of the video knowledge unit are used as vectorized vectors of the video knowledge unit.

[0111] In this implementation, one video knowledge unit is processed to obtain two or three knowledge vectors, that is, the video knowledge unit is multi-way vectorized. In the subsequent retrieval and matching process, the relevant information of the video knowledge unit can be used as much as possible, thereby improving the recall rate of the original knowledge (original private domain knowledge and / or original Internet domain knowledge).

[0112] In the case where the plurality of private domain knowledge units and the plurality of Internet domain knowledge units obtained by segmentation include text knowledge units, the step of vectorizing each text knowledge unit includes: inputting the text knowledge unit into a third text vector representation network to obtain a third text feature vector corresponding to the text knowledge unit. The third text vector representation network and the aforementioned second text vector representation network may be the same network, or the third text vector representation network may be a separately set network.

[0113] S23: adding each private domain knowledge vector to the private domain knowledge vector library, and adding each Internet domain knowledge vector to the Internet domain knowledge vector library.

[0114] After the above steps S22 and S23, a target knowledge vector library including a private domain knowledge vector library and an Internet domain knowledge vector library can be obtained. The private domain knowledge vector library and the Internet domain knowledge vector library both include at least one of a plurality of text feature vectors, a plurality of image feature vectors, and a plurality of multimodal feature vectors.

[0115] Figure 5 is a schematic diagram of the target knowledge vector library provided by this application. Figure 5As shown, the target knowledge vector library includes a private domain knowledge vector library and an Internet domain knowledge vector library. Among them, the private domain knowledge vector library includes several text feature vectors belonging to the private domain, several image feature vectors belonging to the private domain, and several multimodal feature vectors belonging to the private domain (i.e., several private domain knowledge vectors in the private domain knowledge vector library), and the text feature vectors belonging to the private domain can be derived from the original document belonging to the private domain, the original audio belonging to the private domain, or the original video belonging to the private domain, and the image feature vector and multimodal feature vector belonging to the private domain are both derived from the original video belonging to the private domain. The Internet domain knowledge vector library includes several text feature vectors belonging to the Internet domain, several image feature vectors belonging to the Internet domain, and several multimodal feature vectors belonging to the Internet domain (i.e., several Internet domain knowledge vectors in the Internet domain knowledge vector library), and the text feature vector belonging to the Internet domain can be derived from the original document belonging to the Internet domain, the original audio belonging to the Internet domain, or the original video belonging to the Internet domain, and the image feature vector and multimodal feature vector belonging to the Internet domain are both derived from the original video belonging to the Internet domain.

[0116] In one embodiment, the private domain knowledge vector library is constructed based on a number of original private domain knowledge, and the Internet domain knowledge vector library is constructed based on the original Internet domain knowledge associated with each original private domain knowledge. In this embodiment, association mapping information can be further generated. The association mapping information includes a first association mapping relationship, a second association mapping relationship, and a third association mapping relationship.

[0117] The first association mapping relationship is the association mapping relationship between each private domain knowledge vector and each original private domain knowledge. Through the first mapping relationship, it can be known on which original private domain knowledge each private domain knowledge vector is based, which is convenient for subsequent reverse tracing of the original private domain knowledge corresponding to each private domain knowledge vector.

[0118] The second association mapping relationship is the association mapping relationship between each Internet domain knowledge vector and each original Internet domain knowledge. Through the second association mapping relationship, it can be known which original Internet domain knowledge each Internet domain knowledge vector is based on, which is convenient for subsequent reverse tracing of the original Internet domain knowledge corresponding to each Internet domain knowledge vector.

[0119] The third association mapping relationship is the association mapping relationship between each private domain knowledge vector and each Internet domain knowledge vector. Through the third association mapping relationship, it can be known which Internet domain knowledge vectors each private domain knowledge vector can be associated with. It can be understood that if a certain original private domain knowledge is associated with at least one original Internet domain knowledge, then each private domain knowledge vector of the original private domain knowledge is associated with each Internet domain knowledge vector of the at least one original Internet domain knowledge.

[0120] Optionally, in this embodiment, target knowledge vector libraries belonging to different vertical fields may also be constructed, and the target knowledge vector libraries of different vertical fields include a private domain knowledge vector library and an Internet domain knowledge vector library of the corresponding vertical field.

[0121] In this embodiment, knowledge segmentation is performed on a number of original private domain knowledge and a number of original Internet domain knowledge to obtain a number of private domain knowledge units and a number of Internet domain knowledge units; and each private domain knowledge unit and each Internet domain knowledge unit are respectively vectorized to obtain a private domain knowledge vector corresponding to each private domain knowledge unit and an Internet domain knowledge vector corresponding to each Internet domain knowledge unit. In this way, knowledge from different knowledge domains (private domain or Internet domain) and different modalities can be converted into a unified knowledge vector for representation in an offline situation, reducing the representation complexity of knowledge in different knowledge domains and different modalities, and facilitating the retrieval of a number of candidate knowledge matching the user's question in the subsequent real-time question-answering process.

[0122] See also Figure 6 , Figure 6 FIG. 1 is a flow chart of another embodiment of the question-answering method provided by the present application, which can be executed by the aforementioned terminal device. Figure 6 As shown, the method comprises the following steps:

[0123] S61: Obtain user questions.

[0124] In this embodiment, the terminal device can display a search interface, and the user can enter content related to the question in the search interface. The content of the user's question can include at least one of user-input text, user-input picture, user-input audio, and user-input video.

[0125] In one implementation, the content input by the user may be directly obtained as the content of the user question.

[0126] In another embodiment, the user input content may be rewritten in combination with the context information input by the user in the search interface, and the rewritten content may be used as the content of the user question to improve the accuracy and comprehensiveness of the user input content.

[0127] S62: Retrieve a number of candidate knowledge matching the user question, and divide the number of candidate knowledge into at least one candidate knowledge set.

[0128] Retrieving several candidate knowledge matching the user question may include the following steps:

[0129] Step 1: Convert the user question into at least one retrieval vector.

[0130] In one embodiment, the content of the user question only includes the user input video, and the user input video can be processed to organize the user input video into a text retrieval request, an image retrieval request, and an audio retrieval request. Then, the user's text retrieval request, image retrieval request, and audio retrieval request are input into the aforementioned multimodal vector representation model to obtain a multimodal retrieval vector for the user input video. And, the user's text retrieval request and image retrieval request are respectively input into the aforementioned second text vector representation network and second image vector representation network to obtain a text retrieval vector and an image retrieval vector for the user input video. In this embodiment, the user input video can be converted into three retrieval vectors: a text retrieval vector, an image retrieval vector, and a multimodal retrieval vector.

[0131] In another embodiment, the content of the user question only includes the user input text, and the user input text can be processed to regularize the user input text into a text search request. Then, the user's text search request is input into the aforementioned second text vector representation network or the third text vector representation network to obtain a text search vector for the user input text.

[0132] In another embodiment, the content of the user question only includes the user input audio, and the user input audio can be processed to regularize the user input audio into a text search request. Then, the user's text search request is input into the aforementioned second text vector representation network or the third text vector representation network to obtain a text search vector for the user input audio.

[0133] In another embodiment, the content of the user question includes both the user input video and the user input text, then a text retrieval vector, an image retrieval vector and a multimodal retrieval vector about the user input video can be obtained, plus a text retrieval vector about the user input text.

[0134] In another embodiment, the content of the user question includes user input video, user input text and user input audio at the same time, then a text retrieval vector, an image retrieval vector and a multimodal retrieval vector about the user input video can be obtained, plus a text retrieval vector about the user input text and a text retrieval vector about the user input audio.

[0135] Step 2: For each search vector, the search vector is matched with the private domain knowledge vector library and the Internet domain knowledge vector library respectively to obtain at least one candidate private domain knowledge vector in the private domain knowledge vector library that matches the search vector, and at least one first candidate Internet domain knowledge vector in the Internet domain knowledge vector library that matches the search vector.

[0136] Among them, the private domain knowledge vector library includes several private domain knowledge vectors, and the Internet domain knowledge vector library includes several Internet domain knowledge vectors. Exemplarily, the private domain knowledge vector library includes several text feature vectors belonging to the private domain, several image feature vectors belonging to the private domain, and several multimodal feature vectors belonging to the private domain, and the Internet domain knowledge vector library includes several text feature vectors belonging to the Internet domain, several image feature vectors belonging to the Internet domain, and several multimodal feature vectors belonging to the Internet domain. The construction process of the private domain knowledge vector library and the Internet domain knowledge vector library can refer to Figure 2 The embodiments shown will not be described in detail here.

[0137] It should be noted that, considering that the request content corresponding to the user's question may be cross-modal, such as searching for videos with text, searching for audio with text, etc., each search vector in step 2 is matched with each private domain knowledge vector in the private domain knowledge vector library, and is matched with each Internet domain knowledge vector in the Internet domain knowledge vector library.

[0138] Matching the search vector with each private domain knowledge vector refers to respectively calculating the similarity between the search vector and each private domain knowledge vector, and matching the search vector with each Internet domain knowledge vector refers to respectively calculating the similarity between the search vector and each Internet domain knowledge vector.

[0139] Exemplarily, the private domain knowledge vectors may be further sorted based on the similarity between the search vector and each private domain knowledge vector, and at least one private domain knowledge vector located before the second preset number of bits may be selected as at least one candidate private domain knowledge vector matching the search vector. And, further based on the similarity between the search vector and each Internet domain knowledge vector, the Internet domain knowledge vectors may be sorted, and at least one Internet domain knowledge vector located before the third preset number of bits may be selected as at least one first candidate Internet domain knowledge vector matching the search vector. The second preset number of bits and the third preset number of bits may be set according to actual needs, and the second preset number of bits and the third preset number of bits may be the same or different.

[0140] Step three: at least the original private domain knowledge corresponding to each candidate private domain knowledge vector and the original Internet domain knowledge corresponding to each first candidate Internet domain knowledge vector are taken as a plurality of candidate knowledge.

[0141] In step three, based on the aforementioned first association mapping relationship, the original Internet domain knowledge corresponding to each candidate private domain knowledge vector can be traced back, and based on the aforementioned second association mapping relationship, the original Internet domain knowledge corresponding to each first candidate Internet domain knowledge vector can be traced back.

[0142] In one implementation, the original private domain knowledge corresponding to each candidate private domain knowledge vector and the original Internet domain knowledge corresponding to each first candidate Internet domain knowledge vector may be directly used as a plurality of candidate knowledge.

[0143] In another embodiment, for each candidate private domain knowledge vector, at least one second candidate Internet domain knowledge vector associated with the candidate private domain knowledge vector in the Internet domain knowledge vector library is further determined. Specifically, the second candidate Internet domain knowledge vectors associated with each candidate private domain knowledge vector can be determined based on the aforementioned third association mapping relationship. Then, the original private domain knowledge corresponding to each candidate private domain knowledge vector, the original Internet domain knowledge corresponding to each first candidate Internet domain knowledge vector, and the original Internet domain knowledge corresponding to each second candidate Internet domain knowledge vector are used as a plurality of candidate knowledge.

[0144] Furthermore, considering that the same candidate knowledge may exist in the plurality of candidate knowledge obtained in step three, the plurality of candidate knowledge obtained in step three may be further deduplicated to remove the duplicate candidate knowledge.

[0145] Optionally, in step S62, the pre-trained classification model can be used to perform binary classification or multi-domain classification on the user question to obtain a classification result. The classification results of the binary classification include the user question that cannot be answered by knowledge and the user question that can be answered by knowledge. The classification results of the multi-domain classification include the target vertical field to which the user question belongs. Further, after determining the target vertical field to which the user question belongs, the private domain knowledge vector library and the Internet domain knowledge vector library belonging to the target vertical field can be obtained for retrieval matching to improve the retrieval matching efficiency.

[0146] In step S62, after retrieving several candidate knowledge matching the user question, the several candidate knowledge are further divided into at least one candidate knowledge set, each candidate knowledge set includes at least one candidate private domain knowledge and at least one candidate Internet domain knowledge belonging to a modality, and different candidate knowledge sets correspond to different modalities.

[0147] The at least one candidate knowledge set divided includes at least one of a first candidate knowledge set in text mode, a second candidate knowledge set in audio mode, and a third candidate knowledge set in video mode. Each candidate private domain knowledge and each candidate Internet domain knowledge in the first candidate knowledge set are in text mode, each candidate private domain knowledge and each candidate Internet domain knowledge in the second candidate knowledge set are in audio mode, and each candidate private domain knowledge and each candidate Internet domain knowledge in the third candidate knowledge set are in video mode.

[0148] S63: Extract target attention information of several candidate knowledge sets, where the target attention information includes sub-attention information corresponding to each candidate knowledge set.

[0149] In one embodiment, at least one divided candidate knowledge set includes three candidate knowledge sets, namely a first candidate knowledge set, a second candidate knowledge set and a third candidate knowledge set, and the target focus information includes first sub-focus information corresponding to the first candidate knowledge set, second sub-focus information corresponding to the second candidate knowledge set, and third sub-focus information corresponding to the third candidate knowledge set.

[0150] Among them, the first sub-focus information includes at least one of the following information: the matching degree between each candidate knowledge in the first candidate knowledge set and the user's question (or the confidence of each candidate knowledge in the first candidate knowledge set), the knowledge domain to which each candidate knowledge in the first candidate knowledge set belongs, the priority order of each candidate knowledge in the first candidate knowledge set, and the association information between the candidate private domain knowledge in the first candidate knowledge set and the candidate Internet domain knowledge in the first candidate knowledge set.

[0151] The second sub-focus information includes at least one of the following information: the degree of matching between each candidate knowledge in the second candidate knowledge set and the user's question (or the confidence of each candidate knowledge in the second candidate knowledge set), the knowledge domain to which each candidate knowledge in the second candidate knowledge set belongs, the priority order of each candidate knowledge in the second candidate knowledge set, and the association information between the candidate private domain knowledge in the second candidate knowledge set and the candidate Internet domain knowledge in the second candidate knowledge set.

[0152] The third sub-attention information includes at least one of the following information: the matching degree between each candidate knowledge in the third candidate knowledge set and the user question (or the confidence of each candidate knowledge in the third candidate knowledge set), the knowledge domain to which each candidate knowledge in the third candidate knowledge set belongs, the priority order of each candidate knowledge in the third candidate knowledge set, and the association information between the candidate private domain knowledge in the third candidate knowledge set and the candidate Internet domain knowledge in the third candidate knowledge set. The relevant explanation content of the attention information can be referred to the relevant content in the aforementioned step S12, which will not be repeated here.

[0153] S64: For each candidate knowledge set, use the encoding network of the corresponding modality in the question-answering large model to encode the candidate knowledge set and the sub-attention information of the candidate knowledge set to obtain the encoding features of the corresponding modality.

[0154] In this embodiment, the large question-answering model includes a large language model and an encoding network of at least one modality. The encoding network of each modality is used to encode the candidate knowledge set of the corresponding modality and the sub-attention information corresponding to the candidate knowledge set of the corresponding modality into a feature representation that the model can understand.

[0155] The encoding network of the question-answering large model includes at least one of a text encoding network, an audio encoding network, and a video encoding network. When the encoding network of the question-answering large model includes only one of the text encoding network, the audio encoding network, and the video encoding network, the question-answering large model is a large model of single-modal input. When the encoding network of the question-answering large model includes two or three of the text encoding network, the audio encoding network, and the video encoding network, the question-answering large model is a large model of multi-modal input.

[0156] In one embodiment, the encoding network of the question-answering large model only includes a text encoding network. In this embodiment, step S64 includes: fusing the first sub-focus information with each candidate knowledge in the first candidate knowledge set to obtain fused text knowledge; and inputting the fused text knowledge into the text encoding network to obtain text encoding features.

[0157] The first sub-attention information and the first candidate knowledge set are both text modalities, and the first sub-attention information can be integrated into each candidate knowledge of the first candidate knowledge set to obtain integrated text knowledge. For example, based on the knowledge domain to which each candidate knowledge in the first sub-attention information belongs, the knowledge domain to which each candidate knowledge belongs is explicitly added to the corresponding candidate knowledge; based on the association information between the candidate private domain knowledge and the candidate Internet domain knowledge in the first sub-attention information, the candidate private domain knowledge corresponding to the candidate Internet domain knowledge in the first sub-attention information is explicitly added to the corresponding candidate Internet domain knowledge; based on the matching degree between each candidate knowledge in the first sub-attention information and the user's question and the priority order of each candidate knowledge, the candidate knowledge in the first candidate knowledge set is sorted, for example, each candidate private domain knowledge is placed before the candidate Internet domain knowledge, and each candidate private domain knowledge is sorted in order according to the matching degree with the user's question, and each candidate Internet domain knowledge is sorted in order according to the matching degree with the domain user's question. Exemplarily, the generated integrated text knowledge can be as follows Figure 7 shown.

[0158] In another embodiment, the coding network of the question-answering large model includes only an audio coding network or only a video coding network. The audio coding network further includes a first audio coding subnetwork and a second audio coding subnetwork, and the video coding network further includes a first video coding subnetwork and a second video coding subnetwork. In this embodiment, step S64 may include the following steps:

[0159] Step 1: Encode the target sub-attention information to obtain the attention information encoding features.

[0160] The target sub-interest information is the second sub-interest information or the third sub-interest information.

[0161] In a specific embodiment, the encoding feature of the attention information is a two-dimensional matrix. Taking the two-dimensional matrix obtained by encoding the second sub-attention information as an example, the vertical dimension and horizontal dimension of the two-dimensional matrix represent the private domain and the Internet domain respectively (i.e., implicitly representing the knowledge domain of each candidate knowledge in the second candidate knowledge set), the priority of the vertical dimension is higher than the priority of the horizontal dimension, and the sorting of the vertical dimension represents the degree of relevance between each candidate private domain knowledge in the second candidate knowledge set and the user's question, and the sorting of the horizontal dimension represents the degree of matching between each candidate Internet domain knowledge in the second candidate knowledge set and the user's question, and the data in the i-th row and j-th column of the two-dimensional matrix represents the association information between the i-th candidate Internet domain knowledge and the j-th candidate private domain knowledge. Exemplarily, the data in the i-th row and j-th column of the two-dimensional matrix is ​​0 or 1, 0 indicates that there is no association, and 1 indicates that there is an association. Exemplarily, the encoding features of the attention information of the second candidate knowledge set and the second sub-attention information can be as follows Figure 8 As shown, the third candidate knowledge set and the attention information encoding features of the third sub-attention information can be as follows Fig. 9 shown.

[0162] Step 2: Use the first encoding subnetwork to encode the target candidate knowledge set to obtain the first encoding feature.

[0163] When the target sub-attention information is the second sub-attention information, the first encoding sub-network is the first audio encoding sub-network, the target candidate knowledge set is the second candidate knowledge set, and the first encoding feature is the first audio encoding feature. When the target sub-attention information is the third sub-attention information, the first encoding sub-network is the first video encoding sub-network, the target candidate knowledge set is the third candidate knowledge set, and the first encoding feature is the first video encoding feature.

[0164] Step three, multiply the attention information coding feature and the first coding feature.

[0165] When the target sub-attention information is the second sub-attention information, the attention information encoding feature corresponding to the second sub-attention information is multiplied by the first audio encoding feature to establish a priority ranking of each candidate knowledge in the second candidate knowledge set, and to establish an association mapping relationship between the candidate private domain knowledge and the candidate Internet domain knowledge in the second candidate knowledge set.

[0166] When the target sub-attention information is the third sub-attention information, the attention information encoding feature corresponding to the third sub-attention information is multiplied by the first audio encoding feature to establish a priority ranking of each candidate knowledge in the third candidate knowledge set, and to establish an association mapping relationship between the candidate private domain knowledge and the candidate Internet domain knowledge in the third candidate knowledge set.

[0167] Step 4: Use the second encoding sub-network to encode the result of the multiplication operation to obtain a second encoding feature.

[0168] By using the second coding subnetwork to perform cross-domain knowledge association mapping encoding on the result of the multiplication operation corresponding to the attention information coding feature and the first coding feature, a second coding feature that integrates the target sub-attention information and the target candidate knowledge set and carries deeper context information can be obtained. Among them, when the target sub-attention information is the second sub-attention information, the second coding subnetwork is the second audio coding subnetwork, and the second coding feature is the second audio coding feature. When the target sub-attention information is the third sub-attention information, the second coding subnetwork is the second video coding subnetwork, and the second coding feature is the second video coding feature.

[0169] In another embodiment, Fig.10 As shown, the encoding network of the question-answering large model includes a text encoding network, an audio encoding network, and a video encoding network. The text encoding network can be used to encode the first candidate knowledge set and the first sub-focus information to obtain text encoding features, the audio encoding network can be used to encode the second candidate knowledge set and the second sub-focus information to obtain the second audio encoding features, and the video encoding network can be used to encode the third candidate knowledge set and the third sub-focus information to obtain the second video encoding features. The specific steps for obtaining the text encoding features, the second audio encoding features, and the second video encoding features can refer to the first two implementations of this implementation, and will not be repeated here.

[0170] S65: Generate a target answer result about the user's question by using the large language model of the question-answering large model based on the encoding features of each modality.

[0171] Exemplarily, the encoding features of each modality are aligned, fused, and other processes, and then input into the large language model of the question-answering model to obtain the target answer result.

[0172] In this embodiment, the sub-attention information corresponding to the candidate knowledge sets of different modalities is obtained, and the encoding network of each modality in the question-answering large model is used to encode the candidate knowledge set of the corresponding modality and the sub-attention information of the candidate knowledge set to obtain the encoding features of each modality, and the large language model of the question-answering large model is used to generate the target answer result based on the encoding features of each modality. In this way, the question-answering large model can pay attention to the sub-attention information of each modality during the real-time answering process, assist the question-answering large model in understanding and integrating the candidate knowledge of each modality, reduce the difficulty of understanding of the question-answering large model, and thus improve the accuracy of the question-answering large model's answers.

[0173] In this embodiment, by extracting sub-attention information of the candidate knowledge sets of each modality, it helps the question-answering big model to establish the association relationship between the candidate private domain knowledge and the candidate Internet domain knowledge, and form an understanding priority, so that the candidate private domain knowledge of each modality can be understood first. When there is a conflict between the candidate private domain knowledge of each modality and the candidate Internet domain knowledge of each modality, the candidate private domain knowledge of each modality is trusted first. When the candidate private domain knowledge of each modality cannot answer the user's question, the candidate Internet domain knowledge of the corresponding modality is used to answer.

[0174] Optionally, the following training steps can be performed on the question-answering big model to enable the question-answering big model to learn the understanding priority of different domain knowledge: obtain sample data; input the sample data into the question-answering big model to obtain sample answer results about the sample questions generated by the question-answering big model; based on the difference between the sample answer results and the expected answer results of the sample questions, adjust the model parameters of the question-answering big model.

[0175] The sample data includes a sample candidate data set of at least one modality corresponding to the sample question and sample sub-attention information corresponding to each sample candidate data set. Each sample candidate data set includes at least one sample candidate private domain knowledge and at least one sample candidate Internet domain knowledge belonging to the same modality.

[0176] In one example, in order to enable the question-answering model to prioritize understanding of private domain knowledge, a sample question can be constructed as "The price of Anhui Huibao", one of the sample candidate private domain knowledge is "The price of Anhui Huibao is 100 yuan", one of the sample candidate Internet domain knowledge is "The price of Anhui Huibao is 200 yuan", and the expected answer result corresponding to the sample question is "The price of Anhui Huibao is 100 yuan".

[0177] In one example, in order to enable the question-answering model to use Internet domain knowledge to answer when private domain knowledge cannot answer, a sample question can be constructed as "What is the price of Anhui Huibao?". None of the sample candidate private domain knowledge can answer the sample question. One of the sample candidate Internet domain knowledge is "The price of Anhui Huibao is 200 yuan," and the expected answer result corresponding to the sample question is "The price of Anhui Huibao is 200 yuan."

[0178] It should be noted that the above two examples are only illustrative. In the actual training process, more and more complex sample data can be constructed to train the large question-answering model.

[0179] See also Fig.11 , Fig.11It is a schematic diagram of the framework of an embodiment of the question-answering device provided by the present application. In this embodiment, the question-answering device 110 includes a retrieval module 111, an information extraction module 112 and an answer module 113. Among them, the retrieval module 111 is used to retrieve a number of candidate knowledge matching the user's question; the information extraction module 112 is used to extract the target attention information of a number of candidate knowledge; the answer module 113 is used to input the number of candidate knowledge and the target attention information into the question-answering big model, and obtain the target answer result of the user's question output by the question-answering big model.

[0180] Optionally, the plurality of candidate knowledge includes a plurality of candidate private domain knowledge and a plurality of candidate Internet domain knowledge, and the target focus information includes at least one of the following information: the degree of matching between each candidate knowledge and the user's question, the knowledge domain to which each candidate knowledge belongs, the priority order of each candidate knowledge, and the association information between the candidate private domain knowledge and the candidate Internet domain knowledge; wherein, the priority order of the candidate private domain knowledge is higher than the priority order of the candidate Internet domain knowledge.

[0181] Optionally, several candidate knowledge are divided into at least one candidate knowledge set, each candidate knowledge set includes at least one candidate private domain knowledge and at least one candidate Internet domain knowledge belonging to a modality, different candidate knowledge sets correspond to different modalities, the target attention information includes sub-attention information corresponding to each candidate knowledge set, and the question-answering large model includes a large language model and an encoding network of at least one modality. The answer module 113 is used to encode the candidate knowledge set and the sub-attention information of the candidate knowledge set using the encoding network of the corresponding modality for each candidate knowledge set, and obtain the encoding features of the corresponding modality; and generate the target answer result based on the encoding features of each modality using the large language model.

[0182] Optionally, at least one candidate knowledge set includes a first candidate knowledge set of text modality, a second candidate knowledge set of audio modality, and a third candidate knowledge set of video modality, the target attention information includes first sub-attention information of the first candidate knowledge set, second sub-attention information of the second candidate knowledge set, and third sub-attention information of the third candidate knowledge set, and the encoding network of at least one modality includes a text encoding network, an audio encoding network, and a video encoding network, the audio encoding network includes a first audio encoding subnetwork and a second audio encoding subnetwork, and the video encoding network includes a first video encoding subnetwork and a second video encoding subnetwork.

[0183] Optionally, the answer module 113 is used to fuse the first sub-attention information with each candidate knowledge in the first candidate knowledge set to obtain fused text knowledge; and, input the fused text knowledge into the text encoding network to obtain text encoding features; encode the target sub-attention information to obtain the attention information encoding features; use the first encoding subnetwork to encode the target candidate knowledge set to obtain the first encoding features; multiply the attention information encoding features and the first encoding features; and, use the second encoding subnetwork to encode the result of the multiplication to obtain the second encoding features; wherein, when the target sub-attention information is the second sub-attention information, the first encoding subnetwork, the second encoding subnetwork, the target candidate knowledge set, the first encoding features and the second encoding features are respectively the first audio encoding subnetwork, the second audio encoding subnetwork, the second candidate knowledge set, the first audio encoding features and the second audio encoding features; when the target sub-attention information is the third sub-attention information, the first encoding subnetwork, the second encoding subnetwork, the target candidate knowledge set, the first encoding features and the second encoding features are respectively the first video encoding subnetwork, the second video encoding subnetwork, the third candidate knowledge set, the first video encoding features and the second video encoding features.

[0184] Optionally, the retrieval module 111 is used to obtain a private domain knowledge vector library and an Internet domain knowledge vector library; wherein the private domain knowledge vector library includes a number of private domain knowledge vectors, and the Internet domain knowledge vector library includes a number of Internet domain knowledge vectors; convert the user question into at least one retrieval vector; for each retrieval vector, match the retrieval vector with the private domain knowledge vector library and the Internet domain knowledge vector library respectively, and obtain at least one candidate private domain knowledge vector in the private domain knowledge vector library that matches the retrieval vector, and at least one first candidate Internet domain knowledge vector in the Internet domain knowledge vector library that matches the retrieval vector; at least the original private domain knowledge corresponding to each candidate private domain knowledge vector and the original Internet domain knowledge corresponding to each first candidate Internet domain knowledge vector are used as several candidate knowledge.

[0185] Optionally, the retrieval module 111 is used to determine, for each candidate private domain knowledge vector, at least one second candidate Internet domain knowledge vector associated with the candidate private domain knowledge vector in the Internet domain knowledge vector library; and take the original private domain knowledge corresponding to each candidate private domain knowledge vector, the original Internet domain knowledge corresponding to each first candidate Internet domain knowledge vector, and the original Internet domain knowledge corresponding to each second candidate Internet domain knowledge vector as a number of candidate knowledge.

[0186] Optionally, the question-answering device 110 also includes a knowledge vector library construction module 114. The knowledge vector library construction module 114 is used to construct a target knowledge vector library, which includes a private domain knowledge vector library and an Internet domain knowledge vector library. Specifically, the knowledge vector library construction module 114 is used to perform knowledge segmentation on a number of original private domain knowledge to obtain a number of private domain knowledge units; perform knowledge segmentation on a number of original Internet domain knowledge to obtain a number of Internet domain knowledge units; perform vectorization processing on each private domain knowledge unit and each Internet domain knowledge unit respectively to obtain a private domain knowledge vector corresponding to each private domain knowledge unit and an Internet domain knowledge vector corresponding to each Internet domain knowledge unit; add each private domain knowledge vector to the private domain knowledge vector library, and add each Internet domain knowledge vector to the Internet domain knowledge vector library.

[0187] Optionally, the knowledge vector library construction module 114 is used to generate a search request corresponding to each private domain knowledge unit, and call the Internet search interface to search for multiple original Internet domain knowledge associated with the private domain knowledge unit based on the search request. And / or, both the private domain knowledge vector library and the Internet domain knowledge vector library include at least one of a plurality of text feature vectors, a plurality of image feature vectors, and a plurality of multimodal feature vectors.

[0188] Optionally, the plurality of original private domain knowledge and the plurality of original Internet domain knowledge include a plurality of original videos. The knowledge vector library construction module 114 is used to divide the original video into a plurality of video frames; divide the plurality of video frames into a plurality of video frame clusters; for each video frame cluster, obtain the text description information of the video frame cluster, the multi-frame images corresponding to the video frame cluster, and the audio information corresponding to the video frame cluster as the video knowledge unit corresponding to the video frame cluster; input the text description information, the multi-frame images, and the audio information of the video knowledge unit into the multimodal vector representation model to obtain the multimodal feature vector of the video knowledge unit; at least use the multimodal feature vector as the vector after the video knowledge unit is vectorized.

[0189] Optionally, the knowledge vector library construction module 114 is used to extract image features from the images of each video frame to obtain image extraction features of each video frame; convert the voice information of each video frame into first text information, and extract text features from each first text information to obtain text extraction features of each video frame; perform attention processing on the image extraction features and text extraction features of each video frame to obtain fusion features corresponding to each video frame; and divide a number of video frames into a number of video frame clusters based on the similarity between the fusion features. And / or, the knowledge vector library construction module 114 is used to convert the audio information of the video frame cluster into second text information; input the multi-frame images and the second text information of the video frame cluster into the multimodal large model to obtain text description information of the video frame cluster.

[0190] Optionally, the multimodal vector representation model includes a first text vector representation network, a first image vector representation network, an audio vector representation network, and an attention processing network. The knowledge vector library construction module 114 is used to use the first text vector representation network, the first image vector representation network, and the audio vector representation network to process the text description information, multi-frame image, and audio information of the video knowledge unit, respectively, to obtain the first text feature vector, the first image feature vector, and the audio feature vector of the video knowledge unit; perform modal weighting processing on the first text feature vector, the first image feature vector, and the audio feature vector; and use the attention processing network to perform attention processing on the feature vector after modal weighting processing to obtain a multimodal feature vector.

[0191] Optionally, the knowledge vector library construction module 114 is used to input the text description information of the video knowledge unit into a second text vector representation network to obtain a second text feature vector of the video knowledge unit; input multiple frame images of the video knowledge unit into a second image vector representation network to obtain a second image feature vector of the video knowledge unit; and use the second text feature vector, the second image feature vector and the multimodal feature vector as the vector after the video knowledge unit is vectorized.

[0192] Optionally, the plurality of original private domain knowledge and the plurality of original Internet domain knowledge both include a plurality of original text knowledge, the original text knowledge being an original document, or the original text knowledge being obtained by speech recognition of the original audio. The knowledge vector library construction module 114 is used to extract text-related information of the original text knowledge; input the text-related information into the semantic segmentation large model to obtain the semantic segmentation result of the original text knowledge; based on the semantic segmentation result, the original text knowledge is segmented into a plurality of text knowledge units; the text knowledge unit is input into the third text vector representation network to obtain the third text feature vector corresponding to the text knowledge unit.

[0193] It should be noted that the device of this embodiment can execute the steps in the above method. For detailed description of the relevant content, please refer to the above method part, which will not be repeated here.

[0194] See also Fig.12 , Fig.12 1 is a schematic diagram of a framework of an electronic device according to an embodiment of the present application. In this embodiment, the electronic device 120 includes a memory 121 and a processor 122 .

[0195] The processor 122 may also be referred to as a CPU (Central Processing Unit). The processor 122 may be an integrated circuit chip having signal processing capabilities. The processor 122 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 122 may also be any conventional processor 122, etc.

[0196] The memory 121 in the electronic device 120 is used to store program instructions required for the processor 122 to run.

[0197] The processor 122 is used to execute program instructions to implement the question-answering method in the present application.

[0198] See also Fig.13 , Fig.13 It is a schematic diagram of the framework of an embodiment of a computer-readable storage medium provided by the present application. The computer-readable storage medium 130 of the embodiment of the present application stores a program instruction 131, and the program instruction 131 implements the question-and-answer method provided by the present application when it is executed. Among them, the program instruction 131 can form a program file and be stored in the above-mentioned computer-readable storage medium 130 in the form of a software product, so that a computer device (which can be a personal computer, a server, or a network device, etc.) executes all or part of the steps of each implementation method of the present application. The aforementioned computer-readable storage medium 130 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, and a tablet.

[0199] In the above scheme, after retrieving several candidate knowledge matching the user's question, the target attention information of several candidate knowledge is further extracted, and the candidate knowledge and the target attention information are input into the question-answering big model together, and the target answer result of the user's question is generated by the question-answering big model. In this way, the question-answering big model can pay attention to the target attention information during the real-time answering process, assist the question-answering big model in understanding and integrating several candidate knowledge, reduce the difficulty of understanding of the question-answering big model, and thus improve the accuracy of the question-answering big model's answer.

[0200] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0201] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0202] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and systems can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0203] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0204] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0205] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0206] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A question-answering method, characterized in that: The method comprises: Retrieve several candidate knowledge matching the user's question; Extracting target attention information of the plurality of candidate knowledge; The plurality of candidate knowledge and the target focus information are input into the question-answering big model to obtain the target answer result regarding the user question output by the question-answering big model.

2. The method according to claim 1, characterized in that: The plurality of candidate knowledge includes a plurality of candidate private domain knowledge and a plurality of candidate Internet domain knowledge, and the target attention information includes at least one of the following information: a matching degree between each of the candidate knowledge and the user question, a knowledge domain to which each of the candidate knowledge belongs, a priority order of each of the candidate knowledge, and association information between the candidate private domain knowledge and the candidate Internet domain knowledge; Among them, the priority order of the candidate private domain knowledge is higher than the priority order of the candidate Internet domain knowledge.

3. The method according to claim 1, characterized in that The plurality of candidate knowledge are divided into at least one candidate knowledge set, each of the candidate knowledge sets includes at least one candidate private domain knowledge and at least one candidate Internet domain knowledge belonging to a modality, different candidate knowledge sets correspond to different modalities, the target attention information includes sub-attention information corresponding to each of the candidate knowledge sets, and the question-answering large model includes a large language model and an encoding network of at least one modality; The step of inputting the plurality of candidate knowledge and the target attention information into the question-answering big model to obtain the target answer result regarding the user question output by the question-answering big model includes: For each of the candidate knowledge sets, using the encoding network of the corresponding modality to encode the candidate knowledge set and the sub-attention information of the candidate knowledge set to obtain encoding features of the corresponding modality; The target answer result is generated by utilizing the large language model based on the encoding features of each modality.

4. The method according to claim 3, characterized in that The at least one candidate knowledge set includes a first candidate knowledge set of text modality, a second candidate knowledge set of audio modality and a third candidate knowledge set of video modality, the target attention information includes first sub-attention information of the first candidate knowledge set, second sub-attention information of the second candidate knowledge set and third sub-attention information of the third candidate knowledge set, the encoding network of the at least one modality includes a text encoding network, an audio encoding network and a video encoding network, the audio encoding network includes a first audio encoding subnetwork and a second audio encoding subnetwork, and the video encoding network includes a first video encoding subnetwork and a second video encoding subnetwork.

5. The method according to claim 4, characterized in that For each of the candidate knowledge sets, encoding the candidate knowledge set and the sub-attention information of the candidate knowledge set using the encoding network of the corresponding modality to obtain the encoding features of the corresponding modality includes: Fusing the first sub-attention information with each candidate knowledge in the first candidate knowledge set to obtain fused text knowledge; and inputting the fused text knowledge into the text encoding network to obtain text encoding features; Encode the target sub-attention information to obtain an attention information encoding feature; use the first encoding sub-network to encode the target candidate knowledge set to obtain a first encoding feature; perform a multiplication operation on the attention information encoding feature and the first encoding feature; and use the second encoding sub-network to encode the result of the multiplication operation to obtain a second encoding feature; Among them, when the target sub-attention information is the second sub-attention information, the first encoding subnetwork, the second encoding subnetwork, the target candidate knowledge set, the first encoding feature and the second encoding feature are respectively the first audio encoding subnetwork, the second audio encoding subnetwork, the second candidate knowledge set, the first audio encoding feature and the second audio encoding feature; when the target sub-attention information is the third sub-attention information, the first encoding subnetwork, the second encoding subnetwork, the target candidate knowledge set, the first encoding feature and the second encoding feature are respectively the first video encoding subnetwork, the second video encoding subnetwork, the third candidate knowledge set, the first video encoding feature and the second video encoding feature.

6. The method according to claim 1, characterized in that The retrieval of several candidate knowledge matching the user's question includes: Acquire a private domain knowledge vector library and an Internet domain knowledge vector library; wherein the private domain knowledge vector library includes a number of private domain knowledge vectors, and the Internet domain knowledge vector library includes a number of Internet domain knowledge vectors; Converting the user question into at least one search vector; For each of the search vectors, the search vector is matched with the private domain knowledge vector library and the Internet domain knowledge vector library respectively to obtain at least one candidate private domain knowledge vector in the private domain knowledge vector library that matches the search vector, and at least one first candidate Internet domain knowledge vector in the Internet domain knowledge vector library that matches the search vector; At least the original private domain knowledge corresponding to each of the candidate private domain knowledge vectors and the original Internet domain knowledge corresponding to each of the first candidate Internet domain knowledge vectors are used as the plurality of candidate knowledge.

7. The method according to claim 6, characterized in that The at least original private domain knowledge corresponding to each of the candidate private domain knowledge vectors and the original Internet domain knowledge corresponding to each of the first candidate Internet domain knowledge vectors as the plurality of candidate knowledge include: For each of the candidate private domain knowledge vectors, determining at least one second candidate Internet domain knowledge vector associated with the candidate private domain knowledge vector in the Internet domain knowledge vector library; The original private domain knowledge corresponding to each of the candidate private domain knowledge vectors, the original Internet domain knowledge corresponding to each of the first candidate Internet domain knowledge vectors, and the original Internet domain knowledge corresponding to each of the second candidate Internet domain knowledge vectors are used as the plurality of candidate knowledge.

8. The method according to claim 6, characterized in that Constructing the private domain knowledge vector library and the Internet domain knowledge vector library includes the following steps: Perform knowledge segmentation on some original private domain knowledge to obtain some private domain knowledge units; Perform knowledge segmentation on a number of original Internet domain knowledge to obtain a number of Internet domain knowledge units; Performing vectorization processing on each of the private domain knowledge units and each of the Internet domain knowledge units, respectively, to obtain a private domain knowledge vector corresponding to each of the private domain knowledge units and an Internet domain knowledge vector corresponding to each of the Internet domain knowledge units; Each of the private domain knowledge vectors is added to the private domain knowledge vector library, and each of the Internet domain knowledge vectors is added to the Internet domain knowledge vector library.

9. The method according to claim 8, characterized in that The plurality of original private domain knowledge and the plurality of original Internet domain knowledge include a plurality of original videos; The step of performing the knowledge segmentation on each of the original videos comprises: Dividing the original video into a plurality of video frames; Dividing the plurality of video frames into a plurality of video frame clusters; For each of the video frame clusters, obtaining text description information of the video frame cluster, multiple frames of images corresponding to the video frame cluster, and audio information corresponding to the video frame cluster as a video knowledge unit corresponding to the video frame cluster; The step of performing the vectorization processing on each of the video knowledge units includes: Inputting the text description information, the multiple frames of images and the audio information of the video knowledge unit into a multimodal vector representation model to obtain a multimodal feature vector of the video knowledge unit; At least the multimodal feature vector is used as the vector after the video knowledge unit is vectorized.

10. The method according to claim 9, characterized in that The dividing the plurality of video frames into a plurality of video frame clusters comprises: Extracting image features from the images of each of the video frames to obtain image extraction features of each of the video frames; Converting the speech information of each video frame into first text information, and performing text feature extraction on each first text information to obtain text extraction features of each video frame; Performing attention processing on the image extraction features and the text extraction features of each of the video frames to obtain fusion features corresponding to each of the video frames; Based on the similarity between the fusion features, the plurality of video frames are divided into the plurality of video frame clusters; And / or, the obtaining text description information of the video frame cluster includes: Converting the audio information of the video frame cluster into second text information; The multiple frames of images of the video frame cluster and the second text information are input into a multimodal large model to obtain the text description information of the video frame cluster.

11. The method according to claim 9, characterized in that The multimodal vector representation model includes a first text vector representation network, a first image vector representation network, an audio vector representation network and an attention processing network; The step of inputting the text description information, the multiple frames of images and the audio information of the video knowledge unit into a multimodal vector representation model to obtain a multimodal feature vector of the video knowledge unit includes: Using the first text vector representation network, the first image vector representation network and the audio vector representation network to process the text description information, the multiple frames of images and the audio information of the video knowledge unit respectively, to obtain a first text feature vector, a first image feature vector and an audio feature vector of the video knowledge unit; Performing modal weighting processing on the first text feature vector, the first image feature vector, and the audio feature vector; The attention processing network is used to perform attention processing on the feature vector after modality weighting processing to obtain the multimodal feature vector.

12. The method according to claim 9, characterized in that The at least using the multimodal feature vector as the vector after the video knowledge unit is vectorized includes: Inputting the text description information of the video knowledge unit into a second text vector representation network to obtain a second text feature vector of the video knowledge unit; Inputting the multiple frames of images of the video knowledge unit into a second image vector representation network to obtain a second image feature vector of the video knowledge unit; The second text feature vector, the second image feature vector and the multimodal feature vector are used as vectors of the video knowledge unit.

13. An electronic device, characterized in that: comprising a memory and a processor coupled to each other, The memory stores program instructions; The processor is used to execute the program instructions stored in the memory to implement the method according to any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program instructions, and the program instructions can be executed by a processor to implement the method according to any one of claims 1 to 12.

Images