Multi-modal knowledge-based question answering method and system for 5g message

Abstract

Disclosed in the embodiments of the present invention are a multi-modal knowledge-based question answering method and system for a 5G message. The method comprises: in response to an input request of a user, acquiring a user input, and performing modal conversion and vector encoding on the user input, so as to obtain feature vectors; performing single-modal content retrieval on the basis of the feature vectors, so as to obtain the first k retrieval results of each modal which are similar to the user input; combining the retrieval results of the modals to obtain several retrieval result combinations, fusing single-modal feature vectors in each retrieval result combination to obtain a fused feature vector, inputting the fused feature vectors into a comprehensive score prediction model to obtain predicted relevance scores, and sorting the relevance scores to obtain the most relevant retrieval result combination; and outputting a 5G message answer on the basis of the most relevant retrieval result combination. The problems in the prior art of it not being possible to directly process and understand multi-modal data and a single-mode interaction mode not being able to meet diversified requirements of users are solved.

Description

Multimodal knowledge question answering method and system for 5G messages Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a multimodal knowledge question-answering method, system, electronic device, and storage medium for 5G messages. Background Art

[0002] With the rapid development of 5G technology, data types and industry content have become more diverse and complex, leading to a higher demand for richer and more intelligent interaction methods. Currently, large models have achieved significant breakthroughs in natural language processing and image vision processing, demonstrating their powerful expression and comprehension capabilities. They have also promoted the further development of AIGC, with significant results in conversational question-answering and intelligent customer service. However, knowledge question-answering for 5G messaging still faces the following challenges:

[0003] Most existing knowledge question-and-answer systems require steps such as account registration, login, and payment, which have high user costs. In addition, 5G message templates can include multiple media types, such as text, images, videos, buttons, links, etc. Traditional knowledge question-and-answer systems are mainly based on a single interaction mode, such as text input or voice input. The single-modal interaction method cannot meet the diverse needs of users and is difficult to handle complex problems.

[0004] Knowledge question answering based on text input or keyword retrieval lacks the ability to uniformly represent and understand multimodal data. Traditional text input or keyword retrieval methods cannot directly process and understand multimodal data, such as images and videos.

[0005] Therefore, there is an urgent need for a knowledge question-answering method for 5G messages that can effectively improve the accuracy and consistency of knowledge questions and answers and is more intelligent and convenient.

[0006] Summary of the Invention

[0007] The purpose of the embodiments of the present invention is to provide a multimodal knowledge question-answering method, system, electronic device and storage medium for 5G messages, so as to solve the problem that the existing technology cannot directly process and understand multimodal data and the single-modal interaction method cannot meet the diverse needs of users.

[0008] To achieve the above objectives, an embodiment of the present invention provides a multimodal knowledge question-answering method for 5G messages, which specifically includes:

[0009] In response to a user input request, obtaining user input, performing modal conversion and vector encoding on the user input to obtain a feature vector;

[0010] Performing unimodal content retrieval based on the feature vector to obtain top k retrieval results for each modality that are similar to the user input;

[0011] Combining the search results of each modality to obtain several search result combinations, fusing the single-modal feature vectors in each search result combination to obtain a fused feature vector, inputting the fused feature vector into a comprehensive score prediction model to obtain a predicted relevance score, and sorting the relevance scores to obtain the most relevant search result combination;

[0012] Output 5G message answers based on the most relevant search results combination.

[0013] On the basis of the above technical solution, the present invention can also be improved as follows:

[0014] Furthermore, the step of obtaining user input in response to a user input request and performing modal conversion and vector encoding on the user input to obtain a feature vector includes:

[0015] The X2L encoder based on the X-LLM multimodal large model converts multimodal user input into an embedding sequence;

[0016] Wherein, the user input includes text / voice input Q T 、Image inputQ I and video input Q V ; The embedding sequence includes the text / speech input Q T The feature vector E obtained after encoding by the encoder T 、Image inputQ I The feature vector E obtained after encoding by the encoder I and video input Q V The feature vector E obtained after encoding by the encoder V .

[0017] Furthermore, the single-modal content retrieval based on the feature vector obtains top k retrieval results for each modality that are similar to the user input, including:

[0018] Through the scoring function S x Quantify the relevance of user input to retrieval results.

[0019] Furthermore, the scoring function S x The relevance score of the quantitative input questions of the text information and the search results includes:

[0020] Calculate the text / speech input Q by formula 1 T The relevance score of the search results; S T =Sim(V T ,ET ,k) Formula 1;

[0021] Where S T Enter Q for text / voice T The relevance score of the search results, Sim is the similarity calculation function, V T is the vector representation of all text data in the 5G message knowledge base, E T Enter Q for text / voice T The feature vector obtained after encoding by the encoder, k is the number of retrieval returns;

[0022] Calculate the image input Q using formula 2 I The relevance score of the search results; S I =Sim(V I ,E I ,k) Formula 2;

[0023] Where S I Enter Q for the picture I The relevance score of the search results, Sim is the similarity calculation function, V I is the vector representation of all image data in the 5G message knowledge base, E I Enter Q for the picture I The feature vector obtained after encoding by the encoder, k is the number of retrieval returns;

[0024] Calculate the video input Q by formula 3 V The relevance score of the search results; S V =Sim(V V ,E V ,k) Formula 3;

[0025] Where S V Input Q for video V The relevance score of the search results, Sim is the similarity calculation function, V V is the vector representation of all video data in the 5G message knowledge base, E V Input Q for video V The feature vector obtained after encoding by the encoder, k is the number of retrieval returns.

[0026] Furthermore, the retrieval results of each modality are combined to obtain several retrieval result combinations, the single-modal feature vectors in each retrieval result combination are fused to obtain a fused feature vector, the fused feature vector is input into a comprehensive score prediction model to obtain a predicted relevance score, and the relevance scores are sorted to obtain the most relevant retrieval result combination, including:

[0027] Add a question to each historical 5G message, use the entire historical 5G message content as the answer, and assign a score label to the degree of match between the question and the 5G message content;

[0028] Vector encoding is performed on the historical 5G messages with the score labels to obtain the feature vector of each historical 5G message of each modality;

[0029] Concatenating the feature vectors of each modality to obtain a concatenated feature vector, and constructing a training set based on the concatenated feature vector;

[0030] Construct a comprehensive score prediction model;

[0031] The training set is input into the comprehensive score prediction model for training to obtain a trained comprehensive score prediction model.

[0032] Furthermore, the method further comprises combining the search results of each modality to obtain a plurality of search result combinations, fusing the single-modal feature vectors in each search result combination to obtain a fused feature vector, inputting the fused feature vector into a comprehensive score prediction model to obtain a predicted relevance score, and sorting the relevance scores to obtain the most relevant search result combination, further comprising:

[0033] Use a three-layer perceptron MLP for iterative training to obtain the target minimization loss function so that the predicted score of the comprehensive rating prediction model fits the true label score;

[0034] The minimization loss function is calculated by formula 4;

[0035] Where L(W) is the overall loss function, N is the total training sample data, i is the i-th sample data input, l is the loss of a single sample, f is the model function, W is the weight parameter, y i is the label of the i-th sample, F x,i is the fused feature vector;

[0036] The output score predicted by the i-th sample model is calculated using Formula 5:

[0037] Where W is the weight parameter, is the output score predicted by the model for the i-th sample, F x,i is the fused feature vector.

[0038] Furthermore, the outputting of the 5G message answer based on the most relevant search result combination includes:

[0039] After generating the 5G message answer based on the most relevant search result combination, the content of each modality is parsed based on the type of the 5G message answer, and after placing the content of each modality of the 5G message answer in the corresponding position of the 5G message template, the 5G message answer is sent to the user side.

[0040] A multimodal knowledge question-answering system for 5G messages, comprising:

[0041] A feature vector acquisition module is used to respond to a user input request, obtain user input, and perform modal conversion and vector encoding on the user input to obtain a feature vector;

[0042] A single-modal content retrieval module, configured to perform single-modal content retrieval based on the feature vector and obtain the top k retrieval results for each modality that are similar to the user input;

[0043] A multimodal scoring module is used to combine the search results of each modality to obtain several search result combinations, fuse the single-modal feature vectors in each search result combination to obtain a fused feature vector, input the fused feature vector into the comprehensive score prediction model to obtain a predicted relevance score, and sort the relevance scores to obtain the most relevant search result combination;

[0044] The 5G message answer output module is used to output the 5G message answer based on the most relevant search result combination.

[0045] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the method are implemented when the processor executes the computer program.

[0046] A non-transitory computer-readable storage medium stores a computer program, which implements the steps of the method when executed by a processor.

[0047] The embodiments of the present invention have the following advantages:

[0048] The multimodal knowledge question-answering method for 5G messages in the present invention responds to the user's input request, obtains the user input, performs modal conversion and vector encoding on the user input to obtain a feature vector; performs single-modal content retrieval based on the feature vector to obtain the top k retrieval results for each modality that are similar to the user input; combines the retrieval results of each modality to obtain several retrieval result combinations, fuses the single-modal feature vectors in each retrieval result combination to obtain a fused feature vector, inputs the fused feature vector into a comprehensive score prediction model to obtain a predicted relevance score, sorts the relevance scores to obtain the most relevant retrieval result combination; outputs the 5G message answer based on the most relevant retrieval result combination, which solves the problem in the prior art that multimodal data cannot be directly processed and understood and the single-modal interaction method cannot meet the diverse needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can, without inventive effort, derive other implementation drawings based on the provided drawings.

[0050] The structures, proportions, sizes, etc. illustrated in this specification are intended solely to complement the contents disclosed herein and to facilitate understanding and reading by persons skilled in the art. They are not intended to limit the conditions under which the present invention may be implemented and therefore have no substantive technical significance. Any structural modifications, changes in proportions, or adjustments in sizes, without affecting the efficacy and objectives of the present invention, shall remain within the scope of the technical contents disclosed herein.

[0051] FIG1 is a first flow chart of a multimodal knowledge question-answering method for 5G messages according to the present invention;

[0052] FIG2 is a second flow chart of the multimodal knowledge question-answering method for 5G messages according to the present invention;

[0053] FIG3 is an architecture diagram of a multimodal knowledge question-answering system for 5G messages according to the present invention;

[0054] FIG4 is a schematic diagram of the physical structure of an electronic device provided by the present invention.

[0055] The accompanying drawings are marked as follows: feature vector acquisition module 10, unimodal content retrieval module 20, multimodal scoring module 30, 5G message answer output module 40, electronic device 50, processor 501, memory 502, and bus 503. DETAILED DESCRIPTION

[0056] The following describes the implementation of the present invention using specific embodiments. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. Obviously, the embodiments described are only a portion of the present invention, not all of it. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0057] Example

[0058] FIG1-FIG2 is a flow chart of an embodiment of a multimodal knowledge question-answering method for 5G messages according to the present invention. As shown in FIG1-FIG2, a multimodal knowledge question-answering method for 5G messages provided by an embodiment of the present invention includes the following steps:

[0059] S101, in response to a user input request, obtaining user input, performing modal conversion and vector encoding on the user input to obtain a feature vector;

[0060] Specifically, user input can be in the form of text / voice (T), images (I), or videos (V). To facilitate content retrieval between user input and the 5G message knowledge base and to enable the large language model to understand the semantic information of each modality, the voice, image, or video input needs to be represented in a unified form. The X2L encoder based on the X-LLM multimodal large model can convert multimodal (text / voice, image, video) data into an embedded sequence representation (Embedding feature vector).

[0061] The X2L encoder based on the X-LLM multimodal large model converts multimodal user input into an embedding sequence;

[0062] Wherein, the user input includes text / voice input Q T 、Image inputQ I and video input Q V ; The embedding sequence includes the text / speech input Q T The feature vector E obtained after encoding by the encoder T 、Image inputQ I The feature vector E obtained after encoding by the encoder I and video input Q V The feature vector E obtained after encoding by the encoder V .

[0063] Text input is converted into an embedding sequence: E T =X2L(Q T )

[0064] Image input is converted into an embedding sequence: E I =X2L(Q I )

[0065] Video input is converted into an embedded sequence: E V =X2L(Q V ).

[0066] S102, performing single-modal content retrieval based on the feature vector, and obtaining the top k retrieval results for each modality that are similar to the user input;

[0067] Specifically, through the scoring function S x Quantify the relevance of user input to retrieval results. x (x∈{T,I,V}) is the relevance score of the retrieval results for input x.

[0068] Calculate the text / speech input Q by formula 1 T The relevance score of the search results; S T =Sim(V T ,E T ,k) Formula 1;

[0069] Where S T Enter Q for text / voice T The relevance score of the search results, Sim is the similarity calculation function, V T is the vector representation of all text data in the 5G message knowledge base, E T Enter Q for text / voice T The feature vector obtained after encoding by the encoder, k is the number of retrieval returns;

[0070] Calculate the image input Q using formula 2 I The relevance score of the search results; S I =Sim(V I ,E I ,k) Formula 2;

[0071] Where S I Enter Q for the picture I The relevance score of the search results, Sim is the similarity calculation function, V I is the vector representation of all image data in the 5G message knowledge base, E I Enter Q for the picture I The feature vector obtained after encoding by the encoder, k is the number of retrieval returns;

[0072] Calculate the video input Q by formula 3 V The relevance score of the search results; S V =Sim(V V ,E V,k) Formula 3;

[0073] Where S V Input Q for video V The relevance score of the search results, Sim is the similarity calculation function, V V is the vector representation of all video data in the 5G message knowledge base, E V Input Q for video V The feature vector obtained after encoding by the encoder, k is the number of retrieval returns.

[0074] S103: Combining the search results of each modality to obtain several search result combinations, fusing the single-modality feature vectors in each search result combination to obtain a fused feature vector, inputting the fused feature vector into a comprehensive score prediction model to obtain a predicted relevance score, and sorting the relevance scores to obtain the most relevant search result combination;

[0075] Specifically, when tackling multimodal data retrieval, simply searching for similarities between each modality (e.g., text, image, video, etc.) and the user input often fails to guarantee content consistency. To identify a set of results that are highly relevant to the user input across multiple modalities, a dynamically updated multimodal retrieval scoring mechanism is required. A multilayer perceptron (MLP) is used to train the weight parameter W. The trained model is then used for prediction to obtain a relevance score for each combination.

[0076] Add a question to each historical 5G message, use the entire historical 5G message content as the answer, and assign a score label to the degree of match between the question and the 5G message content;

[0077] User input, 5G message content, and matching score can be expressed as: S = (Q, T, I, V)

[0078] Among them, Q represents the user input content, T represents the text information in the 5G message, I represents the image information in the 5G message, V represents the video information in the 5G message, and S represents the matching score (scoring label) between the 5G message content and the input question.

[0079] The database provides high-capacity, high-scalability, and flexible data storage capabilities to support large-scale data management and analysis. It stores raw multimodal data in 5G messages, including text, images, and videos. Each data type has a corresponding index vector library to store semantic feature vectors encoded by the model. Furthermore, industry-specific documents, images, and videos are also stored as vector representations after processing.

[0080] Vector encoding is performed on the historical 5G messages with the score labels to obtain the feature vector of each historical 5G message of each modality;

[0081] Concatenating the feature vectors of each modality to obtain a concatenated feature vector, and constructing a training set based on the concatenated feature vector;

[0082] here represents vector concatenation, F x,i is the fused feature vector, E T,i , E I,i , E V,i They represent the feature vectors obtained after the text, picture and video in the i-th training corpus are encoded by the encoder.

[0083] Construct a comprehensive score prediction model;

[0084] The training set is input into the comprehensive score prediction model for training to obtain a trained comprehensive score prediction model.

[0085] Use a three-layer perceptron MLP for iterative training to obtain the target minimization loss function so that the predicted score of the comprehensive rating prediction model fits the true label score;

[0086] The minimization loss function is calculated by formula 4;

[0087] Where L(W) is the overall loss function, N is the total training sample data, i is the i-th sample data input, l is the loss of a single sample, f is the model function, W is the weight parameter, y i is the label of the i-th sample, F x,i is the fused feature vector;

[0088] The output score predicted by the i-th sample model is calculated using Formula 5:

[0089] Where W is the weight parameter, is the output score predicted by the model for the i-th sample, F x,i is the fused feature vector.

[0090] The user input is first retrieved through the single modal content retrieval to obtain the top k retrieval results of each modality. These retrieval results are semantically similar to the user input content, but for the output Y of each modality x,i =(Y T,i ,Y I,i ,Y V,i )(x∈{T,I,V}), the same result set (Y T,1 ,Y I,1 ,YV,1 ), the image content or video content in the result set may have a large semantic deviation from the text content. Combining the three modal results of text, image, and video in the single-modal result set, the combination example is as follows: (Y T,1 ,Y I,1 ,Y V,1 ), (Y T,1 ,Y I,1 ,Y V,2 ), … (Y T,2 ,Y I,1 ,Y V,1 ), (Y T,2 ,Y I,2 ,Y V,1 ), …

[0091] Then there are k*k*k combinations (when k=10, there are 1000 combinations). The single-modal feature vectors in each combination are fused and then input into the comprehensive score prediction model to obtain the predicted relevance score: Score=f(F x,i ,W), then sort them according to the predicted relevance scores and output the most relevant set of results.

[0092] After optimizing the result set through the multimodal retrieval scoring mechanism, a new set of results is obtained, such as (Y T,1 ,Y I,3 ,Y V,2 ), this group of results has the highest relevance score predicted by the comprehensive score prediction model. The highest relevance score indicates a high relevance to the user input content, and the correlation between the text, picture and video content in the group of results with the highest relevance score is also high, and the content is consistent. The trained comprehensive score prediction model can quickly find the most relevant search result combination from a large number of combinations.

[0093] S104, outputting a 5G message answer based on the most relevant search result combination.

[0094] Specifically, after generating a 5G message answer based on the most relevant search result combination, the content of each modality is parsed based on the type of the 5G message answer, and after placing the content of each modality of the 5G message answer in the corresponding position of the 5G message template, the 5G message answer is sent to the user side.

[0095] When the user input is text / voice input Q T When encoding text / speech into E T , based on unimodal content retrieval, k relevant text, image and video data are retrieved respectively.

[0096] When the user input is a picture input Q I When the picture is encoded as EI , based on unimodal content retrieval, k relevant text, image and video data are retrieved respectively.

[0097] When the user input is video input Q V When the video is encoded as E V , based on unimodal content retrieval, k relevant text, image and video data are retrieved respectively.

[0098] When the user input is text / voice input Q T and picture input Q I When the picture is encoded as E I , based on unimodal content retrieval, k relevant text, image and video data are retrieved respectively.

[0099] When the user input is text / voice input Q T and video input Q V When encoding the video as E V , based on unimodal content retrieval, k relevant text, image and video data are retrieved respectively.

[0100] Then, the output Y is obtained based on the multimodal retrieval scoring mechanism x =(Y T ,Y I ,Y V ), get the original text content T, original image I, and original video V from the output result set.

[0101] The final question that the user inputs needs to be refined and summarized, and the answer needs to be humanized, rather than directly outputting the search results to the user. Therefore, the input and output of the final text answer can be expressed as:

[0102] Input: X input =(Prompt,Q T ,T)

[0103] Output: Y output =GPT(X input )

[0104] Prompt is a text answer prompt template (such as "Please generate the answer to the question based on the input content and text content"), and GPT is a large model for text generation.

[0105] This multimodal knowledge question-and-answer method for 5G messages responds to a user's input request, obtains user input, modally converts and vector-encodes the user input to obtain a feature vector; performs a unimodal content search based on the feature vector to obtain the top k retrieval results for each modality that are similar to the user input; combines the retrieval results for each modality to obtain several retrieval result combinations; fuses the unimodal feature vectors in each retrieval result combination to obtain a fused feature vector; inputs the fused feature vector into a comprehensive score prediction model to obtain a predicted relevance score; sorts the relevance scores to obtain the most relevant retrieval result combination; and outputs a 5G message answer based on the most relevant retrieval result combination. This method solves the problem in existing technologies that multimodal data cannot be directly processed and understood, and that unimodal interaction methods cannot meet the diverse needs of users.

[0106] This multimodal knowledge question-answering method for 5G messages leverages the advantages of large models and database capabilities to uniformly represent and store the multimodal data in 5G messages. By vectorizing different types of data, such as text, images, and videos, it effectively integrates multimodal data.

[0107] A multimodal search scoring mechanism has been designed to ensure the consistency of search results, improving the accuracy and consistency of knowledge questions and answers. This deep learning-based scoring mechanism continuously updates the scoring model based on user feedback, continuously optimizing the accuracy of search results.

[0108] By using a large model for answer generation, intelligent responses can be generated based on user input and search results. Furthermore, if the search results lack relevant images or videos, corresponding images or video data can be generated based on the text content, providing richer and more personalized responses.

[0109] Traditional knowledge quizzes are primarily based on a single interactive model and are difficult to handle complex questions. The multimodal knowledge quiz method of the present invention can simultaneously process input from multiple media types, such as text, images, and videos, meeting the diverse needs of users and providing more comprehensive and accurate answers.

[0110] FIG3 is a flow chart of an embodiment of a multimodal knowledge question-answering system for 5G messages according to the present invention. As shown in FIG3 , an embodiment of the present invention provides a multimodal knowledge question-answering system for 5G messages, including the following steps:

[0111] The feature vector acquisition module 10 is used to obtain user input in response to a user input request, and perform modal conversion and vector encoding on the user input to obtain a feature vector;

[0112] A single-modal content retrieval module 20 is configured to perform single-modal content retrieval based on the feature vector and obtain top k retrieval results for each modality that are similar to the user input;

[0113] A multimodal scoring module 30 is configured to combine the search results of each modality to obtain a plurality of search result combinations, fuse the single-modal feature vectors in each search result combination to obtain a fused feature vector, input the fused feature vector into a comprehensive score prediction model to obtain a predicted relevance score, and sort the relevance scores to obtain the most relevant search result combination;

[0114] The 5G message answer output module 40 is used to output the 5G message answer based on the most relevant search result combination.

[0115] The feature vector acquisition module 10 is further configured to:

[0116] The X2L encoder based on the X-LLM multimodal large model converts multimodal user input into an embedding sequence;

[0117] Wherein, the user input includes text / voice input Q T 、Image inputQ I and video input Q V ; The embedding sequence includes the text / speech input Q T The feature vector E obtained after encoding by the encoder T 、Image inputQ I The feature vector E obtained after encoding by the encoder I and video input Q V The feature vector E obtained after encoding by the encoder V .

[0118] The single-modal content retrieval module 20 is further configured to:

[0119] Through the scoring function S x Quantify the relevance of user input to retrieval results.

[0120] Calculate the text / speech input Q by formula 1 T The relevance score of the search results; S T =Sim(V T ,E T ,k) Formula 1;

[0121] Where S T Enter Q for text / voice T The relevance score of the search results, Sim is the similarity calculation function, V T is the vector representation of all text data in the 5G message knowledge base, E TEnter Q for text / voice T The feature vector obtained after encoding by the encoder, k is the number of retrieval returns;

[0122] Calculate the image input Q using formula 2 I The relevance score of the search results; S I =Sim(V I ,E I ,k) Formula 2;

[0123] Where S I Enter Q for the picture I The relevance score of the search results, Sim is the similarity calculation function, V I is the vector representation of all image data in the 5G message knowledge base, E I Enter Q for the picture I The feature vector obtained after encoding by the encoder, k is the number of retrieval returns;

[0124] Calculate the video input Q by formula 3 V The relevance score of the search results; S V =Sim(V V ,E V ,k) Formula 3;

[0125] Where S V Input Q for video V The relevance score of the search results, Sim is the similarity calculation function, V V is the vector representation of all video data in the 5G message knowledge base, E V Input Q for video V The feature vector obtained after encoding by the encoder, k is the number of retrieval returns.

[0126] The multimodal scoring module 30 is further configured to:

[0127] Add a question to each historical 5G message, use the entire historical 5G message content as the answer, and assign a score label to the degree of match between the question and the 5G message content;

[0128] Vector encoding is performed on the historical 5G messages with the score labels to obtain the feature vector of each historical 5G message of each modality;

[0129] Concatenating the feature vectors of each modality to obtain a concatenated feature vector, and constructing a training set based on the concatenated feature vector;

[0130] Construct a comprehensive score prediction model;

[0131] The training set is input into the comprehensive score prediction model for training to obtain a trained comprehensive score prediction model.

[0132] Use a three-layer perceptron MLP for iterative training to obtain the target minimization loss function so that the predicted score of the comprehensive rating prediction model fits the true label score;

[0133] The minimization loss function is calculated by formula 4;

[0134] Where L(W) is the overall loss function, N is the total training sample data, i is the i-th sample data input, l is the loss of a single sample, f is the model function, W is the weight parameter, y i is the label of the i-th sample, F x,i is the fused feature vector;

[0135] The output score predicted by the i-th sample model is calculated using Formula 5:

[0136] Where W is the weight parameter, is the output score predicted by the model for the i-th sample, F x,i is the fused feature vector.

[0137] The 5G message answer output module 40 is further used to:

[0138] After generating the 5G message answer based on the most relevant search result combination, the content of each modality is parsed based on the type of the 5G message answer, and after placing the content of each modality of the 5G message answer in the corresponding position of the 5G message template, the 5G message answer is sent to the user side.

[0139] The present invention discloses a multimodal knowledge question-and-answer system for 5G messages. A feature vector acquisition module 10 responds to a user's input request, acquires user input, modally converts and vector encodes the user input to obtain a feature vector. A single-modal content retrieval module 20 performs single-modal content retrieval based on the feature vector to obtain the top k retrieval results for each modality that are similar to the user input. A multimodal scoring module 30 combines the retrieval results of each modality to obtain several retrieval result combinations, fuses the single-modal feature vectors in each retrieval result combination to obtain a fused feature vector, inputs the fused feature vector into a comprehensive scoring prediction model to obtain a predicted relevance score, and sorts the relevance scores to obtain the most relevant retrieval result combination. A 5G message answer output module 40 outputs a 5G message answer based on the most relevant retrieval result combination. This multimodal knowledge question-and-answer method for 5G messages addresses the problems in the prior art of being unable to directly process and understand multimodal data and the inability of single-modal interaction methods to meet the diverse needs of users.

[0140] FIG4 is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. As shown in FIG4 , the electronic device 50 includes: a processor 501 (processor), a memory 502 (memory) and a bus 503;

[0141] The processor 501 and the memory 502 communicate with each other via the bus 503.

[0142] The processor 501 is used to call the program instructions in the memory 502 to execute the methods provided by the above-mentioned method embodiments, for example, including: responding to the user's input request, obtaining the user input, performing modal conversion and vector encoding on the user input to obtain a feature vector; performing single-modal content retrieval based on the feature vector, and obtaining the top k retrieval results for each modality that are similar to the user input; combining the retrieval results of each modality to obtain several retrieval result combinations, fusing the single-modal feature vectors in each retrieval result combination to obtain a fused feature vector, inputting the fused feature vector into a comprehensive score prediction model to obtain a predicted relevance score, sorting the relevance score to obtain the most relevant retrieval result combination; and outputting a 5G message answer based on the most relevant retrieval result combination.

[0143] This embodiment provides a non-transitory computer-readable storage medium storing computer instructions that cause a computer to execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining user input in response to a user input request, performing modal conversion and vector encoding on the user input to obtain a feature vector; performing unimodal content retrieval based on the feature vector to obtain the top k retrieval results for each modality that are similar to the user input; combining the retrieval results of each modality to obtain a plurality of retrieval result combinations, fusing the unimodal feature vectors in each retrieval result combination to obtain a fused feature vector, inputting the fused feature vector into a comprehensive score prediction model to obtain a predicted relevance score, ranking the relevance scores to obtain the most relevant retrieval result combination; and outputting a 5G message answer based on the most relevant retrieval result combination. It will be understood by those skilled in the art that all or part of the steps of the above-mentioned method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above-mentioned method embodiments. The aforementioned storage medium includes various storage media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0144] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. That is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0145] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by means of hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or certain parts of the embodiment.

[0146] Although the present invention has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications and improvements may be made thereto. Therefore, such modifications and improvements, without departing from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. A multimodal knowledge question-answering method for 5G messages, characterized in that: The method specifically comprises: In response to a user input request, obtaining user input, performing modal conversion and vector encoding on the user input to obtain a feature vector; Performing unimodal content retrieval based on the feature vector to obtain top k retrieval results for each modality that are similar to the user input; Combining the search results of each modality to obtain a number of search result combinations, fusing the single-modality feature vectors in each search result combination to obtain a fused feature vector, inputting the fused feature vector into a comprehensive score prediction model to obtain a predicted relevance score, and sorting the relevance scores to obtain the most relevant search result combination; Output 5G message answers based on the most relevant search results combination.

2. The multimodal knowledge question-answering method for 5G messages according to claim 1, characterized in that: The step of obtaining user input in response to a user input request and performing modal conversion and vector encoding on the user input to obtain a feature vector includes: The X2L encoder based on the X-LLM multimodal large model converts multimodal user input into an embedding sequence; Wherein, the user input includes text / voice input Q T 、Image inputQ I and video input Q V ; The embedding sequence includes the text / speech input Q T The feature vector E obtained after encoding by the encoder T 、Image inputQ I The feature vector E obtained after encoding by the encoder I and video input Q V The feature vector E obtained after encoding by the encoder V .

3. The multimodal knowledge question-answering method for 5G messages according to claim 1, characterized in that: The single-modal content retrieval based on the feature vector obtains the top k retrieval results for each modality that are similar to the user input, including: Through the scoring function s x Quantify the relevance of user input to retrieval results.

4. The multimodal knowledge question-answering method for 5G messages according to claim 3 is characterized in that: The scoring function S x The quantitative input questions of the text information and the relevance score of the search results include: Calculate the text / speech input Q by formula 1 T The relevance score of the search results; S T =Sim(V T ,E T ,k) Formula 1; In the formula, S T Q for text / voice input T The relevance score of the search results, Sim is the similarity calculation function, V T is the vector representation of all text data in the 5G message knowledge base, E T Q for text / voice input T The feature vector obtained after encoding by the encoder, k is the number of retrieval returns; Calculate the image input Q by formula 2 I The relevance score of the search results; S I =Sim(V I ,E I ,k) Formula 2; In the formula, S I Type Q for the image I The relevance score of the search results, Sim is the similarity calculation function, V I is the vector representation of all image data in the 5G message knowledge base, E I Type Q for the image I The feature vector obtained after encoding by the encoder, k is the number of retrieval returns; The video input Q is calculated by formula 3 V The relevance score of the search results; S V =Sim(V V ,E V ,k) Formula 3; In the formula, S V Q for video input V The relevance score of the search results, Sim is the similarity calculation function, V V is the vector representation of all video data in the 5G message knowledge base, E V Q for video input V The feature vector obtained after encoding by the encoder, k is the number of retrieval returns.

5. The multimodal knowledge question-answering method for 5G messages according to claim 1, characterized in that: The retrieval results of each modality are combined to obtain a plurality of retrieval result combinations, the single modality feature vectors in each retrieval result combination are fused to obtain a fused feature vector, the fused feature vector is input into a comprehensive score prediction model to obtain a predicted relevance score, the relevance scores are sorted to obtain the most relevant retrieval result combination, including: Add a question to each historical 5G message, use the entire historical 5G message content as the answer, and give a score label to the degree of match between the question and the 5G message content; Vector encoding is performed on the historical 5G messages marked with the scoring labels to obtain a feature vector of each historical 5G message of each modality; Concatenating the feature vectors of each modality to obtain a concatenated feature vector, and constructing a training set based on the concatenated feature vector; Construct a comprehensive score prediction model; The training set is input into the comprehensive score prediction model for training to obtain a trained comprehensive score prediction model.

6. The multimodal knowledge question-answering method for 5G messages according to claim 5, characterized in that: The method further comprises combining the search results of each modality to obtain a plurality of search result combinations, fusing the single-modality feature vectors in each search result combination to obtain a fused feature vector, inputting the fused feature vector into a comprehensive score prediction model to obtain a predicted relevance score, and sorting the relevance scores to obtain the most relevant search result combination, and further comprising: Use a three-layer perceptron MLP for iterative training to obtain the target minimization loss function so that the predicted score of the comprehensive rating prediction model fits the true label score; The minimized loss function is calculated by formula 4; Where L(W) is the overall loss function, N is all training sample data, i is the i-th sample data input, l is the loss of a single sample, f is the model function, W is the weight parameter, and y is i is the label of the i-th sample, F x,i is the fused feature vector; The output score predicted by the i-th sample model is calculated using Formula 5: Where W is the weight parameter, is the output score predicted by the model for the i-th sample, F x,i is the fused feature vector.

7. The multimodal knowledge question-answering method for 5G messages according to claim 1, characterized in that: The outputting of the 5G message answer based on the most relevant search result combination includes: After generating the 5G message answer based on the most relevant search result combination, the content of each modality is parsed based on the type of the 5G message answer, and after placing the content of each modality of the 5G message answer in the corresponding position of the 5G message template, the 5G message answer is sent to the user side.

8. A multimodal knowledge question-answering system for 5G messages, characterized in that: include: A feature vector acquisition module, used to respond to a user input request, acquire user input, and perform modal conversion and vector encoding on the user input to obtain a feature vector; A single-modal content retrieval module, used to perform single-modal content retrieval based on the feature vector, and obtain the top k retrieval results for each modality that are similar to the user input; A multimodal scoring module is used to combine the search results of each modality to obtain a plurality of search result combinations, fuse the single-modal feature vectors in each search result combination to obtain a fused feature vector, input the fused feature vector into a comprehensive score prediction model to obtain a predicted relevance score, sort the relevance scores, and obtain the most relevant search result combination; The 5G message answer output module is used to output the 5G message answer based on the most relevant search result combination.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images