Multi-mode intelligent question and answer reasoning method and device based on knowledge distillation

Abstract

The invention relates to a multi-mode intelligent question and answer reasoning method and device based on knowledge distillation, and belongs to the technical field of intelligent question and answer. Extracting multi-modal features of the user data; a pre-trained intention recognition model conducts reasoning on the multi-modal features, reply content is generated, or corresponding operation is executed; the training method of the pre-trained intention recognition model comprises the following steps: acquiring multi-modal data; preprocessing the multi-modal data to obtain a multi-modal fusion vector; performing fine tuning training on the base large model by using the multi-modal fusion vector to obtain a fine-tuned large model; inputting the multi-modal data into the fine-tuned large model for intention classification, judging whether the classified intention slot position is complete or not, and if the judgment result is that the classified intention slot position is incomplete, supplementing the intention to generate final intention information; and performing knowledge distillation on the fine-tuned large model to obtain the model. The real-time performance and accuracy of intelligent question answering are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent question answering, and in particular, relates to a multimodal intelligent question answering reasoning method and device based on knowledge distillation. Background Art

[0002] As one of the important applications in the field of artificial intelligence, intelligent question answering is widely used in business scenarios such as online customer service, office assistant, and public services. Existing intelligent question answering systems have the following problems in terms of reasoning efficiency:

[0003] 1. Large model size: In order to improve the accuracy of responses, existing systems usually use large-scale pre-trained language models. These models contain billions or even hundreds of billions of parameters, and the model size is large, making it difficult to deploy on mobile terminals and edge devices with limited computing resources.

[0004] 2. Slow reasoning speed: Large language models consume a lot of computing resources during the reasoning process, resulting in a slow response speed of the question-answering system, which cannot meet the needs of real-time interaction.

[0005] 3. High resource consumption: Deploying large language models on the server requires high hardware configuration, which leads to increased computing resource consumption and operating costs.

[0006] 4. Single modality: Existing knowledge distillation technology mainly targets single-modal data, such as text or images. In practical applications, the fusion of multimodal data can provide richer information and help improve the applicability of question-answering systems. How to effectively perform knowledge distillation in a multimodal environment to improve the reasoning efficiency of intelligent question-answering systems is still a difficult problem in current technology. Summary of the invention

[0007] In view of the above-mentioned deficiencies in the prior art, the purpose of the invention is to provide a multimodal intelligent question answering reasoning method and device based on knowledge distillation, which improves the real-time and accuracy of intelligent question answering and reduces computing resource requirements and storage costs through technical means such as multimodal data fusion, large model compression and knowledge distillation.

[0008] In a first aspect of the present invention, a multimodal intelligent question-answering reasoning method based on knowledge distillation is proposed, comprising:

[0009] Get user data;

[0010] Extract multimodal features of user data;

[0011] The pre-trained intent recognition model infers the multimodal features and generates reply content or performs corresponding operations;

[0012] Among them, the training method of the pre-trained intent recognition model includes:

[0013] Acquire multimodal data;

[0014] Preprocess the multimodal data to obtain a multimodal fusion vector;

[0015] Use the multimodal fusion vector to fine-tune the base large model to obtain a fine-tuned large model;

[0016] Input the multimodal data into the fine-tuned large model for intent classification, and determine whether the classified intent slots are complete. If the judgment result is that the classified intent slots are incomplete, supplement the intent to generate the final intent information.

[0017] The fine-tuned large model is subjected to knowledge distillation to obtain a pre-trained intent recognition model.

[0018] Furthermore, in the above-mentioned multimodal intelligent question-answering reasoning method based on knowledge distillation, preprocessing the multimodal data to obtain a multimodal fusion vector includes:

[0019] Extract features from text data in multimodal data to obtain text vectors;

[0020] Extract features from audio data in multimodal data to obtain audio vectors;

[0021] Extract features from video data in multimodal data to obtain video vectors;

[0022] Extract features from image data in multimodal data to obtain image vectors;

[0023] The text vector, audio vector, video vector and image vector are fused to obtain a multimodal fusion vector.

[0024] Furthermore, in the above-mentioned multimodal intelligent question-answering reasoning method based on knowledge distillation, the base large model is fine-tuned using the multimodal fusion vector to obtain the fine-tuned large model, including:

[0025] The text, voice, video, and image data are randomly mixed in a ratio of 1:1:1:1, and each part is divided into a training set, a test set, and a validation set according to a preset ratio;

[0026] The intention recognition prompt words and the multimodal fusion vector are concatenated to obtain training data;

[0027] Input the training data into the base model and add multiple fully connected layers to the base model;

[0028] Fine-tune the large base model using the training set and validation set;

[0029] After the loss value is stable, the model parameters of the base large model are saved to obtain the fine-tuned large model;

[0030] The accuracy of the fine-tuned large model is tested on the test set.

[0031] Furthermore, in the above-mentioned multimodal intelligent question-answering reasoning method based on knowledge distillation, if the judgment result is that the classified intent slot is incomplete, the intent is supplemented to generate the final intent information, including:

[0032] Design corresponding slots for different intents, and automatically generate slots through dialogue with users during the question-answering process;

[0033] Determine whether some slot information is missing. If the judgment result is that some slot information is missing, the missing slot information is input into the prompt generator through the prompt word generation module to generate corresponding prompt words to assist the fine-tuned large model in extracting slot information;

[0034] The intent is supplemented by the extracted slot information to obtain the intent output result;

[0035] The intent output results are input into the business dialogue management system, the corresponding business is selected, and the missing slot information is queried through the intent slot extraction unit of the language model to generate the final intent information.

[0036] Furthermore, in the above-mentioned multimodal intelligent question-answering reasoning method based on knowledge distillation, knowledge distillation is performed on the fine-tuned large model to obtain a pre-trained intent recognition model, including:

[0037] The teacher model is the fine-tuned base model, and the student model is the BERT model.

[0038] Through the comparative learning method, the BERT model uses the method of first reducing the dimension and then increasing the dimension to learn the knowledge and prediction results of each layer in the teacher model;

[0039] Among them, the distillation loss function includes: label loss and feature matching loss.

[0040] Furthermore, in the above-mentioned multimodal intelligent question-answering reasoning method based on knowledge distillation, fine-tuning the base large model is performed through a training set and a verification set, including:

[0041] Perform feature extraction on the training set and the validation set, and input the obtained vectors into the base large model;

[0042] Feature combination and classification through fully connected layers;

[0043] Use the cross entropy loss function to calculate the difference between the output of the base model and the true intent label;

[0044] The model parameters of the base large model are updated through the back propagation algorithm.

[0045] Furthermore, in the above-mentioned multimodal intelligent question-answering reasoning method based on knowledge distillation, label loss is used to narrow the gap between the probability distribution of the student model output and the probability distribution of the teacher model output, and feature matching loss is used to narrow the gap between the intermediate layer features of the student model and the intermediate layer features of the teacher model.

[0046] The second aspect of the present invention further proposes a multimodal intelligent question-answering reasoning device based on knowledge distillation, comprising:

[0047] Acquisition module: used to obtain user data;

[0048] Extraction module: used to extract multimodal features of the user data;

[0049] Reasoning module and generation module: used for the pre-trained intent recognition model to reason about the multimodal features, generate reply content or perform corresponding operations;

[0050] The pre-trained intent recognition model is trained using a training device, and the training device includes:

[0051] The first acquisition module is used to acquire multimodal data;

[0052] Preprocessing module: used for preprocessing the multimodal data to obtain a multimodal fusion vector;

[0053] Fine-tuning training module: used to use multimodal fusion vectors to fine-tune the base large model to obtain a fine-tuned large model;

[0054] Slot supplement module: used to input multimodal data into the fine-tuned large model for intent classification, and determine whether the intent slots after classification are complete. If the judgment result is that the intent slots after classification are incomplete, the intent is supplemented to generate the final intent information;

[0055] Knowledge distillation module: used to perform knowledge distillation on the fine-tuned large model to obtain a pre-trained intent recognition model.

[0056] According to a third aspect of the present invention, an electronic device is provided, comprising: a processor and a memory;

[0057] The processor calls the program or instruction stored in the memory to execute any of the above multimodal intelligent question-answering reasoning methods based on knowledge distillation.

[0058] In a fourth aspect of the present invention, a computer-readable storage medium is proposed, which stores a program or instruction, and the program or instruction enables a computer to execute any one of the above multimodal intelligent question-answering reasoning methods based on knowledge distillation.

[0059] The beneficial effects of the present invention are as follows: the present invention obtains user data; extracts multimodal features of user data; the pre-trained intent recognition model infers the multimodal features to generate reply content or perform corresponding operations; wherein the training method of the pre-trained intent recognition model includes: obtaining multimodal data; pre-processing the multimodal data to obtain a multimodal fusion vector; using the multimodal fusion vector to fine-tune the base large model to obtain a fine-tuned large model; inputting the multimodal data into the fine-tuned large model for intent classification, judging whether the classified intent slot is complete, and if the judgment result is that the classified intent slot is incomplete, supplementing the intent to generate the final intent information; performing knowledge distillation on the fine-tuned large model to obtain a pre-trained intent recognition model. The present invention improves the real-time and accuracy of intelligent question and answering and reduces computing resource requirements and storage costs through technical means such as multimodal data fusion, large model compression and knowledge distillation. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings are only used to illustrate specific embodiments and are not considered to limit the present invention. In the entire drawings, the same reference symbols represent the same components. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0061] Figure 1 A diagram of a multimodal intelligent question-answering reasoning method based on knowledge distillation provided in an embodiment of the present invention;

[0062] Figure 2 A diagram of a training method for a pre-trained intent recognition model provided in an embodiment of the present invention;

[0063] Figure 3 A diagram of a method for performing fine-tuning training to obtain a fine-tuned large model provided in an embodiment of the present invention;

[0064] Figure 4 A method diagram for supplementing an intent to generate final intent information provided by an embodiment of the present invention;

[0065] Figure 5 A method diagram for fine-tuning a large base model provided in an embodiment of the present invention;

[0066] Figure 6A diagram of a multimodal intelligent question-answering reasoning device based on knowledge distillation provided in an embodiment of the present invention;

[0067] Figure 7 A schematic block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work should fall within the scope of protection of the present invention.

[0069] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0070] In the description of the present invention, the terms "first", "second", and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be internal communication between two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0071] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of methods and systems consistent with some aspects of the present invention as detailed in the appended claims.

[0072] The present invention proposes a multimodal intelligent question-answering reasoning method, device, electronic device and storage medium based on knowledge distillation. Through technical means such as multimodal data fusion, large model compression and knowledge distillation, the real-time and accuracy of intelligent question-answering are improved, and the computing resource requirements and storage costs are reduced.

[0073] Before introducing the embodiments of the present invention, professional terms involved in the present invention are first introduced.

[0074] Knowledge Distillation: A model compression technique that improves the performance of a student model by transferring knowledge from a large and complex model (teacher model) to a small and simplified model (student model).

[0075] Feature Extraction: A process of extracting useful features from raw data, including text feature extraction, image feature extraction, audio feature extraction, etc.

[0076] Intent Recognition: A process of identifying user intent from input, including text intent recognition, voice intent recognition, etc.

[0077] Slot Extraction: A method for identifying and extracting key information in a conversation, such as time, place, and person, which is called a "slot". Through slot extraction, the system can better understand the user's intentions and generate more accurate responses.

[0078] Loss Function: In deep learning, the loss function is a function that measures the difference between the model prediction and the true value. It is used to evaluate the performance of the model and guide the model training process.

[0079] Method Embodiment

[0080] Figure 1 A diagram of a multimodal intelligent question-answering reasoning method based on knowledge distillation provided in an embodiment of the present invention.

[0081] In the first aspect of the present invention, a multimodal intelligent question answering reasoning method based on knowledge distillation is proposed, combined with Figure 1 , including three steps from S1 to S3:

[0082] S1: Get user data.

[0083] Specifically, in the embodiments of the present invention, user data may be customer consultation data in an intelligent customer service scenario, such as text, voice, image and video data; it may also be family life data in an intelligent furniture control scenario, such as environmental data such as temperature, humidity, air quality and behavioral habits of family members; user data may be data in any scenario, and the scope of protection of the present invention is not limited thereto.

[0084] S2: Extract multimodal features of user data.

[0085] Specifically, in an embodiment of the present invention, extracting multimodal features of user data may be extracting multimodal features from customer consultation data, such as keywords in text, objects and locations in images, emotions in audio, etc., or extracting multimodal features from family life data, such as temperature and humidity in environmental data, as well as behavioral habits of family members.

[0086] S3: The pre-trained intent recognition model infers the multimodal features and generates reply content or performs corresponding actions.

[0087] Specifically, in an embodiment of the present invention, a pre-trained intent recognition model performs question-answering reasoning on keywords in extracted text, objects and locations in images, emotions in audio, etc., generates replies or performs corresponding operations, which can provide more accurate and efficient customer service, reduce manual customer service costs, and improve customer satisfaction; the pre-trained intent recognition model performs reasoning on temperature and humidity in extracted environmental data, generates reply content or performs corresponding operations, such as adjusting the temperature, turning on lights, etc., which can achieve a more intelligent and convenient family life experience, improve the quality of life, and reduce energy consumption.

[0088] Figure 2 A diagram of a training method for a pre-trained intent recognition model provided in an embodiment of the present invention.

[0089] Among them, the training method of the pre-trained intent recognition model is combined with Figure 2 , including five steps from S21 to S25:

[0090] S21: Acquire multimodal data.

[0091] Specifically, in the embodiments of the present invention, multimodal data is a large amount of text, voice, image and video data. Text data refers to text information input by users, from chat records, user comments, etc.; voice data refers to recordings of user voices, usually in WAV or MP3 format; image data refers to related visual information, such as user's environmental photos, screenshots, etc.; video data can be divided into continuous image frames, containing action or scene information.

[0092] S22: Preprocess the multimodal data to obtain a multimodal fusion vector.

[0093] Specifically, in the embodiment of the present invention, preprocessing the multimodal data to obtain the multimodal fusion vector is to prepare clean and unified format training data for subsequent model training. The preprocessing means are described in detail below.

[0094] S23: Use the multimodal fusion vector to fine-tune the base large model to obtain the fine-tuned large model.

[0095] Specifically, in an embodiment of the present invention, a multimodal fusion vector is used to fine-tune the base large model to obtain a fine-tuned large model to accurately classify the intent of the user query, thereby providing precise guidance for subsequent answer generation or information retrieval.

[0096] S24: Input the multimodal data into the fine-tuned large model for intent classification, and determine whether the classified intent slots are complete. If the judgment result is that the classified intent slots are incomplete, supplement the intent to generate the final intent information.

[0097] Specifically, in an embodiment of the present invention, some intents contain specific slots. For example, the intent for air ticket booking needs to provide slots such as the destination and departure time. It is determined whether the classified intent slots are complete. If the judgment result is that the classified intent slots are incomplete, the intent is supplemented to generate the final intent information. The specific judgment and supplementation methods are described in detail below.

[0098] S25: Perform knowledge distillation on the fine-tuned large model to obtain a pre-trained intent recognition model.

[0099] Specifically, in the embodiments of the present invention, the knowledge distillation technology is used to allow the student model to learn to imitate the behavior of the teacher model, so that similar or higher accuracy can be achieved without significant performance loss, thereby solving the challenge of deploying large deep neural network models on edge devices with limited memory and computing power. The model performance is improved and the parameters are compressed, thereby optimizing the reasoning effect of multimodal question answering.

[0100] Furthermore, in the above-mentioned multimodal intelligent question-answering reasoning method based on knowledge distillation, preprocessing the multimodal data to obtain a multimodal fusion vector includes:

[0101] Extract features from text data in multimodal data to obtain text vectors;

[0102] Extract features from audio data in multimodal data to obtain audio vectors;

[0103] Extract features from video data in multimodal data to obtain video vectors;

[0104] Extract features from image data in multimodal data to obtain image vectors;

[0105] The text vector, audio vector, video vector and image vector are fused to obtain a multimodal fusion vector.

[0106] Specifically, in an embodiment of the present invention, data of different modalities are mapped into a vector space using their respective corresponding feature representation algorithms, an attention mechanism is introduced in the knowledge fusion process, the knowledge in text vectors, audio vectors, video vectors, and image vectors are unified into a natural language representation, and the loss function is optimized and the parameters of the vectorized matrix representations of different modalities are adjusted to obtain a multimodal fusion vector.

[0107] Figure 3 A diagram of a method for performing fine-tuning training to obtain a fine-tuned large model provided in an embodiment of the present invention.

[0108] Furthermore, in the above-mentioned multimodal intelligent question-answering reasoning method based on knowledge distillation, the multimodal fusion vector is used to fine-tune the base large model to obtain a fine-tuned large model, and combined with Figure 3 , including six steps from S31 to S37:

[0109] S31: randomly mix the text, voice, video, and image data in a ratio of 1:1:1:1, and divide each part into a training set, a test set, and a validation set according to a preset ratio;

[0110] S32: concatenating the intention recognition prompt word and the multimodal fusion vector to obtain training data;

[0111] S33: input the training data into the base large model, and add multiple fully connected layers to the base large model;

[0112] S34: Fine-tune the base large model through the training set and the validation set;

[0113] S35: After the loss value is stable, the model parameters of the base large model are saved to obtain the fine-tuned large model;

[0114] S37: Test the accuracy of the fine-tuned large model using the test set.

[0115] Specifically, in an embodiment of the present invention, the pre-set ratio can be 7:2:1. Multiple fully connected layers are added to the base large model to adapt to specific intent recognition tasks. The method of fine-tuning the base large model through training sets and validation sets is described in detail below. The accuracy of the fine-tuned large model is tested through the test set to evaluate the effect of the fine-tuned large model.

[0116] Figure 4 A diagram of a method for supplementing intent to generate final intent information provided in an embodiment of the present invention.

[0117] Furthermore, in the above-mentioned multimodal intelligent question-answering reasoning method based on knowledge distillation, if the judgment result is that the classified intent slot is incomplete, the intent is supplemented to generate the final intent information, and combined with Figure 4, including four steps from S41 to S44:

[0118] S41: Design corresponding slots for different intents, and automatically generate slots through dialogue with users during the question-answering process.

[0119] S42: determining whether some slot information is missing. If the determination result is that some slot information is missing, the missing slot information is input into the prompt generator through the prompt word generation module to generate corresponding prompt words for assisting the fine-tuned large model to extract slot information.

[0120] S43: Supplement the intent with the extracted slot information to obtain an intent output result;

[0121] S44: Input the intent output result into the business dialogue management system, select the corresponding business, and query the missing slot information through the intent slot extraction unit of the language model to generate the final intent information.

[0122] Specifically, in the embodiment of the present invention, supplementing the missing slot information is a supplement to the fine-tuned large model. Some intents contain specific slots, for example, the intent of air ticket booking needs to provide slots such as destination and departure time. By adjusting the general large language model to meet the requirements of the slot extraction task of the business dialogue management system, the accuracy and stability of slot information extraction are improved.

[0123] Furthermore, in the above-mentioned multimodal intelligent question-answering reasoning method based on knowledge distillation, knowledge distillation is performed on the fine-tuned large model to obtain a pre-trained intent recognition model, including:

[0124] The teacher model is the fine-tuned base model, and the student model is the BERT model.

[0125] Through the comparative learning method, the BERT model uses the method of first reducing the dimension and then increasing the dimension to learn the knowledge and prediction results of each layer in the teacher model;

[0126] Among them, the distillation loss function includes: label loss and feature matching loss.

[0127] Specifically, in the embodiment of the present invention, the student model uses two matrices W and V. The matrix W is responsible for dimensionality reduction, and the matrix V is responsible for dimensionality increase. By using the method of first reducing the dimensionality and then increasing the dimensionality, irrelevant information is eliminated and the amount of feature matrix calculation is reduced.

[0128] The dimension reduction calculation formula is as follows:

[0129]

[0130] The calculation formula for dimension increase is as follows:

[0131]

[0132] Among them, b and c both represent biases, g and h represent the activation functions of dimensionality reduction and dimensionality increase losses, λ and γ represent regularization penalty terms, Ω and Θ represent regularization penalty functions, and x i represents the i-th training sample after dimensionality reduction and dimensionality increase, x′ i Represents the i-th training sample before dimensionality reduction and dimensionality increase.

[0133] The distillation loss function formula is as follows:

[0134]

[0135] Among them, Loss represents the total distillation loss, L lable represents the label loss, L feature represents feature matching loss, α represents weight coefficient, p teacher represents the output probability distribution of the teacher model, p student represents the output rate distribution of the student model, f teacher represents the intermediate layer features of the teacher model, σ represents the activation function, such as RELU, Sigmoid or Tanh, represents the weight matrix of the Lth layer of the teacher model, represents the output features of the Lth layer of the teacher model, represents the bias vector of the Lth layer of the teacher model, f student represents the intermediate layer features of the student model, σ is the activation function, represents the weight matrix of the Lth layer of the student model, represents the output features of the Lth layer of the student model, Represents the bias vector of the Lth layer of the student model, i represents the sequence number of the training sample, and N represents the maximum sequence number of the training sample.

[0136] Figure 5 A diagram of a method for fine-tuning a large base model provided in an embodiment of the present invention.

[0137] Furthermore, in the above-mentioned multimodal intelligent question-answering reasoning method based on knowledge distillation, the base large model is fine-tuned through the training set and the verification set, and combined with Figure 5 , including four steps from S51 to S54:

[0138] S51: extract features from the training set and the validation set, and input the obtained vectors into the base large model;

[0139] S52: Feature combination and classification through fully connected layers;

[0140] S53: Use the cross entropy loss function to calculate the difference between the output of the base model and the true intent label;

[0141] S54: Update the model parameters of the base large model through the back propagation algorithm.

[0142] Furthermore, in the above-mentioned multimodal intelligent question-answering reasoning method based on knowledge distillation, label loss is used to narrow the gap between the probability distribution of the student model output and the probability distribution of the teacher model output, and feature matching loss is used to narrow the gap between the intermediate layer features of the student model and the intermediate layer features of the teacher model.

[0143] Device Embodiment

[0144] Figure 6 A diagram of a multimodal intelligent question-answering reasoning device based on knowledge distillation provided in an embodiment of the present invention.

[0145] In the second aspect of the present invention, a multimodal intelligent question-answering reasoning device based on knowledge distillation is also proposed, combined with Figure 6 ,include:

[0146] Acquisition module 61: used to obtain user data.

[0147] Specifically, in the embodiment of the present invention, the user data acquired by the acquisition module 61 may be customer consultation data in an intelligent customer service scenario, such as text, voice, image and video data; or it may be family life data in an intelligent furniture control scenario, such as environmental data such as temperature, humidity, air quality and behavioral habits of family members; the user data may be data in any scenario, and the protection scope of the present invention is not limited thereto.

[0148] S2: Extract multimodal features of user data.

[0149] Extraction module 62: used to extract multimodal features of the user data.

[0150] Specifically, in an embodiment of the present invention, the extraction module 62 extracts multimodal features of user data by extracting multimodal features from customer consultation data, such as keywords in text, objects and locations in images, emotions in audio, etc., or by extracting multimodal features from family life data, such as temperature and humidity in environmental data, as well as the behavioral habits of family members.

[0151] Reasoning module 63 and generation module 64: used for the pre-trained intention recognition model to reason about the multimodal features, generate reply content or perform corresponding operations.

[0152] Specifically, in an embodiment of the present invention, the reasoning module 63 in the pre-trained intent recognition model performs question-answering reasoning on the keywords of the extracted text, the objects and positions in the image, the emotions in the audio, etc., and the generation module 64 generates a reply or performs a corresponding operation, which can provide more accurate and efficient customer service, reduce the cost of manual customer service, and improve customer satisfaction; the reasoning module 63 in the pre-trained intent recognition model reasons on the temperature and humidity in the extracted environmental data, and the generation module 64 generates a reply content or performs a corresponding operation, such as adjusting the temperature, turning on the lights, etc., which can achieve a more intelligent and convenient family life experience, improve the quality of life, and reduce energy consumption.

[0153] The pre-trained intent recognition model is trained using a training device, and the training device includes:

[0154] The first acquisition module is used to acquire multimodal data;

[0155] Preprocessing module: used to preprocess multimodal data to obtain multimodal fusion vectors;

[0156] Fine-tuning training module: used to use multimodal fusion vectors to fine-tune the base large model to obtain a fine-tuned large model;

[0157] Slot supplement module: used to input multimodal data into the fine-tuned large model for intent classification, and determine whether the intent slots after classification are complete. If the judgment result is that the intent slots after classification are incomplete, the intent is supplemented to generate the final intent information;

[0158] Knowledge distillation module: used to perform knowledge distillation on the fine-tuned large model to obtain a pre-trained intent recognition model.

[0159] According to a third aspect of the present invention, an electronic device is provided, comprising: a processor and a memory;

[0160] The processor calls the program or instruction stored in the memory to execute any of the above multimodal intelligent question-answering reasoning methods based on knowledge distillation.

[0161] In a fourth aspect of the present invention, a computer-readable storage medium is proposed, which stores a program or instruction, and the program or instruction enables a computer to execute any one of the above multimodal intelligent question-answering reasoning methods based on knowledge distillation.

[0162] Figure 7 It is a schematic block diagram of an electronic device provided by an embodiment of the present invention.

[0163] like Figure 7As shown, the electronic device includes: at least one processor 701, at least one memory 702 and at least one communication interface 703. The various components in the electronic device are coupled together through a bus system 704. The communication interface 703 is used to transmit information between external devices. It can be understood that the bus system 704 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 704 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 7 Various buses are labeled as bus system 704 .

[0164] It can be understood that the memory 702 in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories.

[0165] In some implementations, the memory 702 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system and application programs.

[0166] Among them, the operating system includes various system programs, such as the framework layer, the core library layer, the driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application program includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. A program that implements any method in a multimodal intelligent question-answering reasoning method based on knowledge distillation provided in an embodiment of the present invention can be included in the application program.

[0167] In an embodiment of the present invention, the processor 701 calls a program or instruction stored in the memory 702, specifically, a program or instruction stored in an application, and the processor 701 is used to execute the steps of each embodiment of a multimodal intelligent question-answering reasoning method based on knowledge distillation provided in an embodiment of the present invention.

[0168] Get user data;

[0169] Extracting multimodal features of the user data;

[0170] The pre-trained intent recognition model infers the multimodal features and generates reply content or performs corresponding operations;

[0171] Among them, the training method of the pre-trained intent recognition model includes:

[0172] Acquire multimodal data;

[0173] Preprocess the multimodal data to obtain a multimodal fusion vector;

[0174] Use the multimodal fusion vector to fine-tune the base large model to obtain a fine-tuned large model;

[0175] Input the multimodal data into the fine-tuned large model for intent classification, and determine whether the classified intent slots are complete. If the judgment result is that the classified intent slots are incomplete, supplement the intent to generate the final intent information.

[0176] The fine-tuned large model is subjected to knowledge distillation to obtain a pre-trained intent recognition model.

[0177] Any of the methods in the multimodal intelligent question-answering reasoning method based on knowledge distillation provided in an embodiment of the present invention can be applied to the processor 701, or implemented by the processor 701. The processor 701 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 701 or instructions in the form of software. The above-mentioned processor 701 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0178] The steps of any method in a multimodal intelligent question-answering reasoning method based on knowledge distillation provided by an embodiment of the present invention can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software units in a decoding processor. The software unit can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 702, and the processor 701 reads the information in the memory 702 and completes the steps of the method in combination with its hardware.

[0179] Those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present invention and to form different embodiments.

[0180] Those skilled in the art will appreciate that the description of each embodiment has its own emphasis, and for parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0181] Although the embodiments of the present invention are described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope defined by the appended claims. The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed by the present invention, and these modifications or substitutions should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

[0182] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A multimodal intelligent question-answering reasoning method based on knowledge distillation, characterized in that: include: Get user data; Extracting multimodal features of the user data; The pre-trained intent recognition model infers the multimodal features and generates reply content or performs corresponding operations; Among them, the training method of the pre-trained intent recognition model includes: Acquire multimodal data; Preprocessing the multimodal data to obtain a multimodal fusion vector; Use the multimodal fusion vector to fine-tune the base large model to obtain a fine-tuned large model; Input the multimodal data into the fine-tuned large model for intent classification, and determine whether the classified intent slots are complete. If the judgment result is that the classified intent slots are incomplete, supplement the intent to generate the final intent information. The fine-tuned large model is subjected to knowledge distillation to obtain a pre-trained intent recognition model.

2. According to claim 1, a multimodal intelligent question-answering reasoning method based on knowledge distillation is characterized in that: The preprocessing of the multimodal data to obtain a multimodal fusion vector includes: Extract features from text data in multimodal data to obtain text vectors; Extract features from audio data in multimodal data to obtain audio vectors; Extract features from video data in multimodal data to obtain video vectors; Extract features from image data in multimodal data to obtain image vectors; The text vector, audio vector, video vector and image vector are fused to obtain a multimodal fusion vector.

3. According to claim 1, a multimodal intelligent question-answering reasoning method based on knowledge distillation is characterized in that: The base large model is fine-tuned using the multimodal fusion vector to obtain a fine-tuned large model, including: The text, voice, video, and image data are randomly mixed in a ratio of 1:1:1:1, and each part is divided into a training set, a test set, and a validation set according to a preset ratio; The intention recognition prompt words and the multimodal fusion vector are concatenated to obtain training data; Input the training data into the base model and add multiple fully connected layers to the base model; Fine-tune the large base model using the training set and validation set; After the loss value is stable, the model parameters of the base large model are saved to obtain the fine-tuned large model; The accuracy of the fine-tuned large model is tested on the test set.

4. According to claim 1, a multimodal intelligent question-answering reasoning method based on knowledge distillation is characterized in that: If the judgment result is that the classified intent slot is incomplete, the intent is supplemented to generate the final intent information, including: Design corresponding slots for different intents, and automatically generate slots through dialogue with users during the question-answering process; Determine whether some slot information is missing. If the judgment result is that some slot information is missing, the missing slot information is input into the prompt generator through the prompt word generation module to generate corresponding prompt words to assist the fine-tuned large model in extracting slot information; The intent is supplemented by the extracted slot information to obtain the intent output result; The intent output results are input into the business dialogue management system, the corresponding business is selected, and the missing slot information is queried through the intent slot extraction unit of the language model to generate the final intent information.

5. According to claim 1, a multimodal intelligent question-answering reasoning method based on knowledge distillation is characterized in that: The knowledge distillation of the fine-tuned large model to obtain a pre-trained intent recognition model includes: The teacher model is the fine-tuned base model, and the student model is the BERT model. Through the comparative learning method, the BERT model uses the method of first reducing the dimension and then increasing the dimension to learn the knowledge and prediction results of each layer in the teacher model; Among them, the distillation loss function includes: label loss and feature matching loss.

6. According to claim 3, a multimodal intelligent question-answering reasoning method based on knowledge distillation is characterized in that: The fine-tuning training of the base large model through the training set and the validation set includes: Perform feature extraction on the training set and the validation set, and input the obtained vectors into the base large model; Feature combination and classification through fully connected layers; Use the cross entropy loss function to calculate the difference between the output of the base model and the true intent label; The model parameters of the base large model are updated through the back propagation algorithm.

7. The multimodal intelligent question-answering reasoning method based on knowledge distillation according to claim 5, characterized in that: The label loss is used to narrow the gap between the probability distribution of the student model output and the teacher model output, and the feature matching loss is used to narrow the gap between the intermediate layer features of the student model and the intermediate layer features of the teacher model.

8. A multimodal intelligent question-answering reasoning device based on knowledge distillation, characterized in that: include: Acquisition module: used to obtain user data; Extraction module: used to extract multimodal features of the user data; Reasoning module and generation module: used for the pre-trained intent recognition model to reason about the multimodal features, generate reply content or perform corresponding operations; The pre-trained intent recognition model is trained using a training device, and the training device includes: The first acquisition module is used to acquire multimodal data; Preprocessing module: used for preprocessing the multimodal data to obtain a multimodal fusion vector; Fine-tuning training module: used to use multimodal fusion vectors to fine-tune the base large model to obtain a fine-tuned large model; Slot supplement module: used to input multimodal data into the fine-tuned large model for intent classification, and determine whether the intent slots after classification are complete. If the judgment result is that the intent slots after classification are incomplete, the intent is supplemented to generate the final intent information; Knowledge distillation module: used to perform knowledge distillation on the fine-tuned large model to obtain a pre-trained intent recognition model.

9. An electronic device, characterized in that: include: Processor and memory; The processor is used to execute a multimodal intelligent question-answering reasoning method based on knowledge distillation as described in any one of claims 1 to 7 by calling the program or instruction stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores programs or instructions, and the programs or instructions enable a computer to execute a multimodal intelligent question-answering reasoning method based on knowledge distillation as described in any one of claims 1 to 7.

Images