Multimedia question and answer model quantification method for smart home side end deployment

Abstract

The invention relates to a multimedia question and answer model quantification method for smart home side end deployment, and belongs to the technical field of smart home. The method comprises the following steps: step 1, multimedia information feature extraction: extracting visual features of an input video through video coding to obtain visual representation, and extracting text features of an input question through question coding to obtain question representation; step 2, multimedia information feature fusion: constructing a BART-base codec model, then inputting the visual representation and the problem representation into the BART-base codec model, sequentially passing through an encoder and a decoder, and finally performing mean pooling on the output of the decoder to obtain global representation; and step 3, question decoding: sending the global representation into an answer decoder, and returning the answer with the highest score as a result. According to the method, the simplicity and feasibility of smart home edge deployment are improved, and the performance of a multimedia question and answer model can be effectively improved by utilizing the pre-training model through the time reasoning adapter provided by the invention.

Description

TECHNICAL FIELD

[0001] The application relates to a multimedia question answering model quantization method, in particular to a multimedia question answering model quantization method for intelligent home edge deployment, and belongs to the technical field of intelligent home. BACKGROUND

[0002] An intelligent home system integrates various intelligent devices and services to provide a more comfortable, convenient and energy-saving living environment. With the development of artificial intelligence technology, especially the multimedia question answering system, the intelligent home system can now interact with users more intelligently, such as voice control of home appliances, provision of video-based security monitoring, health monitoring, etc. However, these advanced functions usually require powerful data processing capabilities, which have traditionally relied on cloud computing platforms for processing.

[0003] With the rise of edge computing, migrating data processing tasks from the cloud to devices on the network edge, i.e. edge deployment, has become a trend. Edge computing can significantly reduce latency, improve data processing speed, and reduce dependence on cloud resources, further protecting user data privacy. Multimedia question answering models deployed in the intelligent home environment are usually resource-intensive, requiring a large amount of computing power and memory, which is a major limitation for edge devices. Therefore, these models need to be optimized to adapt to the resource constraints of the edge computing environment.

[0004] Model quantization is an effective model optimization technique that reduces the computational resource requirements of a model, reducing the model size and improving the running efficiency without significantly sacrificing accuracy. In the edge deployment scenario of intelligent home, quantization can enable multimedia question answering models to run on resource-constrained devices, achieving fast response and efficient data processing. SUMMARY

[0005] The application aims to provide a multimedia question answering model quantization method for intelligent home edge deployment to solve the problem of slow response speed and low data processing efficiency of multimedia question answering models on resource-constrained devices in the edge deployment scenario of intelligent home.

[0006] To achieve the above-mentioned purpose, the following technical solutions are implemented: A multimedia question answering model quantization method for intelligent home edge deployment, comprising the following steps: Step 1. Multimedia information feature extraction: multimedia information includes video and question, visual features of input video are extracted through video encoding to obtain visual representation, and text features of input question are extracted through question encoding to obtain question representation; Step 2. Multimedia information feature fusion: a BART-base encoder-decoder model is constructed, which includes an encoder and a decoder, then the visual representation and the question representation are input into the BART-base encoder-decoder model to sequentially pass through the encoder and the decoder, and finally the decoder output is mean-pooled to obtain a global representation; Step 3. Question decoding: the global representation is input into the answer decoder to perform answer prediction, and the answer with the highest score is returned as the result.

[0007] Further, the step 1 specifically comprises: Video encoding: a CLIP framework and a visual projection layer are constructed, and the video frames are input into the CLIP model to extract visual features, and the visual features pass through a visual projection layer to obtain a visual representation with a d-dimensional feature space , wherein K represents the total number of frames processed, and d is the dimension of the feature space; Question encoding: a BERT model and a bidirectional gated recurrent unit Bi-GRU are constructed, the initial word token representation is obtained from the input question by using the fine-tuned BERT model, and then the initial word token representation is processed by using the bidirectional gated recurrent unit Bi-GRU to output a series of context-aware question representations , wherein represents the number of words in the question, is the dimension of the hidden layer in the Bi-GRU.

[0008] Further, the step 2 specifically comprises: The encoder of the BART-base encoder-decoder model includes a multi-head self-attention layer, a TR-adapter, a layer normalization layer, a full connection layer, a TR-adapter, a layer normalization layer, the decoder of the BART-base encoder-decoder model includes a multi-head self-attention layer, a TR-adapter, a layer normalization layer, a full connection layer, a TR-adapter, a layer normalization layer, a full connection layer, a TR-adapter, and a layer normalization layer; A time reasoning operator is integrated into the TR-adapter, which alternates between a depth separable convolution DSConv and a self-attention Att operation, and repeats the operation T times within the TR-adapter structure, and the operation formula of the TR-adapter is as follows: , In the formula, is the initial input representation of the adapter, , are respectively a down-sampling weight matrix and an up-sampling weight matrix, wherein is the input dimension, is the hidden dimension, DWConv represents a deep convolution operation, and PWConv represents a point-wise convolution operation, represents the input variable the result of the i-th execution of the deep convolution, the point-wise convolution and the self-attention mechanism, and Att is a standard self-attention mechanism; represents the input representation After down-sampling, a non-linear transformation operation is performed, and here is a non-linear change ReLu function.

[0009] The visual representation and the question representation After being concatenated, the global representation is obtained by inputting the BART-base encoder-decoder model.

[0010] Further, the step 3 specifically comprises: The answer decoder obtains a predefined answer set according to the global representation , and then predicts the correct answer according to the predefined answer set, and the scoring function of the answer decoder is: , wherein, is the score of each potential answer in the set, represents a first weight matrix, represents a second weight matrix, represents a first bias, represents a second bias.

[0011] Further, all layer normalization layer parameters in the BART-base encoder-decoder model are fine-tuned, and a general optimization strategy is adopted to optimize the model layer normalization layer parameters using smart home scene question and answer data, which is in the form of a triple (video, question, answer).

[0012] Further, the answer decoder is a multi-layer perception network.

[0013] The present application has the following advantages: 1. The parameter efficient transfer learning technology is first extended to the multimedia question and answer field, which promotes the simplicity and feasibility of pre-training large models in the edge deployment of smart home products; 2. A novel time reasoning adapter TR-adapter is developed, which uses a pre-training model to improve the performance of multimedia question and answer, and at the same time improves the inference efficiency and scalability.

[0014] 3. Compared with the existing advanced parameter efficient transfer learning technology, the optimal question and answer performance is obtained under the condition of updating the least model parameters. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, illustrate the present application, and explain the technical solutions of the present application, and do not constitute a limitation on the present application.

[0016] Figure 1 A flow chart of the multimedia question and answer model of the present application; Figure 2 A schematic diagram of the TR-Adaptor structure of the present application. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0018] Embodiment 1 In this embodiment, as shown in the figure, a multimedia question and answer model quantification method for smart home edge deployment is provided, including the following steps: Figure 1 Step 1. Multimedia information feature extraction: the multimedia information includes video and question, the visual features of the input video are extracted through video encoding to obtain visual representation, and the text features of the input question are extracted through question encoding to obtain question representation; Specifically, the video encoding process: by constructing a CLIP framework and a visual projection layer, the video frames are input into the CLIP model to extract visual features, and the visual features pass through a visual projection layer to obtain a visual representation with a d-dimensional feature space , where K represents the total number of processed frames, and d is the dimension of the feature space; the question encoding process: by constructing an open source fine-tuned BERT model and a bidirectional gated recurrent unit Bi-GRU, the fine-tuned BERT model is used to obtain an initial word token representation from the input question, and then the bidirectional gated recurrent unit Bi-GRU is used to process the obtained initial word token representation to output a series of context-aware question representations , where represents the number of words in the question, is the dimension of the hidden layer in the Bi-GRU.

[0019] ​Step 2. Multimedia information feature fusion: a BART-base encoder-decoder model is constructed, which includes an encoder and a decoder, then the visual representation and the question representation are input into the BART-base encoder-decoder model in turn through the encoder and the decoder, and finally the decoder output is mean-pooled to obtain a global representation; Specifically, the encoder of the BART-base encoder-decoder model includes a multi-head self-attention layer, a TR-adapter, a layer normalization layer, a full connection layer, a TR-adapter, a layer normalization layer, and the decoder of the BART-base encoder-decoder model includes a multi-head self-attention layer, a TR-adapter, a layer normalization layer, a full connection layer, a TR-adapter, a layer normalization layer, a full connection layer, a TR-adapter, and a layer normalization layer. Specifically, as shown in Figure 2 The TR-adapter integrates a time reasoning operator, which alternates between a depth separable convolution DSConv and a self-attention Att operation, and repeats the operation T times within the TR-adapter structure, and the operation formula of the TR-adapter is as follows: , In the formula, is the initial input representation of the adapter, , are a down-sampling weight matrix and an up-sampling weight matrix, respectively, where is the input dimension, is the hidden dimension, DWConv represents a depth convolution operation, and PWConv represents a point-wise convolution operation, represents the i-th execution result of the depth convolution, the point-wise convolution, and the self-attention mechanism performed on the input variable Att is a standard self-attention mechanism; represents a nonlinear transformation operation performed on the input representation after down-sampling, where is a nonlinear change ReLu function.

[0020] Specifically, the visual representation and the question representation are input into the BART-base encoder-decoder model after being concatenated to obtain a global representation .

[0021] Step 3. Question decoding: the global representation is input into an answer decoder to perform answer prediction, and the answer with the highest score is returned as the result.

[0022] Specifically, the answer decoder performs the following operations according to the global representation get a predefined answer set, and then predict the correct answer according to the predefined answer set, the score function of the answer decoder is: , wherein, is the score of each potential answer in the set, represents a first weight matrix, represents a second weight matrix, represents a first bias, represents a second bias.

[0023] Specifically, all layer normalization layer parameters in the BART-base codec model are fine-tuned, a general optimization strategy is adopted, and the model layer normalization layer parameters are optimized using smart home scene question and answer data in the form of triplets (video, question, answer).

[0024] Embodiment 2 In this embodiment, we compare the method of the present application with the existing advanced technology in terms of question and answer accuracy and the amount of parameters to be updated on the existing public video question and answer data sets TGIF, MSRVTT, MSVD, Next-QA.

[0025] Table 1 Comparison of the present application method and the existing advanced technology In this embodiment, Table 1 is a comparison of the present application method and the existing advanced technology in terms of question and answer accuracy and the amount of parameters to be updated on the existing public video question and answer data sets TGIF, MSRVTT, MSVD, Next-QA. Action represents the accuracy of the action type question in the TGIF data set, Trans. represents the accuracy of the action transfer type question in the TGIF data set, Frame represents the accuracy of the static visual type question in the TGIF data set, and Updated Params represents the proportion of updated parameters. It can be seen that the present application method achieves optimal performance with only 2.26% of the parameters updated, which illustrates the effectiveness of the present application method.

[0026] Finally, it should be noted that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent replacements to some technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A multimedia question-answering model quantization method for smart home edge deployment, characterized by: The following steps are involved: Step 1. Multimedia information feature extraction: Multimedia information includes video and questions. The visual features of the input video are extracted through video encoding to obtain a visual representation. The text features of the input question are extracted through question encoding to obtain a question representation. Step 2. Multimedia information feature fusion: Construct a BART-base codec model, which includes an encoder and a decoder. Then, input the visual representation and question representation into the BART-base codec model, pass through the encoder and decoder in sequence, and finally perform mean pooling on the decoder output to obtain a global representation. Step 3. Question decoding: The global representation is fed into the answer decoder to predict the answer, and the answer with the highest score is returned as the result.

2. The multimedia question-answering model quantification method for smart home edge deployment according to claim 1 is characterized in that: The step 1 specifically includes: Video encoding: Build CLIP framework and visual projection layer to convert video frames Input into the CLIP model to extract visual features, and the visual features are passed through a visual projection layer to obtain a visual representation with a d-dimensional feature space. , where K represents the total number of frames processed and d is the dimension of the feature space; Question encoding: Use the fine-tuned BERT model to obtain the initial word token representation from the input question, then use the bidirectional gated recurrent unit Bi-GRU to process the obtained initial word token representation and output a series of context-aware question representations ,in Indicates the number of words in the question, is the dimension of the hidden layer in Bi-GRU.

3. The multimedia question-answering model quantification method for smart home edge deployment according to claim 2 is characterized in that: The step 2 specifically includes: The encoder of the BART-base codec model includes a multi-head self-attention layer, a TR-adapter, a layer normalization layer, a fully connected layer, a TR-adapter, and a layer normalization layer; the decoder of the BART-base codec model includes a multi-head self-attention layer, a TR-adapter, a layer normalization layer, a fully connected layer, a TR-adapter, a layer normalization layer, a fully connected layer, a TR-adapter, and a layer normalization layer; The TR-adapter integrates a temporal reasoning operator that alternates between depthwise separable convolution (DSConv) and self-attention (Att) operations and repeats the operation T times within the TR-adapter structure. The operation formula of the TR-adapter is as follows: ， In the formula, is the initial input representation of the adapter, 、 are the downsampling weight matrix and the upsampling weight matrix respectively, where is the input dimension, To hide the dimension, DWConv represents the depth convolution operation, PWConv represents the point-by-point convolution operation, Indicates that for input variables The result of executing the i-th execution of depthwise convolution, pointwise convolution and self-attention mechanism, Att is the standard self-attention mechanism; Represents the input After downsampling, a nonlinear transformation operation is performed. Here is the nonlinear change ReLu function; The visual representation and problem representation After concatenation, it is input into the BART-base codec model to obtain a global representation .

4. The multimedia question-answering model quantification method for smart home edge deployment according to claim 3 is characterized in that: The step 3 specifically includes: The answer decoder is based on the global representation A predefined answer set is obtained, and then the correct answer is predicted based on the predefined answer set. The scoring function of the answer decoder is: , in, is the score of each potential answer in the set, represents the first weight matrix, represents the second weight matrix, represents the first bias, Indicates the second bias.

5. The multimedia question-answering model quantification method for smart home edge deployment according to claim 3 is characterized in that: Fine-tune all layer normalization parameters within the BART-base codec model. Adopting a common optimization strategy, optimize the model layer normalization parameters using smart home scenario question-and-answer data, which is in the form of triples: video, question, and answer.

6. The multimedia question-answering model quantification method for smart home edge deployment according to claim 4 is characterized in that: The answer decoder is a multi-layer perceptron network.

Images