Multi-granularity reasoning optimization method and system for multi-modal flow data

Abstract

The invention provides a multi-granularity reasoning optimization method and system for multi-modal flow data, and belongs to the crossing field of artificial intelligence and Internet of Things technologies, multi-modal data is input into a lightweight feature extraction network to obtain lightweight features of each kind of modal data, and the multi-modal data is input into a feature extraction network of an original model to obtain original features; inputting the lightweight features into a modal cache to obtain cache features, enabling the cache features to pass through a feature mapping network to obtain mapping features, inputting the lightweight features into a strategy network to obtain redundancy judgment vectors, and taking the redundancy judgment vectors as weights to superpose the original features and the mapping features to obtain superposition features; respectively inputting the mapping features, the superposition features and the original features into a fusion network and a task network of an original model, training the original model, and when a loss function is minimum, stopping training to obtain a trained model; and inputting the multi-modal data into the trained model for reasoning, thereby solving the problem that modal missing, partial modal redundancy and continuous reasoning tasks cannot be processed during redundancy processing of the multi-modal data.

Description

Technical Field

[0001] The present invention relates to the technical field of intersection of artificial intelligence and Internet of Things technologies, and in particular to a multi-granularity reasoning optimization method and system for multi-modal streaming data. Background Art

[0002] The Digital China Development Report (2022) released by the Cyberspace Administration of China pointed out that the number of mobile Internet of Things terminal users in my country has reached 1.845 billion. The huge Internet of Things faces a complex application environment, generating massive heterogeneous and spatiotemporally related multimodal streaming data, which puts higher demands on multimodal streaming data reasoning. However, the computing platform of the Internet of Things environment is often small and simple, and it is difficult to carry computing-intensive reasoning tasks. In response to the contradiction between the high computing power requirements of multimodal models and the tight computing power resources, there are currently two solutions, namely, optimizing the model itself by accelerating compression and filtering input data to eliminate unnecessary calculations.

[0003] In terms of input filtering, most of the work is designed for a single modality, such as FilterForward for images, and FoggyCache proposed an input filtering method for images and audio. At present, there is a lack of input filtering technology for multimodal. The paper "InFi: End-to-End Learning to Filter Input for Resource-Efficiency in Mobile-Centric Inference" (Mu Yuan et al., 2022) proposed an input filtering framework Infi (INputFIlter), but it is limited to processing multimodal redundancy in ideal conditions (each modal data is complete and there is redundancy after modal splicing). In actual situations, three defects are exposed: 1. It cannot handle modality missing; 2. It cannot handle partial modal redundancy. Existing work can only judge whether the sample is redundant as a whole, but cannot take corresponding optimization measures; 3. Regression task performance limitations: For example, Infi and FoggyCache, their caching mechanism needs to be converted into discrete tasks for continuous reasoning tasks (such as regression problems), and the reasoning effect will be greatly reduced. Summary of the invention

[0004] The technical problem to be solved by the present invention is how to solve the problems of being unable to handle modality missing, unable to handle partial modality redundancy, and unable to handle continuous reasoning tasks when processing multimodal data redundancy.

[0005] The present invention solves the above technical problems through the following technical solutions: a multi-granularity reasoning optimization method for multimodal streaming data, comprising the following steps:

[0006] Step 1: Input the multimodal data into a lightweight feature extraction network to obtain lightweight features of each modality data, and input the multimodal data into the feature extraction network of the original model to obtain original features;

[0007] Step 2: Input the lightweight features into the modal cache to obtain the cache features. The cache features are passed through the feature mapping network to obtain the mapping features. The lightweight features are input into the strategy network to obtain the redundant judgment vector. The redundant judgment vector is used as the weight to superimpose the original features and the mapping features to obtain the superimposed features.

[0008] Step 3: Input the mapping features, superimposed features, and original features into the fusion network and task network of the original model respectively, train the original model, and stop training when the loss function is the smallest to obtain a trained model;

[0009] Step 4: Input the multimodal data into the trained model for inference.

[0010] The multi-granularity reasoning optimization method for multimodal streaming data of the present invention integrates the original model with a lightweight feature extraction network, a modality cache, a sample cache, and a feature mapping network, and jointly trains with a strategy network to find the overall optimal solution. It is an end-to-end training reasoning method that can improve the robustness of the original model and simplify the training steps. The end-to-end training reasoning method of the present invention has the ability to handle multi-granularity redundancy. At the modality granularity, a modality cache is designed separately for each modality. For a certain modality, when the modality is redundant (cache hit), the cache feature is obtained, and the dimension is aligned with the feature extracted by the original model (optimized model) to obtain the feature of the modality, without using the original model to extract the feature of the modality. At the sample granularity, when all modalities hit the cache, the result of the pre-inference of the cache can be directly indexed without going through the original model reasoning again. It can effectively handle partial modality redundancy and overall sample redundancy, and further improve the reasoning efficiency.

[0011] The present invention processes redundancy at the modal granularity. The missing of some modalities does not affect the redundant judgment of other modalities. For the missing modality, it is regarded as an all-zero vector when inputting the frame. The modality cache uses the all-zero vector as a special redundancy form and outputs the cache feature with the closest distance, which is suitable for modality missing.

[0012] The present invention realizes a general multimodal reasoning optimization framework by abstracting the multimodal task into three independent parts. It can also make the optimization process independent of the specific reasoning task and adapt to various reasoning tasks (discrete and continuous reasoning tasks) without being limited to discrete tasks, and has stronger versatility.

[0013] Preferably, the policy network in step 2 includes a linear layer, a ReLU activation function, a linear layer, and an optimized argmax. The input dimension is the dimension of the lightweight feature of the corresponding modal data, and the hidden layer dimension is flexibly adjusted according to the specific scenario. The optimized argmax uses softmax instead of argmax for calculation during back propagation, outputs the value of argmax during forward propagation, and outputs the gradient of softmax during back propagation. The expression is:

[0014] softout-detach(softout-hardout)

[0015] Among them, softout is the output of softmax, hardout is the output of argmax, and detach means gradient truncation, so that the gradient will not propagate to hardout, so that the forward propagation output result is hardout, and the back propagation output result is the gradient of softmax.

[0016] In the policy network, the output result is a binary value in the form of onehot, and the semantic meaning is whether the input feature is redundant. In the modal cache, the cache size is n, and the output hit decision is an n-dimensional onehot vector. The onehot vectors output by these two modules are obtained by argmax, which is not differentiable. By using softmax instead of argmax for calculation during back propagation, the problem of argmax not being differentiable during end-to-end training can be well solved.

[0017] Preferably, the loss function in step 3 is calculated as follows:

[0018] loss total =αloss cache +βloss origin +γloss add +δloss comp

[0019] Among them, loss cache Represents the loss value of the inference result of the cached feature, loss origin Represents the loss value of the inference result of the original feature, loss add Represents the loss value of the inference result of the superimposed feature, loss comp represents the computational loss, and α, β, γ, and δ are weight coefficients.

[0020] Preferably, the process of training the original model in step 3 includes a pre-training stage and a strategy adjustment stage. In the pre-training stage, the weight coefficient γ=δ=0 is set. In the strategy adjustment stage, α+β=γ=0.5, β≤α≤10β is set. δ is adjusted dynamically, taking a batch as a unit. When the amount of calculation is lower than expected, the δ value is reduced by 10%. When the amount of calculation is higher than expected, the δ value is increased by 10%.

[0021] According to the characteristics of different periods in the training process, the present invention divides the training process into two stages: pre-training and strategy adjustment. In pre-training, the inference results and computational loss of superimposed features are not considered, and only the inference results of cached features and original features are considered. This ensures that each module can converge quickly to an effective level. In the strategy adjustment stage, the convergence speed of the modal cache and the original model is balanced to prevent overfitting or difficulty in convergence. δ is dynamically adjusted according to the computational amount, which can stabilize the computational amount within a reasonable expected range and continuously optimize the inference effect.

[0022] Preferably, step 4 comprises:

[0023] 4.1. Input the multimodal data into the lightweight feature extraction network to obtain lightweight features of each modal data, and obtain cache features through modal caching;

[0024] 4.2. The lightweight feature input strategy network obtains the redundant judgment vector, and uses the redundant judgment vector to determine whether each modality is hit. If all the modalities are hit, the mapping features are directly indexed and the inference results are obtained through the sample cache. If not all the modalities are hit, the hit modal data is input into the feature extraction network of the original model to obtain the original features, and the original features and mapping features are combined and input into the fusion network and task network of the original model to obtain the inference results.

[0025] During the inference process, the cached features obtained from the modality cache can directly index the mapping features in training without going through the feature mapping network. The original features selectively infer some modalities based on the output of the policy network. When the final inference result is obtained, the features are no longer superimposed, and the mapping features and the original features are directly combined to simplify the inference steps.

[0026] The present invention also provides a multi-granularity reasoning optimization system for multi-modal streaming data, comprising:

[0027] A feature extraction module is used to input the multimodal data into a lightweight feature extraction network to obtain lightweight features of each modal data, and to input the multimodal data into the feature extraction network of the original model to obtain original features;

[0028] A redundant filtering module is used to input lightweight features into a modal cache to obtain cache features, the cache features are passed through a feature mapping network to obtain mapping features, the lightweight features are input into a strategy network to obtain redundant judgment vectors, and the redundant judgment vectors are used as weights to superimpose the original features and the mapping features to obtain superimposed features;

[0029] The training module is used to input the mapping features, superimposed features, and original features into the fusion network and task network of the original model respectively, train the original model, and stop the training when the loss function is the smallest to obtain a trained model;

[0030] The inference module is used to input multimodal data into the trained model for inference.

[0031] Preferably, the policy network in the redundant filtering module includes a linear layer, a ReLU activation function, a linear layer, and an optimized argmax. The input dimension is the dimension of the lightweight feature of the corresponding modal data, and the hidden layer dimension is flexibly adjusted according to the specific scenario. The optimized argmax uses softmax instead of argmax for calculation during back propagation, outputs the value of argmax during forward propagation, and outputs the gradient of softmax during back propagation. The expression is:

[0032] softout-detach(softout-hardout)

[0033] Among them, softout is the output of softmax, hardout is the output of argmax, and detach means gradient truncation, so that the gradient will not propagate to hardout, so that the forward propagation output result is hardout, and the back propagation output result is the gradient of softmax.

[0034] Preferably, the loss function in the training module is calculated as follows:

[0035] loss total =αloss cache +βloss origin +γloss add +δloss comp

[0036] Among them, loss cache Represents the loss value of the inference result of the cached feature, loss origin Represents the loss value of the inference result of the original feature, loss add Represents the loss value of the inference result of the superimposed feature, loss comp represents the computational loss, and α, β, γ, and δ are weight coefficients.

[0037] Preferably, the process of training the original model in the training module includes a pre-training stage and a strategy adjustment stage. In the pre-training stage, the weight coefficient γ=δ=0 is set. In the strategy adjustment stage, α+β=γ=0.5 is set, β≤α≤10β, and δ is dynamically adjusted. Taking a batch as a unit, when the amount of calculation is lower than expected, the δ value is reduced by 10%, and when the amount of calculation is higher than expected, the δ value is increased by 10%.

[0038] Preferably, the reasoning module is also used for:

[0039] 4.1. Input the multimodal data into the lightweight feature extraction network to obtain lightweight features of each modal data, and obtain cache features through modal caching;

[0040] 4.2. The lightweight feature input strategy network obtains the redundant judgment vector, and uses the redundant judgment vector to determine whether each modality is hit. If all the modalities are hit, the mapping features are directly indexed and the inference results are obtained through the sample cache. If not all the modalities are hit, the hit modal data is input into the feature extraction network of the original model to obtain the original features, and the original features and mapping features are combined and input into the fusion network and task network of the original model to obtain the inference results. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flowchart of a multi-granularity reasoning optimization method for multi-modal streaming data provided by an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of a multi-granularity reasoning optimization method for multi-modal streaming data provided by an embodiment of the present invention;

[0043] Figure 3 A workflow diagram of modality caching in a multi-granularity reasoning optimization method for multi-modal streaming data provided by an embodiment of the present invention;

[0044] Figure 4 A training framework diagram of a multi-granularity reasoning optimization method for multi-modal streaming data provided by an embodiment of the present invention;

[0045] Figure 5 A reasoning framework diagram of a multi-granularity reasoning optimization method for multi-modal streaming data provided by an embodiment of the present invention;

[0046] Figure 6 A schematic diagram of a multi-granularity reasoning optimization system for multimodal streaming data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in combination with specific embodiments and with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Explanation of relevant technical terms:

[0049] Multimodality refers to the technology strategy of comprehensively using and integrating multiple perception channels or data expression forms (modalities) for analysis, reasoning and interaction in the field of information processing and artificial intelligence. Modalities cover different information carriers, such as text, images, videos, audio and various sensor data, and each modality carries unique and complementary information content.

[0050] Stream data generally refers to a data stream that is continuously and orderly generated, and is a temporally continuous data sequence that is persistent and real-time. The stream data mentioned in the present invention mainly focuses on model reasoning of continuous data.

[0051] Input filtering is a data processing technique that aims to reduce unnecessary and redundant information entering the computational model or analysis process, thereby improving computational efficiency. Common redundant information includes invalid information and repetitive information, such as blank frames in a video, or extremely similar frames that appear consecutively. This technology is suitable for scenarios with limited resources, such as embedded systems or IoT applications.

[0052] End-to-end means that the system starts with the raw data input and automatically goes through the necessary processing, analysis or learning steps until the final output or decision is generated. Its advantage is that it can automatically learn important features, simplify model design, and have better robustness.

[0053] Product Quantization: Product refers to the Cartesian product, which decomposes the original vector space into the Cartesian product of several low-dimensional vector spaces, and quantizes the decomposed low-dimensional vector spaces. In this way, each vector can be represented by a combination of quantized codes in multiple low-dimensional spaces. The collection of quantized code combinations is called codebooks, where the codebook is formed by clustering the low-dimensional vector space using the K-means algorithm, and the code is essentially the cluster centroid of the low-dimensional vector space.

[0054] First, the research idea of the present invention is introduced. Input redundancy is divided into two forms: invalid and duplicate. Considering a data set D, there may be duplicate or highly similar redundant information in it. To measure the similarity between these data and identify redundancy, a similarity function sim: D×D→R is adopted, which measures the similarity degree between any two data points i, j∈D.

[0055] Based on sim(), a subset containing redundant data can be identified from the set D. Where |D′| = n. n represents the number of data points in the subset D′. Inside D′, for any two different elements i, j∈D′, i≠j, the similarity is given by sim(i, j). To reduce the computational burden caused by these redundant data, a clustering method is used to process D′, and only the representative points of each cluster, that is, the cluster centroids, are retained. Let these centroids form a set C = {c1, c2,..., c m}, where |C| = m, m < n, and each c k corresponds to a cluster, and c k is the centroid of this cluster, effectively generalizing the data points similar to it.

[0056] In practical applications, for a new query data point x∈D′, it is first assigned (or mapped) to the centroid in C that is most similar to it. Thus, directly utilize the pre-computed high-level features or inference results, thereby avoiding repeated calculations for x and achieving the purpose of reducing the amount of calculation.

[0057] Suppose a special scenario where D′ only contains data of the invalid type. For any two different elements i, j∈D′, i≠j, sim(i, j) = 0 and |C| = m = 1, that is, only one cluster centroid is retained, and the centroid will have semantic meaning: the invalid class. This method regards "invalid" as a special type (cluster centroid) in duplicate redundancy, unifies the strategies for processing duplicate data and identifying invalid data under a unified framework for input filtering, without the need to separately distinguish different redundancy forms. The present invention aims to filter input redundancy at the modal granularity and sample granularity, unify the two redundancy forms, and adopt an end-to-end training strategy, so that there is no need to separately train the similarity function and clustering. Instead, the clustering center is spontaneously found through training to form a cache, and the classification is decided by the policy network (cache hit). The technical solution of the present invention will be introduced in detail below.

[0058] Example 1

[0059] As Figure 1 and Figure 2 shown, this embodiment provides a multi-granularity inference optimization method for multi-modal flow data, including the following steps:

[0060] Step 1: Input the multimodal data into a lightweight feature extraction network to obtain lightweight features of each modality data, and input the multimodal data into the feature extraction network of the original model to obtain the original features.

[0061] The lightweight feature extraction network is used to quickly extract preliminary features of each modal data. Feature extraction for each modal data works independently. For the text modality, the lightweight feature extraction network uses a word embedding layer to map the sequence to a vector of fixed length through a transformation matrix, and uses a fully connected layer with Sigmoid activation to learn text features. For the image modality, the lightweight feature extraction network can be constructed using a convolutional network and a residual structure.

[0062] The original model is the optimized model. The model should be divided into three independent parts, including feature extraction of the original model (the feature extraction networks of each modality should be independent of each other), modality fusion and task network. The core processing logic of the original model is retained, and the framework is non-invasive to it.

[0063] Step 2: Input the lightweight features into the modal cache to obtain cache features. The cache features are passed through the feature mapping network to obtain mapping features. The lightweight features are input into the strategy network to obtain redundant judgment vectors. The redundant judgment vectors are used as weights to superimpose the original features and the mapping features to obtain superimposed features.

[0064] The modality cache is used to set up a cache scheme for each modality data and store previously valid features. The key value of the modality cache is the cluster centroid of the modality data, and the value is a feature with the same dimension as the original feature extracted by the feature extraction network of the original model. When the policy network decides to use the cached feature, the cached feature is used to replace the original feature.

[0065] like Figure 3 As shown in the figure, the modality cache borrows the idea of ​​"quantization" in product quantization. The sample cache borrows the idea of ​​"product" and takes the Cartesian product of the keys of each modality cache and indexes the inference result. The difference is that the text framework is trained end-to-end, and the feature extraction network and cache are trained simultaneously. Therefore, the cluster centroid of the modality cache (hereinafter referred to as code) and its composed set (cached key set, hereinafter referred to as codebooks) are obtained by gradient descent during the training process.

[0066] The feature mapping network only works during model training. It aligns the dimensions of the cached features with the original features through linear mapping so that the cached features can be used instead of the original features when judging the redundancy of modal data.

[0067] The policy network is used to determine whether the modal data is redundant. The input is the lightweight feature of the modal data, and the output is a binary value [0,1] or [1,0] as a redundant judgment vector. The two-dimensional vector [0,1] indicates that the modal data is redundant. At this time, the policy network decides to use the mapping features of the corresponding modality. The two-dimensional vector [1,0] indicates that the modal data is not redundant. At this time, the policy network decides not to use the mapping features of the corresponding modality, but to use the original features of the corresponding modality.

[0068] Among them, the superposition feature is represented by:

[0069] f add =p1f origin +p2f cache , where f add represents the superposition feature, f cache is the mapping feature, f origin is the original feature, [p1,p2] is the redundant judgment vector, and [p1,p2] is [0,1] or [1,0].

[0070] The policy network includes a linear layer, a ReLU activation function, a linear layer, and an optimized argmax. The input dimension is the dimension of the lightweight features of the corresponding modal data. The hidden layer dimension is flexibly adjusted according to the specific scenario. The optimized argmax uses softmax instead of argmax for calculation during back propagation. The value of argmax is output during forward propagation, and the gradient of softmax is output during back propagation. The expression is:

[0071] softout-detach(softout-hardout)

[0072] Among them, softout is the output of softmax, hardout is the output of argmax, and detach means gradient truncation, so that the gradient will not propagate to hardout, so that the forward propagation output result is hardout, and the back propagation output result is the gradient of softmax.

[0073] It should be noted that the reason for optimizing argmax in the policy network of the present invention is that in the policy network, the output result is a binary value in the form of onehot, and the semantic meaning is whether the input feature is redundant. In the modal cache, the cache size is n, and the output hit decision is an n-dimensional onehot vector. The onehot vectors output by these two modules are obtained by argmax, and argmax is not differentiable, so the present invention uses softmax instead of argmax for calculation during back propagation.

[0074] In order to better fit the model gradient, the present invention adopts temperature learnable softmax (TemperatureScaling Softmax), as follows:

[0075]

[0076] The distribution of the output is adjusted by a temperature parameter T. When T approaches infinity, the temperature-learnable softmax degenerates into the standard softmax, and when T approaches 0, the temperature-learnable softmax degenerates into argmax. The present invention trains T so that the output of softmax is closer to argmax, thereby reducing the deviation of back propagation.

[0077] Step 3: Input the mapped features, superimposed features, and original features into the fusion network and task network of the original model respectively, train the original model, and stop training when the loss function is minimized to obtain a trained model.

[0078] Among them, the mapping feature is input into the fusion network and task network of the original model to obtain the inference result of the cache feature, the original feature is input into the fusion network and task network of the original model to obtain the inference result of the original feature, and the superposition feature is input into the fusion network and task network of the original model to obtain the inference result of the superposition feature. The loss function of the three inference results is the same as the loss function of the original model, and the loss values ​​of the three inference results are respectively recorded as loss cache 、loss origin 、loss add , the computational loss is the square of the quotient of the current computational amount and the maximum computational amount, recorded as loss comp The final loss is the weighted sum of the three inference result losses, and the loss function is calculated as:

[0079] loss total =αloss cache +βloss origin +γloss add +δloss comp

[0080] Among them, loss cache Represents the loss value of the inference result of the cached feature, loss origin Represents the loss value of the inference result of the original feature, loss add Represents the loss value of the inference result of the superimposed feature, loss comp represents the computational loss, α, β, γ, and δ are weight coefficients, which can be adjusted according to actual needs.

[0081] Training the original model is the process of continuously adjusting the weight coefficients α, β, γ, δ according to the needs to achieve different training goals. Figure 4 As shown, the training process of the present invention includes two stages: the pre-training stage and the strategy adjustment stage. In the pre-training stage, that is, the initial stage of training, the weight coefficient γ=δ=0, that is, the reasoning results and computational loss of the superimposed features are not considered, and only the reasoning results of the cached features and the reasoning results of the original features are considered, so that each module can quickly converge to an effective level. It is worth noting that the convergence speed of the modal cache is often slower than that of the original model. Therefore, it is possible to consider appropriately increasing the value of α to balance the training speed of the modal cache and the original model.

[0082] The strategy adjustment phase is to continue training on the model after pre-training. Due to the large difference in the network scale between the modal cache and the original model, the convergence speed is often inconsistent or even very different. Based on the experience of the pre-training phase, adjust α and β, and set α+β=γ=0.5. Usually β≤α≤10β. This is to balance the convergence speed of the modal cache and the original model to prevent overfitting or difficulty in convergence. Then dynamically adjust δ according to the actual demand for computing power. Take a batch as a unit. When the computing power is lower than expected, reduce the δ value by 10%. When the computing power is higher than expected, increase the δ value by 10%, so as to stabilize the computing power within a reasonable expected range and continuously optimize the inference effect.

[0083] The end-to-end training framework proposed in the present invention can guarantee both the upper limit (the reasoning effect of the original model) and the lower limit (the reasoning effect of the cache) through appropriate operating mechanisms, loss function designs and training strategies, and can coordinate the amount of computation and reasoning effect according to demand.

[0084] Step 4: Input the multimodal data into the trained model for inference.

[0085] like Figure 5 As shown, step 4 specifically includes the following processes:

[0086] 4.1. Input the multimodal data into the lightweight feature extraction network to obtain the lightweight features of each modal data. The lightweight features are cached through the modality cache to obtain the cache features.

[0087] 4.2. The lightweight feature input strategy network obtains the redundant judgment vector, and uses the redundant judgment vector to determine whether each modality is hit. If all the modalities are hit, the mapping features are directly indexed and the inference results are obtained through the sample cache. If not all the modalities are hit, the hit modal data is input into the feature extraction network of the original model to obtain the original features, and the original features and mapping features are combined and input into the fusion network and task network of the original model to obtain the inference results.

[0088] The key of the sample cache is the Cartesian product of the keys of each modality cache, and the value is the corresponding final inference result. The model pre-infers the prediction results of all combinations in each modality cache and caches them. During inference, once all modalities of the sample are determined by the policy network to hit the modality cache, the pre-inference results are directly indexed.

[0089] During inference, unlike the training phase, the output of the policy network and the output of the modality cache directly use argmax. The cached features obtained from the modality cache can directly index the mapping features in training without going through the feature mapping network. The original features selectively infer some modalities based on the output of the policy network. When the final inference result is obtained, the features are no longer superimposed. The mapping features and the original features are directly combined and input into the fusion network and task network of the original model to obtain the final inference result. When all modalities are hit, that is, when there is no redundancy in the modal data, the pre-inference result can be directly indexed without the need for re-inference.

[0090] The multi-granularity reasoning optimization method for multimodal streaming data of the present invention integrates the original model with a lightweight feature extraction network, a modality cache, a sample cache, and a feature mapping network, and jointly trains with a strategy network to find the overall optimal solution. It is an end-to-end training reasoning method that can improve the robustness of the original model and simplify the training steps. The end-to-end training reasoning method of the present invention has the ability to handle multi-granularity redundancy. At the modality granularity, a modality cache is designed separately for each modality. For a certain modality, when the modality is redundant (cache hit), the cache feature is obtained, and the dimension is aligned with the feature extracted by the original model (optimized model) to obtain the feature of the modality, without using the original model to extract the feature of the modality. At the sample granularity, when all modalities hit the cache, the result of the pre-inference of the cache can be directly indexed without going through the original model reasoning again. It can effectively handle partial modality redundancy and overall sample redundancy, and further improve the reasoning efficiency.

[0091] The present invention processes redundancy at the modal granularity. The missing of some modalities does not affect the redundant judgment of other modalities. For the missing modality, it is regarded as an all-zero vector when inputting the frame. The modality cache uses the all-zero vector as a special redundancy form and outputs the cache feature with the closest distance, which is suitable for modality missing.

[0092] The present invention realizes a general multimodal reasoning optimization framework by abstracting the multimodal task into three independent parts. It can also make the optimization process independent of the specific reasoning task and adapt to various reasoning tasks (discrete and continuous reasoning tasks) without being limited to discrete tasks, and has stronger versatility.

[0093] The present invention removes the redundancy of multimodal streams from two granularities, namely, between samples and within modalities, taking into account the characteristics of single samples and multiple samples, adapting to sparse reasoning and intensive reasoning tasks, and achieving a better balance between accuracy and efficiency. The inference efficiency of multimodal stream data is optimized from two granularities, namely, within modalities and between samples, by means of input filtering, and an end-to-end general framework is proposed, aiming to reduce the computational burden according to demand while maintaining the model performance as much as possible, and adapting to the different requirements for real-time performance and reasoning accuracy in complex Internet of Things.

[0094] Example 2

[0095] like Figure 6 As shown, this embodiment provides a multi-granularity reasoning optimization system for multi-modal streaming data, including:

[0096] The feature extraction module is used to input the multimodal data into a lightweight feature extraction network to obtain lightweight features of each modal data, and the multimodal data is input into the feature extraction network of the original model to obtain the original features.

[0097] The redundant filtering module is used to input the lightweight features into the modal cache to obtain the cache features. The cache features are passed through the feature mapping network to obtain the mapping features. The lightweight features are input into the strategy network to obtain the redundant judgment vector. The redundant judgment vector is used as the weight to superimpose the original features and the mapping features to obtain the superimposed features.

[0098] The policy network in the redundant filtering module includes a linear layer, a ReLU activation function, a linear layer, and an optimized argmax. The input dimension is the dimension of the lightweight feature of the corresponding modal data. The hidden layer dimension is flexibly adjusted according to the specific scenario. The optimized argmax uses softmax instead of argmax for calculation during back propagation. The value of argmax is output during forward propagation, and the gradient of softmax is output during back propagation. The expression is:

[0099] softout-detach(softout-hardout)

[0100] Among them, softout is the output of softmax, hardout is the output of argmax, and detach means gradient truncation, so that the gradient will not propagate to hardout, so that the forward propagation output result is hardout, and the back propagation output result is the gradient of softmax.

[0101] The loss function in the training module is calculated as follows:

[0102] loss total =αloss cache +βloss origin +γloss add +δlosscomp

[0103] Among them, loss cache Represents the loss value of the inference result of the cached feature, loss origin Represents the loss value of the inference result of the original feature, loss add Represents the loss value of the inference result of the superimposed feature, loss comp represents the computational loss, and α, β, γ, and δ are weight coefficients.

[0104] The training module is used to input the mapping features, superimposed features, and original features into the fusion network and task network of the original model respectively, train the original model, and stop the training when the loss function is minimized to obtain a trained model.

[0105] The process of training the original model in the training module includes a pre-training stage and a strategy adjustment stage. In the pre-training stage, the weight coefficient γ=δ=0 is set. In the strategy adjustment stage, α+β=γ=0.5, β≤α≤10β, and δ is dynamically adjusted. Taking a batch as a unit, when the amount of calculation is lower than expected, the δ value is reduced by 10%, and when the amount of calculation is higher than expected, the δ value is increased by 10%.

[0106] The inference module is used to input multimodal data into the trained model for inference.

[0107] The reasoning module is also used to:

[0108] 4.1. Input the multimodal data into the lightweight feature extraction network to obtain lightweight features of each modal data, and obtain cache features through modal caching;

[0109] 4.2. The lightweight feature input strategy network obtains the redundant judgment vector, and uses the redundant judgment vector to determine whether each modality is hit. If all the modalities are hit, the mapping features are directly indexed and the inference results are obtained through the sample cache. If not all the modalities are hit, the hit modal data is input into the feature extraction network of the original model to obtain the original features, and the original features and mapping features are combined and input into the fusion network and task network of the original model to obtain the inference results.

[0110] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-granularity reasoning optimization method for multimodal streaming data, characterized by: The following steps are involved: Step 1: Input the multimodal data into the lightweight feature extraction network to obtain lightweight features of each modal data, and input the multimodal data into the feature extraction network of the original model to obtain original features; Step 2: Input the lightweight features into the modal cache to obtain cache features. The cache features are passed through the feature mapping network to obtain mapping features. The lightweight features are input into the strategy network to obtain redundant judgment vectors. The redundant judgment vectors are used as weights to superimpose the original features and the mapping features to obtain superimposed features. Step 3: Input the mapped features, superimposed features, and original features into the fusion network and task network of the original model respectively, train the original model, and stop training when the loss function is minimized to obtain a trained model; Step 4: Input the multimodal data into the trained model for inference.

2. The multi-granularity reasoning optimization method for multimodal streaming data according to claim 1, characterized in that: In step 2, the policy network includes a linear layer, a ReLU activation function, a linear layer, and an optimized argmax. The input dimension is the dimension of the lightweight feature of the corresponding modal data. The hidden layer dimension is flexibly adjusted according to the specific scenario. The optimized argmax is calculated using softmax instead of argmax during back propagation. The value of argmax is output during forward propagation, and the gradient of softmax is output during back propagation. The expression is: softout-detach(softout-hardout) Among them, softout is the output of softmax, hardout is the output of argmax, and detach means gradient truncation, so that the gradient will not propagate to hardout, so that the forward propagation output result is hardout and the back propagation output result is the gradient of softmax.

3. The multi-granularity reasoning optimization method for multimodal streaming data according to claim 1, characterized in that: The calculation method of the loss function in step 3 is: loss total =αloss cache +βloss origin +γloss add +δloss comp Among them, loss cache Represents the loss value of the inference result of the cached feature, loss origin Represents the loss value of the inference result of the original feature, loss add Indicates the loss value of the inference result of the superimposed feature, loss comp represents the computational loss, and α, β, γ, and δ are weight coefficients.

4. The multi-granularity reasoning optimization method for multimodal streaming data according to claim 1, characterized in that: The process of training the original model in step 3 includes a pre-training stage and a strategy adjustment stage. In the pre-training stage, the weight coefficient γ is set to δ = 0. In the strategy adjustment stage, α + β is set to γ ​​= 0.5, β≤α≤10β, and δ is dynamically adjusted. Taking a batch as a unit, when the amount of calculation is lower than expected, the δ value is reduced by 10%, and when the amount of calculation is higher than expected, the δ value is increased by 10%.

5. The multi-granularity reasoning optimization method for multimodal streaming data according to claim 1, characterized in that: The step 4 comprises: 4.

1. Input the multimodal data into the lightweight feature extraction network to obtain lightweight features of each modal data. The lightweight features are cached through the modality to obtain cached features. 4.

2. The lightweight feature input strategy network obtains a redundant judgment vector, which is used to determine whether each modality is hit. If all modalities are hit, the mapping features are directly indexed and the inference result is obtained through the sample cache. If not all modalities are hit, the hit modal data is input into the feature extraction network of the original model to obtain the original features. The original features and mapping features are combined and input into the fusion network and task network of the original model to obtain the inference result.

6. A multi-granularity reasoning optimization system for multimodal streaming data, characterized by: include: The feature extraction module is used to input the multimodal data into the lightweight feature extraction network to obtain lightweight features of each modal data, and input the multimodal data into the feature extraction network of the original model to obtain the original features; Redundancy filtering module, used to input lightweight features into modal cache to obtain cache features, cache features are passed through feature mapping network to obtain mapping features, lightweight features are input into strategy network to obtain redundant judgment vectors, and redundant judgment vectors are used as weights to superimpose original features and mapping features to obtain superimposed features; The training module is used to input the mapping features, superimposed features, and original features into the fusion network and task network of the original model respectively, train the original model, and stop training when the loss function is minimized to obtain a trained model; The inference module is used to input multimodal data into the trained model for inference.

7. The multi-granularity reasoning optimization system for multimodal streaming data according to claim 6, characterized in that: The policy network in the redundant filtering module includes a linear layer, a ReLU activation function, a linear layer, and an optimized argmax. The input dimension is the dimension of the lightweight feature of the corresponding modal data. The hidden layer dimension is flexibly adjusted according to the specific scenario. The optimized argmax is calculated using softmax instead of argmax during back propagation. The value of argmax is output during forward propagation, and the gradient of softmax is output during back propagation. The expression is: softout-detach(softout-hardout) Among them, softout is the output of softmax, hardout is the output of argmax, and detach means gradient truncation, so that the gradient will not propagate to hardout, so that the forward propagation output result is hardout and the back propagation output result is the gradient of softmax.

8. The multi-granularity reasoning optimization system for multimodal streaming data according to claim 6, characterized in that: The loss function in the training module is calculated as follows: loss total =αloss cache +βloss origin +γloss add +δloss comp Among them, loss cache Represents the loss value of the inference result of the cached feature, loss origin Represents the loss value of the inference result of the original feature, loss add Indicates the loss value of the inference result of the superimposed feature, loss comp represents the computational loss, and α, β, γ, and δ are weight coefficients.

9. The multi-granularity reasoning optimization system for multimodal streaming data according to claim 6, characterized in that: The process of training the original model in the training module includes a pre-training stage and a strategy adjustment stage. In the pre-training stage, the weight coefficient γ is set to δ = 0. In the strategy adjustment stage, α + β is set to γ ​​= 0.5, β≤α≤10β, and δ is dynamically adjusted. Taking a batch as a unit, when the calculation amount is lower than expected, the δ value is reduced by 10%, and when the calculation amount is higher than expected, the δ value is increased by 10%.

10. The multi-granularity reasoning optimization system for multimodal streaming data according to claim 6, characterized in that: The reasoning module is also used to: 4.

1. Input the multimodal data into the lightweight feature extraction network to obtain lightweight features of each modal data. The lightweight features are cached through the modality to obtain cached features. 4.

2. The lightweight feature input strategy network obtains a redundant judgment vector, which is used to determine whether each modality is hit. If all modalities are hit, the mapping features are directly indexed and the inference result is obtained through the sample cache. If not all modalities are hit, the hit modal data is input into the feature extraction network of the original model to obtain the original features. The original features and mapping features are combined and input into the fusion network and task network of the original model to obtain the inference result.

Images