Data reasoning method and device based on multi-modal characteristic dynamic weighting

Abstract

The invention discloses a data reasoning method and device based on multi-modal characteristic dynamic weighting, and relates to the field of data processing or other related technical fields, and the method comprises the steps: collecting multi-modal data, and carrying out the preprocessing of the multi-modal data; carrying out feature extraction by adopting feature extraction models corresponding to different types of multi-modal data to obtain multi-modal features, and carrying out vector coding on the multi-modal features to obtain multi-modal feature vectors; and inputting the multi-modal feature vector into a reasoning model, configuring weight values for different types of multi-modal features through a gating mechanism and an attention mechanism of the reasoning model, performing feature fusion based on the weight values to obtain a fused feature vector, performing reasoning according to the fused feature vector, and outputting a reasoning result. According to the method and the device, the technical problem of relatively low reasoning accuracy of machine model data reasoning based on a single mode in an application scene with a complex and changeable environment in related technologies is solved.

Description

Technical Field

[0001] This invention relates to the field of data processing or other related technical fields, and more specifically, to a data reasoning method and apparatus based on dynamic weighting of multimodal features. Background Technology

[0002] With the rapid development of big data, cloud computing, and high-performance computing technologies, machine learning, especially deep learning, has made groundbreaking progress in the past decade, becoming one of the core technologies driving the development of artificial intelligence. The complexity and performance of models have continuously improved, evolving from initial linear models and shallow neural networks to today's deep neural networks, convolutional neural networks, recurrent neural networks, and transducers. These models have demonstrated outstanding capabilities in multiple fields such as image recognition, speech analysis, and natural language processing, significantly changing the way data is processed and understood. Data reasoning capabilities based on machine learning models are leading a technological revolution, covering numerous industries from healthcare, financial risk control, and intelligent manufacturing to transportation and smart homes, promoting the widespread adoption of automated decision-making and intelligent services.

[0003] In related technologies, efficient reasoning can be achieved using machine models by performing data reasoning on single-modal data. However, single-modal data reasoning has certain limitations in the face of complex and ever-changing application scenarios, and the accuracy of reasoning is difficult to guarantee.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a data reasoning method and apparatus based on dynamic weighting of multimodal features, which at least solves the technical problem in related technologies where data reasoning based on a single-modality machine model has low reasoning accuracy in application scenarios with complex and ever-changing environments.

[0006] According to one aspect of the present invention, a data inference method based on dynamic weighted multimodal features is provided, comprising: collecting multimodal data and preprocessing the multimodal data, wherein the types of the multimodal data include at least one of the following: image data, audio data, text data, and sensor data; extracting features using a feature extraction model corresponding to different types of multimodal data to obtain multimodal features, and vector encoding the multimodal features to obtain multimodal feature vectors; inputting the multimodal feature vectors into an inference model, configuring weight values ​​for different types of multimodal features through the gating mechanism and attention mechanism of the inference model, performing feature fusion based on the weight values ​​to obtain fused feature vectors, and performing inference based on the fused feature vectors to output an inference result, wherein the inference model is a pre-constructed machine learning model for data inference.

[0007] Furthermore, the step of using a feature extraction model corresponding to different types of multimodal data for feature extraction includes: using a convolutional neural network to extract features from the image data to obtain visual features; using a time-frequency transform network and a temporal convolutional network to extract features from the audio data to obtain audio features; using a natural language processing model to extract features from the text data to obtain semantic features; and using a neural network model to extract features from the sensor data to obtain environmental features.

[0008] Furthermore, the step of configuring weight values ​​for different types of multimodal features through the gating mechanism of the inference model includes: evaluating the quality of the image data in conjunction with the environmental features through the gating mechanism of the inference model, and configuring visual weight values ​​for the visual features based on the quality of the image data; evaluating the quality of the audio data in conjunction with the environmental features through the gating mechanism of the inference model, and configuring audio weight values ​​for the audio features based on the quality of the audio data; and evaluating the quality of the text data through the gating mechanism of the inference model, and configuring semantic weight values ​​for the semantic features based on the quality evaluation results of the text data, wherein evaluating the quality of the text data includes at least one of the following: evaluating the information completeness of the data, and evaluating whether the text data contains preset information.

[0009] Furthermore, the step of configuring weight values ​​for different types of multimodal features through the attention mechanism of the inference model includes: calculating the correlation score between the multimodal features through the attention mechanism; calculating the importance score of the multimodal data to the inference task through the attention mechanism combined with historical inference records; and configuring attention weight values ​​for the multimodal features based on the correlation score and the importance score of the multimodal features.

[0010] Further, the step of performing feature fusion based on the weight values ​​to obtain the fused feature vector includes: performing modality alignment projection on the multimodal features of different types to obtain multimodal feature projections; and performing weighted fusion calculation on the multimodal feature projections based on the weight values ​​of each multimodal feature to obtain the fused feature vector.

[0011] Furthermore, before performing feature fusion based on the weight values, the method further includes: collecting the operating status and computing power budget of the edge device, and calculating a weight threshold based on the operating status and computing power budget, wherein the edge device refers to the device performing the inference task; and removing the multimodal features whose weight values ​​are less than the weight threshold.

[0012] Furthermore, the data inference method based on multimodal feature dynamic weighting also includes: during the inference process, calculating the confidence value of the intermediate inference result in each intermediate layer of the inference model, wherein the intermediate layer represents each hidden layer in the inference model; when the confidence value reaches a preset confidence threshold, stopping the inference, and outputting the final inference result in advance based on the intermediate inference result, wherein the confidence threshold is updated in real time based on the operating status and computing power budget of the edge device.

[0013] According to another aspect of the present invention, a data inference apparatus based on dynamic weighted multimodal features is also provided, comprising: an acquisition unit for acquiring multimodal data and preprocessing the multimodal data, wherein the types of the multimodal data include at least one of the following: image data, audio data, text data, and sensor data; an extraction unit for extracting features using a feature extraction model corresponding to different types of multimodal data to obtain multimodal features, and performing vector encoding on the multimodal features to obtain multimodal feature vectors; and an inference unit for inputting the multimodal feature vectors into an inference model, configuring weight values ​​for different types of multimodal features through the gating mechanism and attention mechanism of the inference model, performing feature fusion based on the weight values ​​to obtain fused feature vectors, and performing inference based on the fused feature vectors to output inference results, wherein the inference model is a pre-built machine learning model for data inference.

[0014] Furthermore, the extraction unit includes: a first extraction module, used to extract features from the image data using a convolutional neural network to obtain visual features; a second extraction module, used to extract features from the audio data using a time-frequency transform network and a temporal convolutional network to obtain audio features; a third extraction module, used to extract features from the text data using a natural language processing model to obtain semantic features; and a fourth extraction module, used to extract features from the sensor data using a neural network model to obtain environmental features.

[0015] Further, the local inference unit includes: a first configuration module, used to evaluate the quality of the image data in conjunction with the environmental features through the gating mechanism of the inference model, and configure visual weight values ​​for the visual features based on the quality of the image data; a second configuration module, used to evaluate the quality of the audio data in conjunction with the environmental features through the gating mechanism of the inference model, and configure audio weight values ​​for the audio features based on the quality of the audio data; and a third configuration module, used to perform quality evaluation on the text data through the gating mechanism of the inference model, and configure semantic weight values ​​for the semantic features based on the quality evaluation results of the text data, wherein the quality evaluation of the text data includes at least one of the following: evaluating the information completeness of the data, and evaluating whether the text data contains preset information.

[0016] Furthermore, the inference unit further includes: a first calculation module, used to calculate the correlation score between the multimodal features through the attention mechanism; a second calculation module, used to calculate the importance score of the multimodal data to the inference task through the attention mechanism combined with historical inference records; and a first configuration module, used to configure attention weight values ​​for the multimodal features based on the correlation score and the importance score of the multimodal features.

[0017] Furthermore, the inference unit further includes: a first projection module, used to perform modal alignment projection on the different types of multimodal features to obtain multimodal feature projections; and a first fusion module, used to perform weighted fusion calculation on the multimodal feature projections based on the weight values ​​of each of the multimodal features to obtain the fused feature vector.

[0018] Furthermore, the data inference device based on dynamic weighting of multimodal features further includes: a third calculation module, used to collect the operating status and computing power budget of the end-side device, and calculate a weight threshold based on the operating status and the computing power budget, wherein the end-side device refers to the device performing the inference task; and a first elimination module, used to eliminate the multimodal features whose weight values ​​are less than the weight threshold.

[0019] Furthermore, the data inference device based on multimodal feature dynamic weighting also includes: a fourth calculation module, used to calculate the confidence value of the intermediate inference result in each intermediate layer of the inference model during the inference process, wherein the intermediate layer represents each hidden layer in the inference model; and a first output module, used to stop the inference when the confidence value reaches a preset confidence threshold, and output the final inference result in advance based on the intermediate inference result, wherein the confidence threshold is updated in real time based on the operating status and computing power budget of the edge device.

[0020] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute any of the above-described data reasoning methods based on multimodal feature dynamic weighting.

[0021] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement any of the above-described data reasoning methods based on multimodal feature dynamic weighting.

[0022] In this application, the following steps are performed: collecting multimodal data and preprocessing the multimodal data, wherein the types of multimodal data include at least one of the following: image data, audio data, text data, and sensor data; then, feature extraction is performed using a feature extraction model corresponding to different types of multimodal data to obtain multimodal features; the multimodal features are then vector-encoded to obtain multimodal feature vectors; finally, the multimodal feature vectors are input into an inference model; weight values ​​are configured for different types of multimodal features through the gating and attention mechanisms of the inference model; feature fusion is performed based on the weight values ​​to obtain fused feature vectors; and inference is performed based on the fused feature vectors to output the inference results. The inference model is a pre-built machine learning model for data inference.

[0023] In this application, data is collected from multiple dimensions to obtain multimodal data, and features are extracted from the multimodal data. Gating and attention mechanisms are used to dynamically and adaptively configure the weights of the multimodal features to perform weighted fusion of the multimodal features. This data is then used as input data for a machine learning model to perform multi-dimensional data inference. This enhances the adaptability and reliability of the machine learning model to complex and changing environments, improves the model's decision quality and environmental adaptability, and thus improves the accuracy of data inference. This solves the technical problem in related technologies where machine learning models based on a single modality have low inference accuracy in complex and changing application scenarios. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0025] Figure 1 This is a flowchart of an optional data reasoning method based on dynamic weighting of multimodal features according to an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of an optional data reasoning process based on dynamic weighting of multimodal features according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of an optional multimodal feature weight configuration process according to an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of an optional multimodal feature fusion process according to an embodiment of the present invention;

[0029] Figure 5 This is a schematic diagram of an optional data inference device based on dynamic weighting of multimodal features according to an embodiment of the present invention;

[0030] Figure 6 This is a hardware structure block diagram of an electronic device (or mobile device) that performs a data reasoning method based on dynamic weighting of multimodal features according to an embodiment of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] To facilitate understanding of the present invention by those skilled in the art, some terms or nouns involved in the various embodiments of the present invention are explained below:

[0034] The softmax function is a mathematical function that is widely used in machine learning, especially in neural networks, to transform a string of arbitrary real numbers into a set of probability distributions.

[0035] LayerNorm is a normalization technique used in neural networks.

[0036] CPU utilization rate refers to the proportion of time that the Central Processing Unit (CPU) is actually used by various processes or threads within a certain period of time, relative to the total available time.

[0037] FLOPS consumption, in computing tasks, is the full name of the number of floating-point operations performed by the processor per second.

[0038] It should be noted that the data reasoning method and apparatus based on dynamic weighting of multimodal features in this application can be used in the field of data processing for reasoning on data based on dynamic weighting of multimodal features, and can also be used in any field other than data processing for reasoning on data based on dynamic weighting of multimodal features. This application does not limit the application field of the data reasoning method and apparatus based on dynamic weighting of multimodal features.

[0039] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) involved in this application are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of such data all comply with the relevant laws, regulations, and standards of the relevant regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse. For example, this system has interfaces with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface, and the relevant information is obtained only after receiving consent from the aforementioned user or organization.

[0040] It should be noted that in this application, when collecting and analyzing customer information, users are provided with corresponding operation entry points to choose whether to agree to or reject the automated decision-making results; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0041] The following embodiments of the present invention can be applied to various data inference systems / applications / devices based on dynamic weighting of multimodal features. The present invention collects multi-source data from multiple dimensions as input for data inference, providing comprehensive data support. Simultaneously, it employs gating and attention mechanisms to adaptively perceive the quality of each modal input (e.g., image occlusion, speech noise, missing text, etc.), dynamically adjusting the modality fusion weights. This enables intelligent redistribution in single or multiple modality degradation scenarios, significantly improving the stability and accuracy of the inference model in complex and variable non-ideal environments.

[0042] The present invention will now be described in detail with reference to various embodiments.

[0043] Example 1

[0044] According to an embodiment of the present invention, an embodiment of a data reasoning method based on dynamic weighting of multimodal features is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0045] Figure 1 This is a flowchart of an optional data inference method based on dynamic weighting of multimodal features according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0046] The data inference in this invention can be used for data inference scenarios such as risk identification and target identification. By applying dynamic modal weighting and lightweight models based on gating and attention mechanisms, it can run well on resource-constrained devices such as mobile terminals, in-vehicle systems, and IoT devices. It can activate key modal paths and computing units on demand, improve inference efficiency, enhance environmental adaptability, and promote the development of multimodal intelligence towards edge computing, real-time computing, and intelligent computing.

[0047] Step S101: Collect multimodal data and preprocess the multimodal data. The types of multimodal data include at least one of the following: image data, audio data, text data, and sensor data.

[0048] In step S101 above, the system collects multimodal information from various data sources, including image data, audio data, text data, and sensor data. Image data may originate from cameras or other image acquisition devices; audio data is captured by microphones or recording devices; text data may be user input or extracted from documents; sensor data covers measurements of various physical quantities such as temperature, humidity, and location information. The purpose of collecting multimodal data is to build a comprehensive information foundation to adapt to the intelligent reasoning needs in various scenarios.

[0049] The acquired multimodal data is then preprocessed. For image data, preprocessing includes resizing, color normalization, and data augmentation. Resizing ensures the image meets the network input requirements; color normalization makes image features more stable during training; data augmentation increases the model's generalization ability through random transformations such as rotation, flipping, or brightness adjustment. The goal of preprocessing is to eliminate unnecessary noise while preserving the original image information, thereby enhancing the model's adaptability to image diversity.

[0050] Audio data preprocessing includes noise reduction, dereverberation, volume normalization, and spectral transformation. Noise reduction and dereverberation aim to improve speech clarity and reduce the impact of ambient noise; volume normalization ensures that audio signals from different sources are processed at the same level; and spectral transformation converts the time-domain signal into a frequency-domain representation that is easy to extract features from. These preprocessing steps ensure that audio features can be accurately captured by the model, maintaining good recognition performance even in noisy environments.

[0051] Text preprocessing involves word segmentation, word embedding, and sequence length normalization. Word segmentation breaks down text into words or phrases, facilitating subsequent semantic analysis; word embedding transforms words into dense vectors, capturing semantic relationships between words; and sequence length normalization ensures the text sequence fits the input requirements of the model. The goal of preprocessing is to convert natural language into a form suitable for deep learning models, thereby enabling the effective use of textual information.

[0052] Sensor data preprocessing includes data cleaning, feature extraction, and signal smoothing. Data cleaning removes invalid or erroneous readings; feature extraction extracts key information from the raw signal, such as trends, periodicity, or peaks; signal smoothing reduces data fluctuations through filtering techniques, maintaining signal continuity and consistency. The purpose of preprocessing is to extract useful information from sensor data, reduce interference caused by signal fluctuations, and improve the decision quality and stability of the model.

[0053] Step S102: Feature extraction is performed using a feature extraction model corresponding to different types of multimodal data to obtain multimodal features, and the multimodal features are vector-encoded to obtain multimodal feature vectors.

[0054] In step S102 above, a corresponding feature extraction model is selected for processing based on the type of input multimodal data. Each type of multimodal data is fed into the corresponding feature extraction model for processing. For example, image data undergoes feature extraction through convolutional layers to obtain a series of visual feature maps; audio signals are converted into spectrograms through Fourier transform, and then feature extraction is performed through a temporal network; text data is first converted into word vectors, and then processed through multiple network layers to extract semantic features. The extracted multimodal features are then vector-encoded, converting them into fixed-size feature vectors for easier subsequent fusion and processing. The encoding process typically involves linear transformations, pooling, or low-rank matrix factorization to compress feature maps or sequences into a unified vector form. Vector encoding simplifies feature representation and facilitates comparison and fusion between modalities.

[0055] Furthermore, the steps of feature extraction using feature extraction models corresponding to different types of multimodal data include: using convolutional neural networks to extract features from image data to obtain visual features; using time-frequency transformation networks and temporal convolutional networks to extract features from audio data to obtain audio features; using natural language processing models to extract features from text data to obtain semantic features; and using neural network models to extract features from sensor data to obtain environmental features.

[0056] By selecting a suitable lightweight feature extraction model for each type of data (image, audio, text, and sensor data), representative information is effectively extracted from the raw data without adding unnecessary computational burden. Specifically, convolutional neural networks (CNNs) are used for feature extraction from image data. CNNs automatically identify local features in images, such as edges, textures, and shapes, using filters in their hierarchical structure. Through a series of convolutional layers, activation functions, and pooling layers, CNNs can gradually construct a high-level abstract representation of the image. In this embodiment, the CNN is configured as a lightweight version to reduce computational resource requirements while maintaining the key ability to extract visual features.

[0057] Feature extraction from audio data is performed using time-frequency transform networks and temporal convolutional networks. Time-frequency transforms, such as short-time Fourier transforms, convert audio signals in the time domain into frequency domain representations, facilitating the capture of frequency and temporal features. Temporal convolutional networks, on the other hand, can handle the temporal dependencies of audio signals, learning and extracting features such as rhythm, pitch, and semantics through a deep learning architecture. This combination ensures rich audio features even in complex audio scenarios. Both the time-frequency transform networks and temporal convolutional networks are configured in lightweight versions.

[0058] Natural Language Processing (NLP) models are used to extract features from text data. NLP models understand the semantic structure of text, including the meaning of words, sentence grammar, and paragraph logic. These models transform text into meaningful semantic feature vectors by learning word embeddings and analyzing context.

[0059] For sensor data, a neural network model is used for feature extraction. Sensor data may include signals such as temperature, humidity, acceleration, and geographic location. The neural network model, such as a multilayer perceptron, can process the time-series characteristics of sensor signals and extract features reflecting the environmental state. This step is crucial for understanding environmental context information and for integrating visual, audio, and textual modalities.

[0060] Step S103: Input the multimodal feature vector into the inference model, configure weight values ​​for different types of multimodal features through the gating and attention mechanisms of the inference model, perform feature fusion based on the weight values ​​to obtain the fused feature vector, and perform inference based on the fused feature vector to output the inference result.

[0061] In step S103 above, the multimodal feature vectors are input into the inference model, which is a pre-built machine learning model for data inference. The inference model first dynamically generates a gate value between 0 and 1 based on the quality and relevance of the input feature vectors using a built-in gating mechanism. This gate value determines the degree of participation of specific modal features in the subsequent fusion process. The gate value is calculated using a parameterized function, and the activation intensity of each modal feature can be determined based on the parameters learned from the input features and the current device's computing power budget threshold.

[0062] Subsequently, the inference model uses an attention mechanism to calculate attention weights. This mechanism, based on feature vectors and generated through a softmax function, ensures that the sum of the attention weights for all modalities equals 1. The attention weights reflect the degree of influence of each modal feature on the overall inference result in a specific context, thus allowing the model to prioritize the most relevant modal information during inference.

[0063] Based on the calculated gating values ​​and attention weights, the inference model performs feature fusion operations to ensure that features from different modalities are transformed into a unified representation space. By introducing a LayerNorm layer, the fusion process is normalized and performed smoothly, avoiding the adverse effects of differences in feature scaling.

[0064] The fused feature vector is then passed to the hidden layer of the inference model to perform the final inference task, such as classification, recognition, or understanding. The output reflects a comprehensive judgment based on multimodal information, and its accuracy and robustness benefit from the adaptive characteristics of the dynamic weighted fusion process.

[0065] In one optional embodiment, the inference model is trained using a combination of multimodal joint training, knowledge distillation mechanism, and quantization mechanism.

[0066] Furthermore, the step of configuring weight values ​​for different types of multimodal features through the gating mechanism of the inference model includes: evaluating the quality of image data in conjunction with environmental features through the gating mechanism of the inference model, and configuring visual weight values ​​for visual features based on the quality of the image data; evaluating the quality of audio data in conjunction with environmental features through the gating mechanism of the inference model, and configuring audio weight values ​​for audio features based on the quality of the audio data; and evaluating the quality of text data through the gating mechanism of the inference model, and configuring semantic weight values ​​for semantic features based on the quality evaluation results of the text data, wherein the quality evaluation of text data includes at least one of the following: evaluating the information completeness of the data, and evaluating whether the text data contains preset information.

[0067] Specifically, environmental features are input into the inference model. These features include, but are not limited to, light intensity, ambient noise level, temperature, humidity, and other physical quantities that may affect the quality of modal data. These features provide real-time environmental context information for dynamic weight configuration, helping the model assess the reliability of different modal data based on current environmental conditions.

[0068] Subsequently, the inference model uses its gating mechanism to evaluate the quality of the image data in conjunction with environmental features. For example, the model can dynamically adjust visual weights by considering factors such as lighting conditions, image sharpness, and field-of-view occlusion. If the image data is in a low-light environment or has significant occlusion, the visual weights will be reduced, and vice versa. This process ensures that when image quality is poor, the model can reduce its reliance on visual features and instead utilize more audio or text features for inference.

[0069] The inference model uses a gating mechanism to evaluate the quality of audio data and dynamically configures audio weight values ​​based on environmental features such as background noise levels and the stability of the audio source. In high-noise environments, the audio weight values ​​will be reduced accordingly to minimize the impact of noise on audio features, thereby allowing more attention to the image or text modality during inference.

[0070] For text data, the inference model performs quality assessment through a gating mechanism, including checking information completeness and identifying pre-defined information. If the text data lacks key information or deviates from the pre-defined topic, the semantic weight value will be reduced accordingly, and vice versa. In this way, even when the text is incomplete or contains misleading information, the inference model can still improve the accuracy and robustness of the final inference result by emphasizing features from other modalities through a weighting mechanism.

[0071] Through the above steps, dynamic weight configuration for multimodal data is achieved. This dynamic weight configuration process allows the inference model to automatically adjust the contribution of different modal features based on real-time environmental conditions and the quality of input data, thereby maintaining stable inference performance in varied real-world scenarios. By assessing the quality of modal data through environmental features, the inference model can intelligently shut down or reduce computational paths for low-quality modalities on resource-constrained devices, effectively reducing computational resources and energy consumption, and improving overall efficiency. Furthermore, dynamic weight configuration ensures that when the quality of data in a particular modality deteriorates, the model can automatically switch to a higher-quality modality, using its information for compensation, thus making more reliable and accurate decisions in complex environments.

[0072] Furthermore, the steps of configuring weight values ​​for different types of multimodal features through the attention mechanism of the inference model include: calculating the correlation score between multimodal features through the attention mechanism; calculating the importance score of multimodal data to the inference task through the attention mechanism combined with historical inference records; and configuring attention weight values ​​for multimodal features based on the correlation score and importance score of multimodal features.

[0073] Specifically, while configuring gating weights, the thrust model employs an attention mechanism to assign attention weights to multimodal features. First, the attention mechanism calculates the correlation scores between different modal features, which can be achieved by calculating the similarity of modal feature vectors (e.g., dot product), thus revealing the strength of the association between each modality. If visual and audio features show a high correlation, it may mean that the currently input visual and audio information are related in expressing the same or complementary concepts; therefore, their correlation score will be high.

[0074] Subsequently, the inference model utilizes historical inference records to assess the relative importance of each modal feature in the inference task. This scoring mechanism considers historical statistical information on the contribution of modal features to the outcome in past inferences; for example, if a modal feature has been a decisive source of information in previous inferences, its importance score will be higher. The model can maintain a historical database of attention weights, statistically analyzing the frequency and influence of each modal feature in historical inferences, as the basis for calculating the importance score.

[0075] Finally, the inference model configures attention weights based on the relevance and importance scores of multimodal features. Relevance scores reflect the immediate interactions between modal features, while importance scores reveal the long-term value of each modal feature. The calculation of attention weights comprehensively considers both aspects, ensuring that the inference process involves both immediate synergy between modalities and long-term considerations based on historical experience. The final attention weights are assigned to each modality, guiding the dynamic trade-off process of feature fusion.

[0076] Dynamic configuration of attention weights ensures efficient use of resources, avoiding wasting computational power on irrelevant or low-quality modal features, while concentrating resources on high-quality or highly relevant features, thereby improving inference efficiency and accuracy.

[0077] Furthermore, before feature fusion based on weight values, the process includes: collecting the operating status and computing power budget of the edge device, and calculating the weight threshold based on the operating status and computing power budget, where the edge device refers to the device that performs the inference task; and removing multimodal features with weight values ​​less than the weight threshold.

[0078] Specifically, to ensure stable operation of the inference process under limited resource conditions, this embodiment of the invention employs feature sparsity processing to intelligently select modal features for inference, avoiding unnecessary computation and improving resource utilization efficiency. Before performing feature fusion, the current operating status and computing power budget of the device executing the inference task are first obtained. The operating status may include indicators such as CPU utilization, memory usage, battery level, network bandwidth, and temperature. The computing power budget represents the maximum computing resources that the device can allocate under the current task. Combining the obtained operating status and computing power budget, the system calculates a dynamic weight threshold. The calculation of the weight threshold is based on the device's current resource constraints and needs to achieve efficient resource utilization. For example, in cases of low battery or high CPU utilization, the weight threshold may be set higher to save resources, thereby eliminating more low-contribution modal features. After calculating the weight threshold, the system checks the weight value of each modal feature. If the weight value of a modal feature is less than the weight threshold, the feature will be marked as a low-contribution feature and removed from the subsequent feature fusion process. In this way, only modal features that are highly relevant and contribute significantly to the current environment and task will be retained and participate in subsequent feature fusion operations. Through this process, the system can significantly reduce computational load and resource consumption while ensuring inference accuracy.

[0079] The calculation of weight thresholds takes into account environmental factors and equipment status, which helps the system maintain stable inference performance and improves overall robustness even under resource-constrained or harsh environments. Removing low-contribution modal features reduces the use of computing resources, especially under resource-constrained conditions, ensuring the normal execution of inference tasks.

[0080] Furthermore, the steps for fusing features based on weight values ​​to obtain a fused feature vector include: performing modal alignment projection on different types of multimodal features to obtain multimodal feature projections; and performing weighted fusion calculation on the multimodal feature projections based on the weight values ​​of each multimodal feature to obtain a fused feature vector.

[0081] Specifically, different types of multimodal features are transformed into the same feature space through modality alignment projection. This process typically involves linear transformations or neural network layers, such as fully connected layers, to ensure that features from different modalities match in dimensionality and semantics, facilitating subsequent fusion operations. Then, based on the weight values ​​of each multimodal feature, a weighted fusion calculation is performed on the projected multimodal feature vectors. The weight values ​​reflect the relative importance and reliability of each modal feature under the current input conditions. The weighted fusion calculation is performed using the formula... To achieve, among which, To use a sparse mask, control sparse activation to save computational resources. It is the modal projection matrix. It is the projected feature vector, while the LayerNorm layer ensures the consistency of the fused feature vector z in terms of scale, avoiding numerical instability issues.

[0082] Furthermore, the data inference method based on multimodal feature dynamic weighting also includes: during the inference process, calculating the confidence value of the intermediate inference result at each intermediate layer of the inference model, where the intermediate layer represents each hidden layer in the inference model; when the confidence value reaches a preset confidence threshold, stopping the inference, and outputting the final inference result in advance based on the intermediate inference result, where the confidence threshold is updated in real time based on the operating status and computing power budget of the edge device.

[0083] Specifically, for each intermediate hidden layer of the inference model, the system calculates the confidence value of the intermediate inference result. The confidence value represents the credibility of the inference result of the current layer and can be determined in various ways. For example, in classification tasks, it may correspond to the maximum class probability value; in generation tasks, it may be based on the coherence and diversity of the generated sequences. This calculation process is performed on each hidden layer of the model, allowing the system to capture possible termination points early in the inference process.

[0084] To adapt to the operating status and computing budget of edge devices, the system updates a dynamic confidence threshold in real time. This threshold is based on the device's current resource status, such as remaining battery power, CPU load, and memory usage, as well as computing budget requirements, such as the maximum acceptable FLOPS consumption. The strategy for setting the confidence threshold may be to set a lower threshold under high resource conditions, allowing the model to run more layers to improve inference accuracy; conversely, to set a higher threshold when resources are scarce, triggering inference termination earlier to save computing power and energy consumption.

[0085] The system continuously monitors the confidence values ​​of intermediate layers. When the confidence value of any layer reaches or exceeds the real-time updated confidence threshold, the inference process is terminated, and the system outputs the final inference result in advance based on the intermediate inference results of that layer. This early termination of the inference process aims to avoid continuing unnecessary calculations when high-confidence results have already appeared, thereby ensuring the quality of inference while achieving efficient utilization of computing resources.

[0086] Through the above steps, multimodal data is collected and preprocessed. The types of multimodal data include at least one of the following: image data, audio data, text data, and sensor data. Then, feature extraction models corresponding to different types of multimodal data are used to extract features, resulting in multimodal features. These features are then vector-encoded to obtain multimodal feature vectors. Finally, the multimodal feature vectors are input into the inference model. The inference model uses gating and attention mechanisms to assign weight values ​​to different types of multimodal features. Feature fusion is performed based on these weight values ​​to obtain fused feature vectors. Inference is then performed based on these fused feature vectors to output the inference results. The inference model is a pre-built machine learning model for data inference.

[0087] In this embodiment, data is collected from multiple dimensions to obtain multimodal data, and features are extracted from the multimodal data. Gating and attention mechanisms are used to dynamically and adaptively configure the weights of the multimodal features to perform weighted fusion of the multimodal features. This data is then used as input data for a machine learning model to perform multi-dimensional data inference. This enhances the adaptability and reliability of the machine learning model to complex and changing environments, improves the model's decision quality and environmental adaptability, and thus improves the accuracy of data inference. This solves the technical problem in related technologies where machine learning models based on a single modality have low inference accuracy in complex and changing application scenarios.

[0088] The following describes in detail another optional implementation method.

[0089] Figure 2 This is a schematic diagram of an optional data inference process based on dynamic weighting of multimodal features according to an embodiment of the present invention, such as... Figure 2 As shown, the data inference process based on dynamic weighting of multimodal features specifically includes:

[0090] Step 1: Data Collection;

[0091] It collects image data, audio data, text data, and sensor data from multiple dimensions.

[0092] Step 2, multimodal feature extraction;

[0093] Modal features are extracted from images, audio, and text respectively. Furthermore, visual features are extracted from image modalities using lightweight convolutional neural networks or visual neural network models. Acoustic features are extracted from audio modalities using time-frequency transformation combined with a small temporal network. Semantic features of the text modality were extracted using a compressed version of the large language model. .

[0094] Step 3, dynamic weighting;

[0095] Figure 3 This is a schematic diagram of an optional multimodal feature weight configuration process according to an embodiment of the present invention, such as... Figure 3 As shown, the multimodal feature weight configuration process includes:

[0096] Step 1: Obtain multimodal features;

[0097] Step two, feature encoding;

[0098] Step 3: Calculate the weights using a gating mechanism to determine the gating values ​​for each modal feature;

[0099] Step 4: Calculate the attention weight values ​​for each modality feature using the attention mechanism.

[0100] Step 5: Configure modal feature weights by combining gating values ​​and attention weights.

[0101] Specifically, a gated attention mechanism is employed to generate dynamic fusion weights for each modality, enabling environment-adaptive modality selection and weighted fusion. Specifically, multimodal data is processed through a feature encoding layer to obtain a representation, and a gating value is calculated. With attention weight This allows for the configuration of gating weights and attention weights for multimodal features. Indicates the first The activation intensity of each mode. For the Sigmoid activation function, the parameters are... This weight is learned through training. It comprehensively considers the input quality and semantic importance of the modality, achieving more robust adaptive decision-making.

[0102] Step four, feature fusion;

[0103] Figure 4 This is a schematic diagram of an optional multimodal feature fusion process according to an embodiment of the present invention, such as... Figure 4 As shown, the multimodal feature fusion process specifically includes: processing visual features... Acoustic features (corresponding to the audio features mentioned above) semantic features Perform modal alignment projection (such as linear transformation to a unified dimension) to obtain a linear projection. , , And combined with modal weights , , Perform weighted fusion. ,in For modal projection, To use a sparse mask, control sparse activation to save computational resources. It is the projected feature vector, and a sparse activation mechanism is introduced. When the weight of a certain mode is less than When the preset threshold is reached, the deep network computation path related to the modality feature in subsequent inference tasks is closed, conditional computation is implemented, energy consumption is further reduced, lightweight inference is achieved, and the adaptability of the inference system to the environment is improved.

[0104] Furthermore, to improve the deployment efficiency of the model on edge devices, this invention adopts a collaborative optimization strategy of knowledge distillation and quantization-aware training. The joint training process of knowledge distillation and quantization-aware training is carried out synchronously in the end-to-end framework, ensuring that the lightweight model achieves an optimal balance between accuracy and efficiency. The further enhanced knowledge distillation uses a high-performance multimodal model as the teacher model to guide the lightweight inference model to learn its output distribution (soft labels) and intermediate layer feature representations (feature distillation), preserving cross-modal semantic alignment capabilities. Quantization training employs quantization-aware training or post-training quantization techniques to quantize model weights and activation values ​​to INT8 or FP16 precision, simulating quantization errors during training to mitigate accuracy loss.

[0105] Step 5, data reasoning.

[0106] For example, in in-vehicle voice assistant scenarios, when the vehicle is traveling in high-noise environments (such as highways or tunnels), the audio modal signal-to-noise ratio decreases, and the gating mechanism automatically reduces the weight of the voice modal. Simultaneously, the system increases the weight of the visual modal (lip movement recognition) from the camera and the text input from the central control screen, achieving more accurate intent understanding. In nighttime or backlit environments, if image quality degrades, the system automatically enhances the fusion ratio of voice and text modalities to ensure continuous interaction.

[0107] In video surveillance scenarios, when a target person's face is obscured or backlighting renders the image information unreliable, the gating attention module identifies this anomaly and significantly reduces the image modality weights, instead relying on synchronously acquired voice alarm information or log text records for event judgment. If no critical visual changes are confirmed, the image encoding path can be shut down, allowing only the audio and text channels to operate, significantly reducing system power consumption.

[0108] When running a multimodal inference system on a smartphone or wearable device, the computing power budget control module triggers a power-saving mode when the device is in a low-power or weak network state: only key modalities (such as voice wake-up + text response) are activated, and a lower inference depth threshold is set, allowing the inference model to output results early in the intermediate layers. Combined with a dynamic weighting mechanism, this enables the system to still perform inference tasks under resource-constrained conditions.

[0109] This invention collects multi-source data from multiple dimensions as input for data inference, providing comprehensive data support for data inference. At the same time, it adopts gating and attention mechanisms to adaptively perceive the quality of each modal input (e.g., image occlusion, speech noise, missing text, etc.), dynamically adjusts the modality fusion weights, and achieves intelligent redistribution in single or multiple modality degradation scenarios, significantly improving the stability and inference accuracy of the inference model in complex and ever-changing non-ideal environments.

[0110] The following is a detailed description with reference to another embodiment.

[0111] Example 2

[0112] The data reasoning device based on dynamic weighting of multimodal features provided in this embodiment includes multiple implementation units, each of which corresponds to a specific implementation step in the above embodiment 1. The specific implementation method and beneficial effects can be referred to the aforementioned method embodiment, and will not be repeated here.

[0113] Figure 5 This is a schematic diagram of an optional data inference device based on dynamic weighted multimodal features according to an embodiment of the present invention, such as... Figure 5 As shown, the data inference device based on multimodal feature dynamic weighting may include: a data acquisition unit 51, an extraction unit 52, and an inference unit 53, wherein,

[0114] The acquisition unit 51 is used to acquire multimodal data and preprocess the multimodal data, wherein the types of multimodal data include at least one of the following: image data, audio data, text data, and sensor data.

[0115] Extraction unit 52 is used to extract features using a feature extraction model corresponding to different types of multimodal data, obtain multimodal features, and perform vector encoding on the multimodal features to obtain multimodal feature vectors;

[0116] The inference unit 53 is used to input multimodal feature vectors into the inference model, configure weight values ​​for different types of multimodal features through the gating and attention mechanisms of the inference model, perform feature fusion based on the weight values ​​to obtain fused feature vectors, and perform inference based on the fused feature vectors to output the inference results. The inference model is a pre-built machine learning model for data inference.

[0117] The aforementioned data inference device based on dynamic weighted multimodal features collects multimodal data through acquisition unit 51 and preprocesses the multimodal data. The types of multimodal data include at least one of the following: image data, audio data, text data, and sensor data. The extraction unit 52 extracts features using a feature extraction model corresponding to different types of multimodal data to obtain multimodal features, and then performs vector encoding on the multimodal features to obtain multimodal feature vectors. The inference unit 53 inputs the multimodal feature vectors into the inference model, configures weight values ​​for different types of multimodal features through the gating and attention mechanisms of the inference model, performs feature fusion based on the weight values ​​to obtain fused feature vectors, and performs inference based on the fused feature vectors to output the inference result. The inference model is a pre-built machine learning model for data inference.

[0118] In this embodiment, data is collected from multiple dimensions to obtain multimodal data, and features are extracted from the multimodal data. Gating and attention mechanisms are used to dynamically and adaptively configure the weights of the multimodal features to perform weighted fusion of the multimodal features. This data is then used as input data for a machine learning model to perform multi-dimensional data inference. This enhances the adaptability and reliability of the machine learning model to complex and changing environments, improves the model's decision quality and environmental adaptability, and thus improves the accuracy of data inference. This solves the technical problem in related technologies where machine learning models based on a single modality have low inference accuracy in complex and changing application scenarios.

[0119] Furthermore, the extraction unit includes: a first extraction module, used to extract features from image data using a convolutional neural network to obtain visual features; a second extraction module, used to extract features from audio data using a time-frequency transform network and a temporal convolutional network to obtain audio features; a third extraction module, used to extract features from text data using a natural language processing model to obtain semantic features; and a fourth extraction module, used to extract features from sensor data using a neural network model to obtain environmental features.

[0120] Furthermore, the local inference unit includes: a first configuration module, used to evaluate the quality of image data by combining environmental features through the gating mechanism of the inference model, and to configure visual weight values ​​for visual features based on the quality of the image data; a second configuration module, used to evaluate the quality of audio data by combining environmental features through the gating mechanism of the inference model, and to configure audio weight values ​​for audio features based on the quality of the audio data; and a third configuration module, used to evaluate the quality of text data by combining the gating mechanism of the inference model, and to configure semantic weight values ​​for semantic features based on the quality evaluation results of the text data, wherein the quality evaluation of text data includes at least one of the following: evaluating the information completeness of the data, and evaluating whether the text data contains preset information.

[0121] Furthermore, the inference unit also includes: a first calculation module for calculating the correlation score between multimodal features through an attention mechanism; a second calculation module for calculating the importance score of multimodal data to the inference task through an attention mechanism combined with historical inference records; and a first configuration module for configuring attention weight values ​​for multimodal features based on the correlation score and importance score of the multimodal features.

[0122] Furthermore, the inference unit also includes: a first projection module, used to perform modal alignment projection on different types of multimodal features to obtain multimodal feature projections; and a first fusion module, used to perform weighted fusion calculation on the multimodal feature projections based on the weight values ​​of each multimodal feature to obtain a fused feature vector.

[0123] Furthermore, the data inference device based on dynamic weighting of multimodal features also includes: a third calculation module, used to collect the operating status and computing power budget of the end-side device, and calculate the weight threshold based on the operating status and computing power budget, wherein the end-side device refers to the device that performs the inference task; and a first elimination module, used to eliminate multimodal features with weight values ​​less than the weight threshold.

[0124] Furthermore, the data inference device based on multimodal feature dynamic weighting also includes: a fourth calculation module, used to calculate the confidence value of the intermediate inference result at each intermediate layer of the inference model during the inference process, wherein the intermediate layer represents each hidden layer in the inference model; and a first output module, used to stop the inference when the confidence value reaches a preset confidence threshold, and output the final inference result in advance based on the intermediate inference result, wherein the confidence threshold is updated in real time based on the operating status and computing power budget of the edge device.

[0125] The aforementioned data inference device based on dynamic weighting of multimodal features may also include a processor and a memory. The aforementioned acquisition unit 51, extraction unit 52, inference unit 53, etc., are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0126] The processor described above contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and inference can be performed on the data by adjusting kernel parameters.

[0127] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0128] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute any of the above-described data reasoning methods based on multimodal feature dynamic weighting.

[0129] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement any of the above-described data reasoning methods based on multimodal feature dynamic weighting.

[0130] According to another aspect of the present invention, a computer program product is also provided, the computer program product including a computer program, wherein when the computer program is executed by a processor, it implements any of the above-described data reasoning methods based on dynamic weighting of multimodal features.

[0131] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having the following method steps: collecting multimodal data and preprocessing the multimodal data, wherein the types of multimodal data include at least one of the following: image data, audio data, text data, and sensor data; performing feature extraction using a feature extraction model corresponding to different types of multimodal data to obtain multimodal features, and performing vector encoding on the multimodal features to obtain multimodal feature vectors; inputting the multimodal feature vectors into an inference model, configuring weight values ​​for different types of multimodal features through the gating mechanism and attention mechanism of the inference model, performing feature fusion based on the weight values ​​to obtain fused feature vectors, and performing inference based on the fused feature vectors to output the inference result, wherein the inference model is a pre-built machine learning model for performing data inference.

[0132] Figure 6 This is a hardware structure block diagram of an electronic device (or mobile device) that executes a data reasoning method based on dynamic weighting of multimodal features according to an embodiment of the present invention. Figure 6 As shown, an electronic device may include one or more processors ( Figure 6 The processor, denoted by 602a, 602b, ..., 602n, can include, but is not limited to, a processing device such as a microprocessor (MCU) or a programmable logic device (FPGA), and a memory 604 for storing data. In addition, it may include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports in the I / O interface), a network interface, a keyboard, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 6 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device may also include components that are more... Figure 6 The more or fewer components shown, or having the same Figure 6 The different configurations shown.

[0133] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0134] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0136] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0137] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0138] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0139] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data reasoning method based on dynamic weighting of multi-modal features, characterized in that, The method comprises the following steps: Collecting multi-modal data and pre-processing the multi-modal data, wherein the types of the multi-modal data include at least one of the following: image data, audio data, text data, sensor data; Performing feature extraction using feature extraction models corresponding to different types of multi-modal data to obtain multi-modal features, and performing vector encoding on the multi-modal features to obtain multi-modal feature vectors; Inputting the multi-modal feature vectors into an inference model, configuring weight values for different types of multi-modal features through the gating mechanism and attention mechanism of the inference model, performing feature fusion based on the weight values to obtain a fusion feature vector, and performing inference based on the fusion feature vector to output an inference result, wherein the inference model is a machine learning model pre-constructed for data inference.

2. The method of claim 1, wherein, The step of performing feature extraction using feature extraction models corresponding to different types of multi-modal data includes: Performing feature extraction on the image data using a convolutional neural network to obtain visual features; Performing feature extraction on the audio data using a time-frequency transform network and a time sequence convolution network to obtain audio features; Performing feature extraction on the text data using a natural language processing model to obtain semantic features; Performing feature extraction on the sensor data using a neural network model to obtain environmental features.

3. The method of claim 2, wherein, The step of configuring weight values for different types of multi-modal features through the gating mechanism of the inference model includes: Evaluating the quality of the image data in combination with the environmental features through the gating mechanism of the inference model, and configuring a visual weight value for the visual features based on the quality of the image data; Evaluating the quality of the audio data in combination with the environmental features through the gating mechanism of the inference model, and configuring an audio weight value for the audio features based on the quality of the audio data; Performing quality evaluation on the text data through the gating mechanism of the inference model, and configuring a semantic weight value for the semantic features based on the quality evaluation result of the text data, wherein the quality evaluation on the text data includes at least one of the following: evaluating the information completeness of the data, and evaluating whether the text data contains preset information.

4. The method of claim 2, wherein, The step of configuring weight values for different types of multi-modal features through the attention mechanism of the inference model includes: Calculating the correlation scores between the multi-modal features through the attention mechanism; Calculating the importance scores of the multi-modal data for the inference task in combination with historical inference records through the attention mechanism; Configuring attention weight values for the multi-modal features based on the correlation scores and the importance scores of the multi-modal features.

5. The method of claim 1, wherein, The step of performing feature fusion based on the weight values to obtain a fusion feature vector includes: Performing modality alignment projection on different types of multi-modal features to obtain multi-modal feature projections; Performing weighted fusion calculation on the multi-modal feature projections based on the weight values of the multi-modal features to obtain the fusion feature vector.

6. The method of claim 1, wherein, Before performing feature fusion based on the weight values, the method further comprises the following steps: Collect an operating state and a computing power budget of an end-side device, and calculate a weight threshold based on the operating state and the computing power budget, wherein the end-side device represents a device performing an inference task; Prune the multi-modal feature whose weight value is less than the weight threshold.

7. The method of claim 6, wherein, Further comprising: In the process of performing inference, calculate a confidence value of an intermediate inference result in each intermediate layer of the inference model, wherein the intermediate layer represents a respective hidden layer in the inference model; When the confidence value reaches a preset confidence threshold, stop inference and output a final inference result in advance based on the intermediate inference result, wherein the confidence threshold is updated in real time based on the operating state and the computing power budget of the end-side device.

8. A data reasoning device based on dynamic weighting of multi-modal features, characterized in that, Comprise: A collection unit configured to collect multi-modal data and pre-process the multi-modal data, wherein the types of the multi-modal data comprise at least one of the following: image data, audio data, text data, and sensor data; An extraction unit configured to perform feature extraction using feature extraction models corresponding to different types of multi-modal data to obtain multi-modal features, and perform vector encoding on the multi-modal features to obtain multi-modal feature vectors; An inference unit configured to input the multi-modal feature vectors to an inference model, configure weight values for different types of the multi-modal features through a gating mechanism and an attention mechanism of the inference model, perform feature fusion based on the weight values to obtain a fusion feature vector, and perform inference based on the fusion feature vector to output an inference result, wherein the inference model is a machine learning model pre-constructed for data inference.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium comprises a stored computer program, wherein when the computer program is running, the computer-readable storage medium controls the device where the computer-readable storage medium is located to perform the data inference method based on dynamic weighting of multi-modal features according to any one of claims 1 to 7.

10. An electronic device, comprising: Comprise one or more processors and a memory, the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data inference method based on dynamic weighting of multi-modal features according to any one of claims 1 to 7.

Images