A recognition processing method and device and electronic equipment

Abstract

This application discloses a recognition processing method, apparatus, and electronic device. The method includes: obtaining multimodal data, wherein the multimodal data includes modal data acquired by different types of sensors; extracting features from each modal data in the multimodal data to obtain corresponding multiple modal features; determining dynamic fusion weights corresponding to each modal feature based on the correlation between different modalities, wherein the dynamic fusion weights are updated as the reached modal set changes, wherein the reached modal set is the set of modal features for which feature extraction has been completed within the current time window; performing a weighted summation of each modal feature based on the dynamic fusion weights of each modal feature to obtain a fused feature vector; and using a first model to perform recognition processing based on the fused feature vector to obtain a recognition result.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a recognition processing method, apparatus, and electronic device. Background Technology

[0002] Against the backdrop of the rapid development of artificial intelligence and sensor technology, human behavior recognition has become an important research direction in the field of human-computer interaction and has been widely applied in scenarios such as smart homes, virtual reality, office assistance, and autonomous driving.

[0003] Traditional human-computer interaction methods are mainly limited to contact-based or conventional action input, voice input, and other simple non-contact input methods such as mouse, keyboard, touch screen, etc. They are usually based on data collected by a single sensor for behavior recognition, which has problems such as low recognition accuracy and poor robustness. Summary of the Invention

[0004] In view of the above, this application provides the following technical solution:

[0005] The first aspect of this application provides an identification processing method, the method comprising:

[0006] Obtain multimodal data, which includes modal data acquired by different types of sensors;

[0007] Feature extraction is performed on each modality data in the multimodal data to obtain corresponding multiple modal features;

[0008] The dynamic fusion weights corresponding to each modality feature are determined by combining the correlation between different modalities. The dynamic fusion weights are updated as the set of reached modalities changes. The set of reached modalities is the set of modal features that have completed feature extraction within the current time window.

[0009] The dynamic fusion weights of each modality feature are used to sum the modality features in a weighted manner to obtain the fusion feature vector.

[0010] The first model is used to perform recognition processing based on the fused feature vector to obtain the recognition result.

[0011] In one possible implementation, the correlation between the different modalities includes complementary correlation and mutually exclusive correlation. The complementary correlation indicates that both modal features conform to the target behavior, and the mutually exclusive correlation indicates that one of the two modal features conforms to the target behavior, while the other does not.

[0012] If two modal features are complementary and correlated, increase the dynamic fusion weight of the two modal features;

[0013] If two modal features are mutually exclusive, increase the dynamic fusion weight of the modal feature with higher confidence and decrease the dynamic fusion weight of the modal feature with lower confidence.

[0014] In one possible implementation, the dynamic fusion weights corresponding to each modality feature are determined by combining the correlation relationships between different modalities, including:

[0015] Scene features are obtained by performing scene recognition based on various modal features;

[0016] Quality assessment is performed on each modal feature to obtain the corresponding assessment results. The quality assessment is determined at least based on the signal data of the acquired signals corresponding to each modal feature. The signal data includes at least the signal-to-noise ratio and the temporal consistency used to characterize the overlap of sampling periods of different signals.

[0017] Based on the evaluation results, the confidence level of the modal features corresponding to each modality is determined;

[0018] The dynamic fusion weights corresponding to each modality feature are determined by combining the correlation between different modalities, the scene features, and the confidence level.

[0019] In one possible implementation, scene recognition is performed based on various modal features, including:

[0020] Obtain reference data, which includes at least one of the following: current modality features, historical modality features, and user behavior data;

[0021] Based on the reference data, the current scenario type is analyzed, which includes changing scenarios, non-changing scenarios, and fluctuating scenarios.

[0022] The method for determining the changing scenario is as follows: calculate the difference data between the current modal features and the historical modal features; if the difference data is greater than a first set threshold, it is determined to be a changing scenario.

[0023] The method for determining the unchanging scenario is as follows: if the difference data is less than the second preset threshold, it is determined to be an unchanging scenario;

[0024] The method for determining the fluctuation scenario is as follows: if the difference data is between the first set threshold and the second set threshold, then the user behavior data is combined to further determine whether it is a change scenario.

[0025] In one possible implementation, the dynamic fusion weights corresponding to each modality feature are determined by combining the correlation relationships between different modalities, including:

[0026] When the user's facial orientation deviates from the sensor detection area based on visual modality, the dynamic fusion weight of the visual modality is reduced;

[0027] When the auditory modality detects that the current frequency domain signal does not belong to the preset human voice frequency band, the dynamic fusion weight of the auditory modality is reduced.

[0028] In one possible implementation, obtaining multimodal data includes:

[0029] Modal data of different modes are obtained sequentially within the current time window, and the modal data includes at least one of the following: air pressure data, humidity data, and carbon dioxide concentration data;

[0030] Feature extraction is performed on each modality of the multimodal data, including:

[0031] Once the current time window is activated, during the continuous or intermittent reception of modal data, feature extraction is performed on the acquired dynamic data. Whenever a new modal data has completed feature extraction, its corresponding modal features are added to the arrived modal set.

[0032] The dynamic fusion weights corresponding to each modality feature are determined by combining the correlation relationships between different modalities; based on the dynamic fusion weights of each modality feature, the modality features are weighted and summed to obtain the fusion feature vector, including:

[0033] After the current time window is activated, the temporary dynamic fusion weights corresponding to the feature vectors of each mode in the currently reached mode set are determined by combining the correlation between different modes;

[0034] According to the temporary dynamic fusion weights, the modal features in the currently arrived modal set are weighted and summed to obtain the cumulative fusion feature vector;

[0035] When subsequent modal feature vectors arrive, the above update process is repeated until all modal feature vectors arrive, and the last accumulated fused feature vector is used as the fused feature vector.

[0036] One possible implementation also includes:

[0037] If the number of modal features that have completed feature extraction within the current time window reaches a preset threshold, and the confidence levels of all modal features that have completed feature extraction are higher than the preset confidence threshold, the cumulative fusion feature vector of the modal features that have completed feature extraction within the current time window will be determined as the final fusion feature vector, and no further feature extraction or dynamic fusion weight determination will be performed on subsequent arriving modal data.

[0038] One possible implementation also includes:

[0039] Dynamically adjusting the length of the current time window based on the modal features includes:

[0040] If the signal-to-noise ratio of the acquired signal corresponding to the modal feature exceeds the first threshold, the length of the current time window will be reduced.

[0041] If the signal-to-noise ratio of the acquired signal corresponding to the modal feature is lower than the second threshold, the length of the current time window is increased.

[0042] Wherein, the first threshold is greater than the second threshold.

[0043] A second aspect of this application provides an identification processing apparatus, comprising:

[0044] The data acquisition module is used to acquire multimodal data, which includes modal data acquired by different types of sensors;

[0045] The feature extraction module is used to extract features from each modality in the multimodal data to obtain corresponding multiple modal features;

[0046] The weight determination module is used to determine the dynamic fusion weights corresponding to each modality feature by combining the correlation between different modalities. The dynamic fusion weights are updated as the set of reached modalities changes. The set of reached modalities is the set of modality features that have completed feature extraction within the current time window.

[0047] The feature fusion module is used to perform weighted summation of each modal feature based on the dynamic fusion weights of each modal feature to obtain a fused feature vector;

[0048] The recognition processing module is used to perform recognition processing based on the fused feature vector using the first model to obtain the recognition result.

[0049] A third aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0050] The memory is used to store computer programs;

[0051] The processor executes the computer program to enable the electronic device to: acquire multimodal data, which includes modal data acquired by different types of sensors; extract features from each modal data in the multimodal data to obtain corresponding multiple modal features; determine dynamic fusion weights corresponding to each modal feature based on the correlation between different modalities, wherein the dynamic fusion weights are updated as the reached modal set changes, and the reached modal set is the set of modal features for which feature extraction has been completed within the current time window; perform weighted summation of each modal feature based on the dynamic fusion weights of each modal feature to obtain a fused feature vector; and perform recognition processing using a first model based on the fused feature vector to obtain a recognition result. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0053] Figure 1 This is a flowchart of an identification processing method disclosed in an embodiment of this application;

[0054] Figure 2 This is a flowchart of the process for determining dynamic fusion weights disclosed in an embodiment of this application;

[0055] Figure 3 This is a schematic diagram illustrating the implementation of obtaining dynamic fusion weights as disclosed in an embodiment of this application;

[0056] Figure 4 This is a flowchart of the modal feature fusion process disclosed in an embodiment of this application;

[0057] Figure 5 This is a schematic diagram of the cross-modal attention calculation process disclosed in an embodiment of this application;

[0058] Figure 6 This is a schematic diagram of the structure of an identification processing device disclosed in an embodiment of this application;

[0059] Figure 7 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. Detailed Implementation

[0060] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] The embodiments of this application can be applied to electronic devices. This application does not limit the product form of the electronic device, which may include but is not limited to smartphones, tablets, wearable devices, personal computers (PCs), netbooks, etc., and can be selected according to application requirements.

[0062] Figure 1 This is a flowchart illustrating an identification processing method disclosed in an embodiment of this application. See also... Figure 1 As shown, the identification processing method may include:

[0063] Step 101: Obtain multimodal data, which includes modal data acquired by different types of sensors.

[0064] In this embodiment, the system can synchronously or asynchronously collect data from the environment using multiple different types of sensors. The multimodal data includes modal data acquired by different types of sensors. For example, in a smart home interaction scenario, the device may simultaneously be equipped with an image sensor (visual modality), a microphone array (auditory modality), a barometric pressure sensor, a humidity sensor, and a carbon dioxide sensor (environmental modality). These sensors operate at their respective inherent frequencies; for example, a camera may output image frames at a frame rate of 30fps, a microphone may collect audio streams at a sampling rate of 16kHz, and an environmental sensor may report data at a period of 0.5 to 1 second. The system timestamps the data from each sensor using a unified clock source or timestamp service to facilitate subsequent time alignment. The process of acquiring multimodal data can be continuous, forming a multi-parallel data stream.

[0065] The multimodal data can be obtained within the current time window. The size of the time window can be a default value, but during implementation, the length of the default time window can be adjusted as needed, which will be described in the following embodiments.

[0066] The multimodal data described in this article is the foundation for identifying user target behaviors. These target behaviors can be, for example, a user blowing air into a specific area. In the visual modality, this would result in an image of the user pouting when blowing air; in the auditory modality, it would result in the sound of the user blowing air; and in the environmental modality, it would result in a brief and rapid increase in air pressure and humidity in the detection area, as well as an increase in carbon dioxide concentration in the detection area. Examples of these effects will be presented later.

[0067] Step 102: Extract features from each modal data in the multimodal data to obtain corresponding multiple modal features.

[0068] When extracting features from each modality of the multimodal data, the system performs specific feature extraction processing on each input raw data to obtain corresponding modal features. These modal features are feature data representing corresponding modal attributes. Their purpose is to transform high-dimensional, redundant raw data into low-dimensional feature vectors containing semantic information, thereby reducing subsequent computation and improving the model's generalization ability.

[0069] Specifically, for visual modal data, the system first performs face detection and key point localization, then focuses on the mouth region to extract geometric features such as lip curvature and mouth opening, as well as texture features. For example, when a user inputs a blowing motion, the system captures the dynamic change of the lips from a closed state to an outward protrusion. For auditory modal data, the system performs a Fast Fourier Transform (FFT) on the audio signal to obtain frequency domain information, then extracts features such as energy spectrum and zero-crossing rate, and determines the energy changes within the corresponding frequency band of human voice, which typically includes the airflow sound characteristics produced by human blowing. For environmental modal data, such as humidity sensor data, the system directly reads its percentage value; for barometric pressure sensor data, the system calculates its change within a short time window, capturing instantaneous fluctuations in air pressure.

[0070] Step 103: Determine the dynamic fusion weights corresponding to each modality feature by combining the correlation between different modalities. The dynamic fusion weights are updated as the set of reached modalities changes. The set of reached modalities is the set of modal features that have completed feature extraction within the current time window.

[0071] The correlation between different modalities characterizes the correlation between them, and the dynamic fusion weights corresponding to each modal feature are determined by combining the correlations between different modal features. Unlike the static or fixed weight allocation strategy in traditional methods, the dynamic fusion weights in this invention are updated as the set of arrived modalities changes.

[0072] In this step, the system first utilizes a cross-modal attention mechanism to calculate the correlation between features from different modalities. This mechanism measures the temporal and semantic association between features from one modality and features from another. For example, if the visual modality detects a user pouting while the auditory modality detects a sound of airflow, then there is a strong correlation between these two modal features. The system constructs a correlation matrix, where each element C_ij represents the correlation score between features from modality i and features from modality j.

[0073] Based on this correlation matrix, the system can determine the dynamic fusion weights corresponding to each modality feature through a Dynamic Weighting Network (DWN). DWN is a lightweight neural network or rule-based decision module. Its input includes the aforementioned correlation matrix, the confidence scores of each modality (detailed below), and the current scene features. The output of DWN is the weight vector W = [w1, w2, ... wn] for each modality, where wi represents the weight of the i-th modality, and the sum of all weights is 1. Whenever a new modality feature is added to the set of arriving modalities, DWN recalculates the weights. For example, initially, only visual features are present, with a weight of 1.0; when auditory features arrive and are added to the set, DWN adjusts the visual weight to 0.6 and the auditory weight to 0.4 based on their correlation. This dynamic update mechanism ensures that the weight allocation is always based on the latest and most comprehensive information.

[0074] Step 104: Based on the dynamic fusion weights of each modal feature, perform a weighted summation of each modal feature to obtain the fused feature vector.

[0075] The system uses dynamic fusion weights to weight and sum the modal features to obtain a fused feature vector. Specifically, the system multiplies each modal feature vector by its corresponding dynamic fusion weight, and then sums all the weighted vectors to obtain a unified, low-dimensional feature representation, i.e., the fused feature vector. Transforming heterogeneous feature data into a unified fused feature vector effectively simplifies the input complexity of subsequent recognition models and improves model recognition efficiency.

[0076] Step 105: Use the first model to perform recognition processing based on the fused feature vector to obtain the recognition result.

[0077] The first model is used to perform recognition processing based on the fused feature vector to obtain the recognition result. The first model can be a pre-trained deep learning model, such as a combination of Support Vector Machine (SVM), Random Forest model, Convolutional Neural Network (CNN), and Spatiotemporal Graph Convolutional Network (ST-GCN). This model takes the fused feature vector as input and outputs a probability distribution representing the likelihood that the current behavior belongs to each predefined category. For example, the model might output a 95% probability for "blowing behavior," a 3% probability for "normal breathing," and a 2% probability for "ambient noise." The system sets a decision threshold (e.g., 85%). When the probability of "blowing behavior" exceeds this threshold, it is determined that the user has executed a blowing interaction command, triggering a corresponding device response, such as waking up a voice assistant, turning off an alarm clock, or controlling home appliances.

[0078] The identification and processing method described in this embodiment is based on the dynamic weight allocation of intermodal correlation, which enables the system to adaptively adjust the fusion strategy according to the real-time quality and correlation of each modal data, thereby effectively suppressing the influence of noise and interference, improving the accuracy and robustness of identifying specific behaviors (such as blowing air) in complex environments, and reducing the false trigger rate.

[0079] In one embodiment, the correlation between the different modalities includes complementary correlation and mutually exclusive correlation. The complementary correlation indicates that both modal features conform to the target behavior, and the mutually exclusive correlation indicates that one of the two modal features conforms to the target behavior, while the other does not.

[0080] The complementary correlation indicates that two modal features simultaneously conform to the target behavior. For example, when identifying the target behavior of "blowing air," if the "lip protrusion" feature captured by the visual modality and the "specific frequency airflow sound" feature captured by the auditory modality appear simultaneously and synchronously, then these two features constitute a complementary correlation. This simultaneous occurrence greatly enhances the certainty of behavior determination. Conversely, the mutually exclusive correlation indicates that one of the two modal features conforms to the target behavior, while the other does not. For example, the visual modality shows that the user is facing away from the device, while the auditory modality detects a sound resembling blowing air. These two features are logically conflicting and constitute a mutually exclusive correlation.

[0081] If two modal features are detected to be complementary, the dynamic fusion weight of these two modal features will be increased accordingly.

[0082] For example, in a quiet home environment, when a user blows air towards the device, the system detects a high correlation between visual features (mouth movements) and auditory features (airflow sounds). In this case, DWN will increase the visual weight from the default 0.4 to 0.55, the auditory weight from 0.4 to 0.45, and decrease the humidity weight, which may be slightly affected by airflow but whose changes are not significant, from 0.2 to 0.0. In this way, it strengthens the supporting data for the target behavior, making the fused feature vector more accurately reflect the user's intent.

[0083] If two modal features are mutually exclusive, increase the dynamic fusion weight of the modal feature with higher confidence and decrease the dynamic fusion weight of the modal feature with lower confidence.

[0084] For example, in a noisy office environment, the airflow noise from an air conditioner might cause auditory modality misjudgment. In this case, if the visual modality clearly shows that the user is not facing the device (high confidence), while the auditory modality detects a sound similar to blowing air (low confidence due to ambient noise), the system determines that the two are mutually exclusive. DWN will decide based on confidence: significantly increasing the weight of the visual modality (e.g., increasing it to 0.9) and significantly decreasing the weight of the auditory modality (e.g., decreasing it to 0.1). This confidence-based arbitration mechanism effectively avoids false triggering caused by a single unreliable modality, ensuring the correctness of the system's decisions in complex environments.

[0085] This embodiment clarifies the specific implementation of the correlation between different modal features and establishes a dynamic weight adjustment mechanism based on the correlation relationship. In the implementation of the scheme, the mutual verification between multimodal information can be used to improve the fusion weight, and conflict information between modalities can be effectively identified and the fusion weight of the corresponding modality can be reduced. The above mechanism can effectively improve the system's immunity to noise and interference and reduce the false alarm rate.

[0086] Figure 2 This is a flowchart of the process for determining dynamic fusion weights disclosed in an embodiment of this application. Figure 3 This is a schematic diagram illustrating the implementation of obtaining dynamic fusion weights as disclosed in an embodiment of this application. (Combined with...) Figure 2 and Figure 3 As shown, in one embodiment, determining the dynamic fusion weights corresponding to each modality feature by combining the correlation relationships between different modalities may include:

[0087] Step 201: Perform scene recognition based on each modal feature to obtain scene features.

[0088] Scene features describe the current environmental state of the device. For example, the system identifies whether the current scene is a quiet home, a noisy office, or a windy outdoor scene by analyzing the statistical distribution of historical modal features (such as average noise level, average light intensity, and average temperature and humidity baseline). Scene features can be used to guide weight allocation strategies. For example, in a home scene, the fusion weight of auditory modalities can be set higher; while in a noisy office scene, the fusion weight of environmental modalities (such as humidity and CO2) can be higher to compensate for the shortcomings of auditory modalities.

[0089] Step 202: Perform quality assessment on each modal feature and obtain the corresponding assessment results.

[0090] The quality assessment is determined at least based on the signal data of the acquired signals corresponding to each modal feature. The signal data includes at least the signal-to-noise ratio and the temporal consistency used to characterize the overlap of sampling periods of different signals.

[0091] Signal-to-noise ratio (SNR) is a metric for signal purity; a high SNR indicates a strong signal and weak noise. Temporal consistency characterizes the degree of overlap between the sampling periods of different signals. Because different sensors have different sampling frequencies, their data at a given point in time actually cover different time spans. Temporal consistency is measured by calculating the intersection of different modal data along the time axis. If the overlap between two modal data points is large within a time window, the temporal consistency is high, indicating that they describe events within the same time period, and thus have high reliability. Conversely, if the overlap is small or there is significant misalignment, the temporal consistency is low, indicating potential data delays or mismatches, and thus low reliability.

[0092] Step 203: Determine the confidence level of the modal features corresponding to each modality based on the evaluation results.

[0093] In this implementation, confidence level is a quantitative indicator that combines signal-to-noise ratio (SNR) and timing consistency, typically represented by a value between 0 and 1. SNR and confidence level are positively correlated, as is the degree of timing consistency. When determining confidence level, both SNR and timing consistency can be considered, and the confidence level can be determined based on a defined algorithm.

[0094] Step 204: Determine the dynamic fusion weights corresponding to each modality feature by combining the correlation between different modalities, the scene features, and the confidence level.

[0095] When performing dynamic weight allocation, Dynamic Wavelength Analysis (DWN) comprehensively considers the correlation between different modalities, the scene features, and the confidence level. For example, in an office scene (scene features), the visual modality detects a user's mouth movement, but its confidence level decreases as the user turns around (confidence level); simultaneously, the humidity sensor detects a tiny pulse (as an environmental modality, correlated with the visual modality corresponding to the user's mouth movement), and its confidence level is high (high signal-to-noise ratio, high temporal consistency). After integrating this information, DWN generates a final weight allocation scheme: reducing the visual weight, increasing the humidity weight, and potentially introducing other modal data (such as auditory modality) as auxiliary verification.

[0096] Step 204 can be implemented by a small neural network that can be trained on a large amount of labeled data to learn how to optimally fuse multimodal information in different scenarios.

[0097] This embodiment enriches the basis for dynamic weight allocation by introducing scene recognition and quality assessment, making the weight allocation more reasonable. By comprehensively considering correlation, scene and confidence in weight calculation, the system can select reliable features for fusion in various complex and variable scenes, thereby ensuring the accuracy of the recognition results and the robustness of the system from the source.

[0098] In one embodiment, scene recognition based on various modal features may include: obtaining reference data, which includes at least one of the following: current modal features, historical modal features, and user behavior data; and analyzing the current scene type based on the reference data, which includes changing scenes, non-changing scenes, and fluctuating scenes.

[0099] The method for determining the changing scenario is as follows: calculate the difference data between the current modal features and the historical modal features. If the difference data is greater than a first set threshold, it is determined to be a changing scenario. The method for determining the non-changing scenario is as follows: if the difference data is less than a second set threshold, it is determined to be a non-changing scenario. The method for determining the fluctuating scenario is as follows: if the difference data is between the first set threshold and the second set threshold, then further determine whether it is a changing scenario by combining user behavior data.

[0100] Current modal features refer to the latest extracted data; historical modal features refer to data accumulated over a period of time (such as the past few minutes or hours) that represents the normal state of the environment; user behavior data refers to information about the user's location and posture obtained through other sensors (such as infrared sensors, ultrasonic sensors, or visual tracking algorithms). For example, the system continuously records the baseline value of ambient humidity as historical features, and simultaneously records whether the user is in front of the device as user behavior data.

[0101] The changing scenario, for example, is in a home environment where the ambient humidity is typically stable at around 45%. If the current humidity reading suddenly jumps to 65%, and the calculated difference exceeds a first preset threshold, the system determines it to be a changing scenario. This changing scenario represents a sudden event occurring in the environment, such as someone turning on a humidifier or a user blowing air into it.

[0102] In a non-changing scenario, for example, if the current humidity reading is 51%, which is very close to the historical average of 50% and below the second preset threshold, the system determines it to be a non-changing scenario. The second preset threshold is less than the first preset threshold.

[0103] The fluctuating scenario, for example, involves humidity readings fluctuating between 52% and 55%, with the difference falling between a first and a second set threshold. In this case, modal characteristics alone are insufficient for an accurate judgment. The system incorporates user behavior data to aid in the judgment. If user behavior data shows that the user is currently seated in front of the device and has a recent interaction history, the system may tend to classify it as a changing scenario; if the user is not present, the system may classify it as normal environmental fluctuation.

[0104] This embodiment details the specific implementation of scene recognition, which enables the solution to accurately identify the causes of changes in environmental state and maintain recognition accuracy in complex environments. Thus, the solution will neither produce false alarms due to oversensitivity nor miss key interactive behaviors due to slow response.

[0105] In one embodiment, determining the dynamic fusion weights corresponding to each modal feature by combining the correlation between different modalities may include: reducing the dynamic fusion weight of the visual modality when the user's facial orientation deviates from the sensor detection area based on the visual modality; and reducing the dynamic fusion weight of the auditory modality when the current frequency domain signal does not belong to the preset human voice frequency band based on the auditory modality.

[0106] In implementation, the system can determine whether a user is facing the device using visual modality data. For example, a facial landmark detection algorithm calculates the angle between the user's nose tip and the camera's optical axis. If this angle exceeds a preset facing threshold (e.g., ±30 degrees), it is determined that the user's face is facing out of alignment. In this case, the information about mouth movements provided by the visual modality becomes unreliable or completely invalid because the user may be looking elsewhere. At this point, DWN will significantly reduce the weight of the visual modality.

[0107] The sound of human breath is concentrated within a specific frequency range. The system can pre-set the relevant frequency range of a person's voice / breathing sound as a reference. During operation, the auditory feature extraction module analyzes the spectrum of the audio signal in real time. If it finds that the signal energy is mainly concentrated outside the preset frequency range, such as power frequency noise in the 50Hz-60Hz range, or high-frequency noise above 1000Hz (such as keyboard typing), or a stable low-frequency airflow sound (such as air conditioner noise), the system determines that the signal does not belong to the frequency range related to the target behavior. At this time, DWN will control and reduce the weight of the auditory modality. For example, when a continuous, stable low-frequency air conditioner noise is detected, the auditory weight is reduced from 0.4 to 0.1, thereby effectively filtering out most of the interference from ambient sound.

[0108] This embodiment provides two specific and efficient weight reduction strategies. Through these two strategies, the system can promptly reduce the corresponding fusion weights after detecting unreliable modal features, thereby eliminating or reducing the interference of unreliable information.

[0109] In one embodiment, obtaining multimodal data includes acquiring modal data of different modes sequentially within a current time window. The modal data includes at least one of the following: air pressure data, humidity data, and carbon dioxide concentration data. The time window has a default length, for example, 1 second. Due to differences in sampling rates and response speeds of different sensors, data enters this time window at different times. For example, the first frame of data from a camera enters at time t=0ms, the first audio packet from a microphone enters at t=5ms, while the first reading from a humidity sensor may not enter until t=200ms. The system does not wait for all data to arrive; instead, it processes data immediately as soon as it enters the time window.

[0110] Feature extraction is performed on each modality in the multimodal data, including: after the current time window is started, during the continuous or intermittent reception of modal data, feature extraction is performed on the dynamic data that has been obtained, and whenever a new modality is extracted, its corresponding modal features are added to the arrived modality set.

[0111] The above process is parallel and asynchronous. For example, at t=0ms, the system receives visual data, immediately performs feature extraction to obtain visual feature V1, and adds it to the set of arrived modalities, at which point the set is {V1}; at t=5ms, the system receives auditory data, performs feature extraction to obtain auditory feature S1, and adds it to the set, making the set {V1, S1}. This mechanism ensures that computing resources are promptly allocated to the arrived data, rather than idle and waiting, thus shortening the overall output time of the recognition processing results.

[0112] Based on the above, the process of determining the dynamic fusion weights and obtaining the fusion feature vectors can be found in [reference needed]. Figure 4 Combining Figure 4 As shown, the dynamic fusion weights corresponding to each modality feature are determined by combining the correlation relationships between different modalities; based on the dynamic fusion weights of each modality feature, the modality features are weighted and summed to obtain the fusion feature vector, which may include:

[0113] Step 401: After the current time window is started, determine the temporary dynamic fusion weights corresponding to the feature vectors of each modality in the currently reached modality set by combining the correlation between different modalities.

[0114] Step 402: According to the temporary dynamic fusion weight, the modal features in the currently arrived modal set are weighted and summed to obtain the cumulative fusion feature vector.

[0115] Step 403: When subsequent modal feature vectors arrive, repeat the above update process until all modal feature vectors arrive, and use the last obtained cumulative fusion feature vector as the fusion feature vector.

[0116] The update process refers to the incremental update closed loop triggered by the arrival of the new modal feature vector, which is the process of incremental update of the already arrived modal set → incremental recalculation of the correlation between modalities → redistribution of the dynamic fusion weights of the entire set → generation of a new cumulative fusion vector.

[0117] Referring back to the example above, when the set is {V1}, the system determines the visual weight to be 1.0, and the fusion vector is V1 itself. When the set is updated to {V1, S1}, the system recalculates the weights, for example, a visual weight of 0.6 and an auditory weight of 0.4, and then calculates the first cumulative fusion vector F1 = 0.6V1 + 0.4S1. Subsequently, when the humidity feature H1 arrives at t=200ms, the set becomes {V1, S1, H1}, and the system again calculates the fusion weights for each modality feature, for example, 0.5, 0.3, 0.2, and calculates the second cumulative fusion vector F2 = 0.5V1 + 0.3S1 + 0.2H1. This process repeats continuously, generating a new cumulative fusion vector each time new data arrives. This continues until the time window ends or all expected modality data have arrived, at which point the final cumulative fusion vector F_n becomes the final fusion feature vector for that time window. This incremental fusion method allows the system to generate preliminary recognition results at an earlier possible time.

[0118] The incremental recalculation of intermodal relationships, that is, determining the correlation between features of different modes, can be calculated and determined based on cross-modal attention mechanisms. Figure 5 This is a schematic diagram of the cross-modal attention calculation process disclosed in the embodiments of this application, which can be combined with... Figure 5 Understand the process of obtaining joint features (i.e., fused feature vectors).

[0119] The incremental fusion mechanism proposed in this embodiment fundamentally changes the traditional serial waiting mode of multimodal fusion. The scheme establishes a dynamically growing set of arrived modalities and performs real-time fusion weight calculation and fusion for each new modal data in the set. It can then quickly identify the data based on the fusion results. Furthermore, the scheme utilizes the time difference in the arrival of multimodal data, distributing the computation evenly within a time window, avoiding the long delays caused by performing a large amount of computation at the last minute.

[0120] In one embodiment, the identification processing method may further include: if the number of modal features that have completed feature extraction within the current time window reaches a preset number threshold, and the confidence of the modal features that have completed feature extraction is higher than a preset confidence threshold, the cumulative fusion feature vector of the modal features that have completed feature extraction within the current time window is determined as the final fusion feature vector, and no further feature extraction or dynamic fusion weight determination is performed on the modal data that arrives later.

[0121] The preset quantity threshold can be determined according to the system design. For example, if the system has a total of 5 modalities, the preset quantity threshold can be set to 3, meaning that data from 3 modalities is sufficient for the representation application. The preset confidence threshold is configurable, for example, 0.85. The system continuously monitors the above two conditions within each time window. For example, in an ideal home nighttime scenario, the camera, microphone, and barometer all quickly return high-quality data with confidence levels of 0.95, 0.92, and 0.94, respectively, and the quantity has reached 3. Then, the cumulative fusion feature vector obtained from these three modalities can be used as the final fusion feature vector input to the first model to obtain the recognition result, without waiting for other modal data, thus triggering the early termination of the subsequent process of determining fusion weights and performing feature fusion.

[0122] In other words, once the above two conditions are met, the system determines that the existing information is sufficient and reliable, and there is no need to wait for the remaining modal data (such as the slow-responding humidity or CO2 sensor data). The system will immediately send the latest accumulated fusion vector as the final result to the recognition model. At the same time, the system will send a control signal to pause or cancel the feature extraction and weight calculation tasks for subsequent data that have not yet arrived.

[0123] The scheme described in this embodiment allows the system to autonomously terminate the subsequent waiting modal data, determine the fusion weights, and perform feature fusion processes when the information is reliable and sufficient. This design further reduces the average response latency of the system and can also reduce the processor load and the overall power consumption of the system.

[0124] In one embodiment, the identification processing method may further include a step of dynamically adjusting the length of the current time window based on the modal features. Specifically, it may include: if the signal-to-noise ratio of the acquired signal corresponding to the modal feature exceeds a first threshold, controlling the length of the current time window to decrease; if the signal-to-noise ratio of the acquired signal corresponding to the modal feature is lower than a second threshold, controlling the length of the current time window to increase; wherein the first threshold is greater than the second threshold.

[0125] The first and second thresholds are preset empirical values. For example, the first threshold can be set to 30dB and the second threshold can be set to 15dB.

[0126] In high signal-to-noise ratio (SNR) scenarios (e.g., SNR > 30dB), the signal is clear and the noise is weak. In such cases, a shorter time window (e.g., 100ms-200ms) is sufficient to capture the complete characteristics of the behavior, such as a brief breath. Furthermore, the confidence level of modal features is higher in high SNR scenarios, eliminating the need to wait for all modal data to participate in recognition. Therefore, the time window can be appropriately reduced. Reducing the time window can decrease the system's response latency and decrease the amount of data processed per operation.

[0127] In low signal-to-noise ratio (SNR) scenarios (e.g., SNR < 15dB), the signal is not clear enough and the noise is significant. In such cases, a longer time window (e.g., 500ms-800ms) is needed for temporal smoothing and statistical averaging to extract effective signal features from the noise. Furthermore, the confidence level of modal features is relatively low in low SNR scenarios, requiring the integration of more modal data for identification. Therefore, the time window can be appropriately increased. While increasing the time window increases the latency of a single processing iteration, it significantly improves the accuracy of identification and avoids misjudgments due to insufficient data in high-noise environments.

[0128] This embodiment dynamically adjusts the time window length, enabling the system to have better environmental adaptability. The solution transforms a fixed system parameter into a variable that can be adaptively optimized according to the environmental noise level, ensuring that the system can provide the best user experience in various real-world scenarios and meet the needs of different scenarios (such as low-latency-priority interactive scenarios and robustness-priority complex environments).

[0129] In other implementations, before feature extraction from each modality in the multimodal data, the process may further include preprocessing the multimodal data. The preprocessing may include at least one of the following: denoising, noise reduction, time synchronization, and normalization.

[0130] The above preprocessing can filter out useless or interfering data in the modal data, providing a reliable data foundation for subsequent feature extraction and helping to improve the accuracy of the final recognition results.

[0131] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0132] The methods described in the above-disclosed embodiments of this application are detailed in terms of the methods. The methods of this application can be implemented by various forms of apparatus. Therefore, this application also discloses an apparatus. Specific embodiments are given below for detailed description.

[0133] Figure 6 This is a schematic diagram of the structure of an identification processing device disclosed in an embodiment of this application. See also... Figure 6 As shown, the identification processing device 60 may include:

[0134] The data acquisition module 601 is used to acquire multimodal data, which includes modal data acquired by different types of sensors.

[0135] The feature extraction module 602 is used to extract features from each modality in the multimodal data to obtain corresponding multiple modal features.

[0136] The weight determination module 603 is used to determine the dynamic fusion weights corresponding to each modality feature by combining the correlation between different modalities. The dynamic fusion weights are updated as the set of reached modalities changes. The set of reached modalities is the set of modality features that have completed feature extraction within the current time window.

[0137] The feature fusion module 604 is used to perform weighted summation of each modal feature based on the dynamic fusion weight of each modal feature to obtain the fused feature vector.

[0138] The recognition processing module 605 is used to perform recognition processing based on the fused feature vector using the first model to obtain the recognition result.

[0139] The recognition and processing device described in this embodiment is based on dynamic weight allocation of intermodal correlation, which enables the system to adaptively adjust the fusion strategy according to the real-time quality and correlation of each modal data, thereby effectively suppressing the influence of noise and interference, improving the accuracy and robustness of recognizing specific behaviors (such as blowing air) in complex environments, and reducing the false trigger rate.

[0140] For details on the specific implementation of the above-mentioned identification processing device and its various modules, as well as other possible implementations, please refer to the relevant sections of the method embodiments; they will not be repeated here.

[0141] Any of the recognition processing devices described in the above embodiments includes a processor and a memory. The data acquisition module, feature extraction module, weight determination module, feature fusion module, recognition processing module, etc. in the above embodiments are all stored as program modules in the memory, and the processor executes the above program modules stored in the memory to realize the corresponding functions.

[0142] The processor contains a kernel, which retrieves the corresponding program modules from memory. One or more kernels can be configured, and the processing of backtracking data can be achieved by adjusting kernel parameters.

[0143] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0144] In an exemplary embodiment, a computer-readable storage medium is also provided, which can be directly loaded into the internal memory of a computer and contains software code. After being loaded and executed by the computer, the computer program can implement the steps shown in any embodiment of the above-described identification processing method.

[0145] In an exemplary embodiment, a computer program product is also provided, which can be directly loaded into the internal memory of a computer and contains software code. After being loaded and executed by the computer, the computer program can implement the steps shown in any embodiment of the identification processing method described above.

[0146] Furthermore, embodiments of this application provide an electronic device. Figure 7 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. See also... Figure 7 As shown, the electronic device 70 includes at least one processor 701, at least one memory 702 connected to the processor, and a bus 703; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute the above-mentioned identification processing method.

[0147] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0148] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0149] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0150] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A recognition processing method, the method comprising: Obtain multimodal data, which includes modal data acquired by different types of sensors; Feature extraction is performed on each modality data in the multimodal data to obtain corresponding multiple modal features; The dynamic fusion weights corresponding to each modality feature are determined by combining the correlation between different modalities. The dynamic fusion weights are updated as the set of reached modalities changes. The set of reached modalities is the set of modal features that have completed feature extraction within the current time window. The dynamic fusion weights of each modality feature are used to sum the modality features in a weighted manner to obtain the fusion feature vector. The first model is used to perform recognition processing based on the fused feature vector to obtain the recognition result.

2. The identification processing method according to claim 1, wherein the association relationship between the different modalities includes complementary correlation and mutually exclusive correlation, wherein the complementary correlation indicates that both modal features conform to the target behavior, and the mutually exclusive correlation indicates that one of the two modal features conforms to the target behavior, while the other does not conform to the target behavior; If two modal features are complementary and correlated, increase the dynamic fusion weight of the two modal features; If two modal features are mutually exclusive, increase the dynamic fusion weight of the modal feature with higher confidence and decrease the dynamic fusion weight of the modal feature with lower confidence.

3. The identification processing method according to claim 1, comprising determining the dynamic fusion weights corresponding to each modality feature by combining the correlation relationships between different modalities, including: Scene features are obtained by performing scene recognition based on various modal features; Quality assessment is performed on each modal feature to obtain the corresponding assessment results. The quality assessment is determined at least based on the signal data of the acquired signals corresponding to each modal feature. The signal data includes at least the signal-to-noise ratio and the temporal consistency used to characterize the overlap of sampling periods of different signals. Based on the evaluation results, the confidence level of the modal features corresponding to each modality is determined; The dynamic fusion weights corresponding to each modality feature are determined by combining the correlation between different modalities, the scene features, and the confidence level.

4. The recognition processing method according to claim 3, wherein scene recognition is performed based on various modal features, including: Obtain reference data, which includes at least one of the following: current modality features, historical modality features, and user behavior data; Based on the reference data, the current scenario type is analyzed, which includes changing scenarios, non-changing scenarios, and fluctuating scenarios. The method for determining the changing scenario is as follows: calculate the difference data between the current modal features and the historical modal features; if the difference data is greater than a first set threshold, it is determined to be a changing scenario. The method for determining the unchanging scenario is as follows: if the difference data is less than the second preset threshold, it is determined to be an unchanging scenario; The method for determining the fluctuation scenario is as follows: if the difference data is between the first set threshold and the second set threshold, then the user behavior data is combined to further determine whether it is a change scenario.

5. The identification processing method according to claim 1, comprising determining the dynamic fusion weights corresponding to each modality feature by combining the correlation relationships between different modalities, including: When the user's facial orientation deviates from the sensor detection area based on visual modality, the dynamic fusion weight of the visual modality is reduced; When the auditory modality detects that the current frequency domain signal does not belong to the preset human voice frequency band, the dynamic fusion weight of the auditory modality is reduced.

6. The identification processing method according to claim 1, wherein obtaining multimodal data includes: Modal data of different modes are obtained sequentially within the current time window, and the modal data includes at least one of the following: air pressure data, humidity data, and carbon dioxide concentration data; Feature extraction is performed on each modality of the multimodal data, including: Once the current time window is activated, during the continuous or intermittent reception of modal data, feature extraction is performed on the acquired dynamic data. Whenever a new modal data has completed feature extraction, its corresponding modal features are added to the arrived modal set. The dynamic fusion weights corresponding to each modality feature are determined by combining the correlation relationships between different modalities; based on the dynamic fusion weights of each modality feature, the modality features are weighted and summed to obtain the fusion feature vector, including: After the current time window is activated, the temporary dynamic fusion weights corresponding to the feature vectors of each mode in the currently reached mode set are determined by combining the correlation between different modes; According to the temporary dynamic fusion weights, the modal features in the currently arrived modal set are weighted and summed to obtain the cumulative fusion feature vector; When subsequent modal feature vectors arrive, the above update process is repeated until all modal feature vectors arrive, and the last accumulated fused feature vector is used as the fused feature vector.

7. The identification processing method according to claim 6 further includes: If the number of modal features that have completed feature extraction within the current time window reaches a preset threshold, and the confidence levels of all modal features that have completed feature extraction are higher than the preset confidence threshold, the cumulative fusion feature vector of the modal features that have completed feature extraction within the current time window will be determined as the final fusion feature vector, and no further feature extraction or dynamic fusion weight determination will be performed on subsequent arriving modal data.

8. The identification processing method according to claim 6 further includes: Dynamically adjusting the length of the current time window based on the modal features includes: If the signal-to-noise ratio of the acquired signal corresponding to the modal feature exceeds the first threshold, the length of the current time window will be reduced. If the signal-to-noise ratio of the acquired signal corresponding to the modal feature is lower than the second threshold, the length of the current time window is increased. Wherein, the first threshold is greater than the second threshold.

9. An identification processing device, comprising: The data acquisition module is used to acquire multimodal data, which includes modal data acquired by different types of sensors; The feature extraction module is used to extract features from each modality in the multimodal data to obtain corresponding multiple modal features; The weight determination module is used to determine the dynamic fusion weights corresponding to each modality feature by combining the correlation between different modalities. The dynamic fusion weights are updated as the set of reached modalities changes. The set of reached modalities is the set of modality features that have completed feature extraction within the current time window. The feature fusion module is used to perform weighted summation of each modal feature based on the dynamic fusion weights of each modal feature to obtain a fused feature vector; The recognition processing module is used to perform recognition processing based on the fused feature vector using the first model to obtain the recognition result.

10. An electronic device comprising at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program to enable the electronic device to: acquire multimodal data, the multimodal data including modal data acquired by different types of sensors; and extract features from each modal data in the multimodal data to obtain corresponding multiple modal features; The dynamic fusion weights corresponding to each modality feature are determined by combining the correlation between different modalities. The dynamic fusion weights are updated as the set of reached modalities changes. The set of reached modalities is the set of modal features that have completed feature extraction within the current time window. The modal features are weighted and summed based on the dynamic fusion weights of each modality feature to obtain a fusion feature vector. The first model is used to perform recognition processing based on the fusion feature vector to obtain the recognition result.

Images