Feature fusion and engineering processing method of multi-modal data
By using feature distribution difference analysis and nonlinear mapping processing, the problems of information conflict and redundancy in multimodal data monitoring of postoperative laryngeal surgery patients were solved, and efficient multimodal collaborative analysis and early airway abnormality identification were achieved in complex environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HANGZHOU HONGDI DATA TECHNOLOGY CO LTD
- Filing Date
- 2026-06-25
- Publication Date
- 2026-07-31
AI Technical Summary
In the monitoring of anesthesia recovery in patients undergoing postoperative laryngeal surgery, multimodal analysis of image and audio data is prone to intermodal information conflicts and redundant feature superposition due to occlusion, missing frames, environmental noise, and interference from nursing actions, resulting in a low ability to express early airway abnormalities.
Multimodal data is acquired through a preset time window, and image and audio features are extracted using a preset feature extraction network. Feature distribution difference analysis is performed, dynamic weight suppression coefficient is determined, feature dimensionality reduction and nonlinear mapping are performed, cross-modal mutual information gain rate is calculated, and feature weighted fusion is achieved to suppress intermodal information conflict and redundant features.
Despite interference from occlusion, missing frames, environmental noise, and nursing actions, the accuracy and stability of multimodal collaborative analysis were improved, enhancing the ability to identify early airway abnormalities.
Smart Images
Figure CN122490448A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and multimodal data processing technology, specifically to a method for feature fusion and engineering processing of multimodal data. Background Technology
[0002] During the postoperative monitoring of patients recovering from anesthesia after laryngeal surgery, the medical system needs to continuously collect the patient's image and audio data in order to make a comprehensive judgment on respiratory status, chest rise and fall, changes in facial and lip color, and abnormal breath sounds, so as to detect airway abnormalities as early as possible; and the accuracy of the monitoring results largely depends on the collaborative analysis capability between multimodal data.
[0003] In traditional methods, image and audio data are often analyzed independently or simply stitched together before recognition processing. In cases of occlusion, missing frames, environmental noise, or interference from nursing actions, problems such as intermodal information conflict, redundant feature superposition, and dilution of effective abnormal features can easily occur, resulting in a still low ability to express early airway abnormalities. Summary of the Invention
[0004] To address the aforementioned technical problems, this invention provides a method for feature fusion and engineering processing of multimodal data. Specifically, the technical solution of this invention includes:
[0005] The initial multimodal data of the target object is obtained according to a preset time window. The initial multimodal data includes image data collected by an image sensor and audio data collected by an audio acquisition device within the preset time window.
[0006] Based on a preset feature extraction network, feature extraction processing is performed on the initial multimodal data to determine the initial modal feature set. The initial modal feature set includes the first modal feature corresponding to the image data and the second modal feature corresponding to the audio data.
[0007] The initial modal feature set is subjected to feature distribution difference analysis to determine the dynamic weight suppression coefficient;
[0008] Based on the dynamic weight suppression coefficient, the initial modal feature set is subjected to feature dimensionality reduction processing to determine the intermediate modal feature set, wherein the intermediate modal feature set includes the first dimensionality reduction feature corresponding to the first modal feature and the second dimensionality reduction feature corresponding to the second modal feature;
[0009] The intermediate modal feature set is subjected to nonlinear mapping processing to determine the cross-modal mutual information gain rate; based on the cross-modal mutual information gain rate, the intermediate modal feature set is subjected to feature weighted fusion processing to determine the target multimodal fusion feature.
[0010] Preferably, based on a preset feature extraction network, feature extraction processing is performed on the initial multimodal data to determine the initial modal feature set, including: extracting first modal features from image data through a convolutional neural network in the preset feature extraction network, and extracting second modal features from audio data through an acoustic feature extraction model in the preset feature extraction network.
[0011] Preferably, the initial modal feature set is subjected to feature distribution difference analysis to determine the dynamic weight suppression coefficient, including: calculating the feature distribution difference of the initial modal feature set, determining the degree of feature distribution difference and the corresponding conflict feature subset; and determining the dynamic weight suppression coefficient based on the conflict feature subset.
[0012] Preferably, determining the dynamic weight suppression coefficient based on the conflict feature subset includes: if the degree of difference corresponding to the conflict feature subset is greater than a preset conflict threshold, then the dynamic weight suppression coefficient is obtained by performing a nonlinear activation function calculation on the conflict feature subset; if the degree of difference corresponding to the conflict feature subset is less than or equal to the preset conflict threshold, then the dynamic weight suppression coefficient is obtained by performing a linear mapping calculation on the conflict feature subset.
[0013] Preferably, the initial modal feature set is subjected to feature dimensionality reduction processing based on the dynamic weight suppression coefficient to determine the intermediate modal feature set, including: performing channel-level weight attenuation processing on the first modal feature based on the dynamic weight suppression coefficient to determine the first dimensionality-reduced feature; performing channel-level weight attenuation processing on the second modal feature based on the dynamic weight suppression coefficient to determine the second dimensionality-reduced feature; and concatenating the first dimensionality-reduced feature and the second dimensionality-reduced feature to determine the intermediate modal feature set.
[0014] Preferably, the intermediate modal feature set is subjected to nonlinear mapping processing to determine the cross-modal mutual information gain rate, including: performing fully connected layer dimension mapping processing on the intermediate modal feature set to determine the cross-modal complementarity features; and determining the cross-modal mutual information gain rate based on the cross-modal complementarity features and the intermediate modal feature set.
[0015] Preferably, the intermediate modal feature set is subjected to feature weighting and fusion processing based on the cross-modal mutual information gain rate to determine the target multimodal fusion feature, including: performing feature reconstruction processing on the intermediate modal feature set based on the cross-modal mutual information gain rate to determine the weighted feature set; and performing merging processing on the weighted feature set to determine the target multimodal fusion feature.
[0016] Preferably, the weighted feature set is merged to determine the target multimodal fusion feature, including: dividing the weighted feature set into image sub-vectors and audio sub-vectors according to the modality source; calculating the inner product of the image sub-vectors and audio sub-vectors to determine the orthogonality index of the feature space; and merging the weighted feature set according to the orthogonality index of the feature space to determine the target multimodal fusion feature.
[0017] Preferably, the target multimodal fusion feature is determined by merging the weighted feature set according to the orthogonality index of the feature space, including: multiplying the orthogonality index of the feature space and the weighted feature set to determine the temporary fusion result; and performing global average pooling and normalization on the temporary fusion result to obtain the target multimodal fusion feature.
[0018] Preferably, the method further includes: obtaining preset labels; calculating model loss values based on the target multimodal fusion features and preset labels; and updating the parameters of the preset feature extraction network through backpropagation based on the model loss values.
[0019] Compared with the prior art, the present invention has the following beneficial effects:
[0020] This invention first extracts first-modal and second-modal features from image and audio data in the postoperative laryngeal surgery patient anesthesia recovery monitoring. Then, it combines feature distribution difference calculation, conflict feature subset identification, dynamic weight suppression coefficient generation, channel-level weight attenuation, nonlinear mapping to obtain cross-modal mutual information gain rate, and weighted merging processing based on feature space orthogonality index. This provides a feature fusion scheme that can achieve multimodal collaborative analysis under conditions of occlusion, missing frames, environmental noise, and nursing action interference. Compared with traditional independent analysis or simple splicing methods, it can more effectively suppress intermodal information conflict and redundant feature superposition, reduce the dilution of effective abnormal features, and improve the ability to express early airway abnormality risks, fusion stability, and monitoring and identification accuracy. Attached Figure Description
[0021] The present invention will be further explained below with reference to the accompanying drawings and embodiments:
[0022] Figure 1 This is a flowchart illustrating the feature fusion and engineering processing method for multimodal data provided in an embodiment of the present invention. Detailed Implementation
[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.
[0024] Methods for feature fusion and engineering processing of multimodal data, including:
[0025] The initial multimodal data of the target object is obtained according to a preset time window. The initial multimodal data includes image data collected by an image sensor and audio data collected by an audio acquisition device within the preset time window.
[0026] Based on a pre-defined feature extraction network, feature extraction processing is performed on the initial multimodal data to determine the initial modal feature set, which includes the first modal feature corresponding to the image data and the second modal feature corresponding to the audio data; feature distribution difference analysis is performed on the initial modal feature set to determine the dynamic weight suppression coefficient;
[0027] Based on the dynamic weight suppression coefficient, the initial modal feature set is subjected to feature dimensionality reduction processing to determine the intermediate modal feature set, wherein the intermediate modal feature set includes the first dimensionality reduction feature corresponding to the first modal feature and the second dimensionality reduction feature corresponding to the second modal feature; the intermediate modal feature set is subjected to nonlinear mapping processing to determine the cross-modal mutual information gain rate;
[0028] Based on the cross-modal mutual information gain rate, the intermediate modal feature set is subjected to feature weighting and fusion processing to determine the target multimodal fusion features.
[0029] This embodiment provides a feature fusion processing mechanism for multimodal data, such as... Figure 1 As shown; specifically, this mechanism is deployed in the post-anesthesia recovery room monitoring system for patients undergoing laryngeal surgery. The target subjects are patients entering the recovery observation period. The image sensor is fixedly installed above the bed to continuously acquire images of the patient's facial and lip areas, chest rise and fall areas, and apparent neck movements. The audio acquisition device is arranged above the head of the bed to acquire respiratory sounds, wheezing sounds, cough sounds, and short vocal fragments. The system slices the raw data with a preset time window, for example, forming an image sequence fragment and a corresponding audio fragment every 2 seconds to ensure that the two modalities can be aligned and processed on the time axis.
[0030] Specifically, the system first performs feature extraction on the initial multimodal data; on the image side, it can output a set of first modal features representing changes in lip color, nasal flaring, chest rise and fall amplitude, and abnormal head and neck swaying; on the audio side, it can output a set of second modal features representing the dominant frequency energy, bandwidth distribution, short-time zero-crossing rate, expiratory and inspiratory rhythm, and high-frequency wheezing components.
[0031] For ease of explanation, a simplified numerical example can be constructed: within a certain time window, a four-dimensional vector is obtained from the image. The higher values of 0.8 and 0.7 correspond to the trend of cyanosis of the lips and abnormal chest undulations, respectively; a four-dimensional vector is obtained from the audio side. , where 0.9 corresponds to high-frequency wheezing and 0.6 corresponds to respiratory cycle instability; together they constitute the initial modal feature set;
[0032] The system performs feature distribution difference analysis on the initial modal feature set to identify which modal components complement each other and which components conflict under the current task. The feature distribution difference analysis can be understood as: based on the loss regression calculation information recorded in advance during the training phase, calling the sensitivity change direction and change magnitude of each modality and each channel to risk judgment, and comparing the feature responses within the current time window.
[0033] Using the example vector above, if the third dimension of image features shows a positive increase as risk rises, and the second dimension of audio features also shows a positive increase, then they can be considered to support each other in the same direction. If the second dimension of image features tends to decrease in risk prediction, while the second dimension of audio features significantly increases in risk prediction, then it indicates that there is intermodal expression bias or feature conflict within this time window. Based on this, the system generates a dynamic weight suppression coefficient, for example... This indicates the degree to which each channel should be suppressed or preserved, with a larger value indicating that the corresponding channel needs stronger suppression in subsequent processing;
[0034] Based on this, the system performs feature dimensionality reduction on the initial modal feature set according to the dynamic weight suppression coefficient; specifically, channel compression, weight decay, or selective retention methods can be used to reduce the expression intensity of high-conflict and high-redundancy channels; in the above example, the image vector can be transformed into The audio vector can be transformed into Then, the retained core components are combined into an intermediate modal feature set. Through the above processing, the redundant channels that might have compressed the effective discrimination space are shrunk, and the processing burden of subsequent mapping is reduced.
[0035] Furthermore, the system performs nonlinear mapping on the intermediate modal feature set to estimate the cross-modal mutual information gain rate; the processing here is not a simple concatenation, but rather a re-encoding of cross-modal associations such as abnormal lip color in the image and enhanced wheezing frequency band in the audio into a more complementary combined representation through at least one layer of nonlinear transformation.
[0036] Using the example values again, if the discrimination of the intermediate feature against hypoxia risk is 0.55 before mapping and increases to 0.73 after mapping, this increase can be converted into the mutual information gain rate of the current time window. Based on this, the system performs weighted fusion of the intermediate modal features to obtain the target multimodal fusion feature, such as a unified risk expression vector. This is for use in subsequent graded early warning models;
[0037] Furthermore, if a certain time window experiences image frame loss, severe occlusion, or audio burst noise exceeding a preset noise threshold, the system does not directly terminate the fusion process. Instead, it first sets a confidence metric for the missing modality. If the image confidence level is lower than the preset value, the system reduces the weight of the image channel in dynamic suppression, allowing the audio side to dominate the current window. If device collision noise occurs on the audio side, the system can mark the window as an abnormal window and use the smoothing result of the previous normal window as a transitional input. If both modalities have low confidence levels, the system outputs a pending review status and triggers resampling, instead of issuing a high confidence warning conclusion.
[0038] At the 35-minute recovery point after a thyroidectomy, the camera observed that the chest wall fluctuation amplitude was greater than a preset amplitude threshold, but the difference in lip color change was less than a preset color change threshold. At the same time, the microphone detected mild inspiratory high-frequency wheezing. Traditional simple stitching methods are prone to diluting audio abnormalities due to insufficient image changes. This implementation method first identifies the weight distribution relationship between audio dominance and local image supplementation through gradient analysis, then attenuates conflicting channels, and strengthens the combined pattern of wheezing and chest wall compensation through nonlinear mapping. Finally, it provides a medium-to-high risk warning before clinically visible cyanosis appears.
[0039] To further clarify, to maintain consistency in terminology throughout the text, "camera" is used as an instantiation of the aforementioned image sensor, and "microphone" is used as an instantiation of the aforementioned audio acquisition device; neither indicates a new hardware type. Accordingly, in subsequent embodiments, "image-side branch" and "image-side channel" refer to the first modal feature processing path formed by the image data acquired by the image sensor; "audio-side branch" and "audio-side channel" refer to the second modal feature processing path formed by the audio data acquired by the audio acquisition device.
[0040] Furthermore, to avoid the problem of unclear objects in the dynamic weighted suppression coefficient between different modalities, the dynamic weighted suppression coefficient in this embodiment is preferably understood as a set of coefficients generated according to feature groups and then allocated according to channel affiliation; that is, the system can first obtain group-level suppression results for feature groups such as respiratory load-related groups, rhythm abnormality-related groups, and interference motion-related groups, and then apply them to the corresponding channels in the first modal feature and the second modal feature according to the preset channel affiliation relationship; when written as a vector of the same length for ease of illustration, it only represents the channel-level schematic result after backfilling under this window, and does not require that the image features and audio features be synonymously corresponding in the original dimensional position;
[0041] Furthermore, in engineering implementation, the above-mentioned feature distribution difference analysis, feature dimensionality reduction, nonlinear mapping, and feature weighted fusion can be executed by the same processor in sequence according to the time window, or by multiple functional modules working together. However, the data objects between each step remain consistent, that is, the input of the next processing step comes from the output of the previous processing step, so as to ensure that the processing link is clear and traceable.
[0042] The purpose of this step is to reorganize the truly relevant information in different modalities in a real monitoring scenario to suppress redundant expression, reduce feature space compression effect, and improve the ability to express early airway abnormalities.
[0043] Furthermore, based on a preset feature extraction network, feature extraction processing is performed on the initial multimodal data to determine the initial modal feature set, including: extracting the first modal features from the image data through a convolutional neural network in the preset feature extraction network, and extracting the second modal features from the audio data through an acoustic feature extraction model in the preset feature extraction network.
[0044] This embodiment provides a separate feature extraction step for heterogeneous image and audio signals. Specifically, in the aforementioned anesthesia recovery room scenario, the physical properties of image data and audio data have heterogeneous features. If they are processed directly with a unified extractor, it is easy for the spatial texture information of the image and the temporal rhythm information of the audio to interfere with each other. Therefore, a convolutional neural network is used to process the image data and an acoustic feature extraction model is used to process the audio data.
[0045] Specifically, the image input can be a series of cropped images of the patient's facial and lip regions and chest region; for example, the system can superimpose 8 frames of images extracted within 2 seconds into a temporal image block, with each frame first uniformly scaled to [size missing]. The pixels are then fed into a convolutional neural network. The front layer of the network extracts edges, colors, and local motion textures, the middle layer extracts local combined features such as nasal flaring, chest contour undulation, and color changes around the mouth, and the back layer outputs the first modality features used for risk assessment. For example, a certain image patch can output a feature vector of length 6, where the first two dimensions reflect color and brightness shifts, the middle two dimensions reflect chest periodic motion, and the last two dimensions reflect abnormal posture.
[0046] The audio input can be a 2-second audio clip time-aligned with the aforementioned image patch. The system first performs frame segmentation and window function processing on the audio clip, then converts it into a spectrogram, Mel frequency cepstral coefficient sequence, or other acoustic representation, before feeding it into an acoustic feature extraction model. This model can extract second-modal features such as sharp inspiratory components, expiratory tail components, intermittent cough bursts, and periodic rhythm disturbances. For example, audio clips within the same time window can be represented as a single... The time-frequency matrix of the acoustic model is compressed into a vector of length 6, where higher dimensional values correspond to a significant enhancement of high-frequency abnormal breathing sounds.
[0047] If only image extraction is used, the thoracic features are prone to geometric inconsistencies or drift when the patient's head turns, the bed sheet obscures the view, or the lighting changes. If only audio extraction is used, abnormal breathing sounds are easily masked when there are conversations between patients in the same bed, equipment alarms, and footsteps in the environment. Therefore, configuring the image convolutional network and the acoustic extraction model separately can obtain more stable initial modal features in their respective most suitable representation spaces, laying the foundation for subsequent conflict recognition and fusion.
[0048] As an anomaly tolerance mechanism, if image region detection fails, for example, if a patient is briefly occluded by a caregiver, the system can fall back to the previous valid cropping box or use full-frame reduced-precision extraction; if the proportion of silence in an audio segment is greater than a preset silence ratio threshold, causing the acoustic model output to be close to all zeros, the system retains a low-activity marker in that window to avoid misidentifying it as normal breathing; if the output dimensions of the two extraction branches are inconsistent, a unified mapping layer is added before entering subsequent processing to convert both branches to the agreed dimension range.
[0049] When a patient awakens and groans briefly but the changes in facial expression are below a preset action threshold, the convolutional neural network may primarily capture the accelerated rise and fall of the chest, while the acoustic model captures high-frequency blocking sounds and intermittent vocalizations. By splitting the extraction, the two modalities can retain their most sensitive abnormal cues without being prematurely mixed by the unified encoding process.
[0050] The purpose of this step is to enable different modalities to complete initial encoding in their appropriate network structures, thereby improving feature quality from the source and reducing mutual contamination of heterogeneous signals.
[0051] Furthermore, the initial modal feature set is subjected to feature distribution difference analysis to determine the dynamic weight suppression coefficient, including: calculating the feature distribution difference of the initial modal feature set, determining the degree of feature distribution difference and the corresponding conflict feature subset; and determining the dynamic weight suppression coefficient based on the conflict feature subset.
[0052] This embodiment provides a step for dynamically identifying modal conflicts. Specifically, in the aforementioned scenario, if image features and audio features are directly concatenated into a classifier after obtaining them, one modality may be amplified while the other modality exhibits a biased expression. For example, a patient turning over causes a sudden increase in motion on the image side, but the audio side remains relatively stable. In this case, the motion information in the image does not necessarily represent an increased risk. To avoid such expression bias, this embodiment first calculates the difference in feature distribution between the two modalities and then extracts a subset of conflicting features from it.
[0053] Specifically, the system can compare image vectors and audio vectors within the same time window by dimension; assuming the image side output... Audio side output The system does not require a one-to-one correspondence between the dimensions. Instead, it uses a preset mapping relationship to merge features such as respiratory load-related groups, abnormal rhythm-related groups, and motion-related groups before comparing the degree of difference.
[0054] If a group has strong image representation and weak audio representation, and the group contributes to the results in opposite directions in recent training backhaul, then the group is marked as a conflict feature subset. For example, if the difference value of the second group reaches 0.7, while the average difference of the preset stable group is only 0.2, then the second group can enter the conflict candidate.
[0055] After determining the subset of conflicting features, the system generates dynamic weight suppression coefficients based on this subset. This can be understood as meaning that the stronger the conflict and the more unstable the channel, the more weight attenuation is needed subsequently. For example, the difference values are obtained for each of the four sets of features. Afterwards, the system can map out the inhibition coefficient. Therefore, the second group will be compressed more significantly during subsequent dimensionality reduction, while the third group can be basically retained due to its smaller differences.
[0056] Furthermore, to avoid image errors in subsequent steps... Dimensions and Audio The ambiguity of inconsistent semantics being directly shared by the same suppression value is addressed by optimizing the dynamic weight suppression coefficients, which are generated according to feature groups and then backfilled into their respective modal channels. Used merely as an illustrative marker for channel number or dimension index, indicating the first element in each modal feature vector. The position does not imply that the image side and the audio side have the same natural physiological semantics at that position;
[0057] In other words, the system first calculates the group-level inhibition coefficient for the above-mentioned groups such as respiratory load-related group, abnormal rhythm-related group, and motion-interference-related group, and then assigns the group-level inhibition coefficient to the corresponding channels in the image branch and audio branch according to the preset channel assignment table.
[0058] If a certain group corresponds to channels 1 and 3 on the image side and channels 2 and 5 on the audio side, then the suppression coefficient of that group is applied to these channels respectively, instead of requiring the two modalities to be mechanically aligned in vector position; this maintains the processing logic of determining the dynamic weight suppression coefficient based on the conflict feature subset in the embodiment, and also makes the subsequent channel-level attenuation have a clear target;
[0059] Furthermore, the feature distribution difference analysis and processing and feature distribution difference calculation are related by screening followed by verification: the system can first locate conflict candidate groups based on the distribution differences in the current window, and then combine the gradient direction consistency information recorded in the training or verification phase to determine whether the candidate group truly constitutes a task conflict; if a group has a numerical difference greater than the preset difference threshold, but its gradient direction is consistent for a long time, it indicates that the two modes may only have different response amplitudes rather than mutual deviations. At this time, the priority of this group entering the conflict feature subset can be reduced; thus, normal modal complementarity can be avoided as a conflict.
[0060] Without this step, a simple fusion model often misjudges short-term non-pathological changes as risk peaks when encountering complex situations such as patient coughing, nursing turning, and bedside instrument movement. This step separates and processes the parts with changes exceeding the preset threshold and inconsistent feature gradient directions, making the subsequent fusion more targeted.
[0061] Under certain boundary conditions, if the difference values of each group within a certain time window are all below the lower limit, it means that the two modes are roughly consistent in this window. At this time, the system can output a low suppression configuration to avoid excessively weakening the effective information. If the difference values of all groups are greater than the preset difference upper limit threshold, it may mean that the time alignment has failed or the acquisition link is abnormal. The system can first perform time recalibration or mark this window as not participating in parameter updates.
[0062] At the moment when the nurse helped the patient up and patted his back, the chest and head movements on the image side increased significantly, but no sustained increase in wheezing was observed on the audio side. After system calculation, it was found that the differences in the movement-related group were prominent, but the consistency of the respiratory load-related group was still relatively high. Therefore, the former was included in the conflict feature subset and its inhibition coefficient was increased to avoid misjudging nursing actions as a deterioration of the condition.
[0063] The purpose of this step is to identify the incoherent parts between modes before fusion, so as to provide a basis for subsequent differentiated attenuation processing.
[0064] Furthermore, based on the conflict feature subset, the dynamic weight suppression coefficient is determined, including: if the degree of difference corresponding to the conflict feature subset is greater than a preset conflict threshold, then the dynamic weight suppression coefficient is obtained by performing a nonlinear activation function calculation on the conflict feature subset; if the degree of difference corresponding to the conflict feature subset is less than or equal to the preset conflict threshold, then the dynamic weight suppression coefficient is obtained by performing a linear mapping calculation on the conflict feature subset.
[0065] This embodiment provides a step for generating suppression coefficients in a hierarchical manner. Specifically, although a subset of conflict features has been identified in the previous embodiment, if the same mapping rule is used for all conflicts, two types of problems may still occur: excessive suppression for minor conflicts and insufficient suppression for severe conflicts. Therefore, this embodiment adopts nonlinear activation or linear mapping depending on whether the degree of difference exceeds a preset conflict threshold.
[0066] Specifically, a preset conflict threshold is set to 0.6. This threshold can be adaptively set based on the mean of the difference distribution of redundant feature samples annotated by doctors in historical monitoring data, or it can be calibrated within the range of 0.5 to 0.8 based on human experience. If the difference value of a conflict group is 0.75, it indicates that the group shows obvious opposition or strong redundancy in the two modalities. The system uses non-linear activation processing to rapidly increase the inhibition coefficient. For example, a difference value of 0.75 can be converted to 0.92 after activation, and a difference value of 0.90 can be converted to 0.98 after activation. This can more decisively suppress the subsequent impact of problematic channels. Conversely, if the difference value of a conflict group is 0.35, it only indicates a slight inconsistency. The system uses linear mapping to keep the inhibition coefficient growing smoothly, for example, mapping to 0.35 or 0.4, to avoid prematurely weakening channels that still have discriminative value.
[0067] The above processing is based on the following mechanism: since the previous step only determined the location of the conflict, but did not determine whether the conflict should be mildly suppressed or significantly attenuated; this embodiment uses threshold grading for quantitative compensation to match the suppression intensity with the severity of the conflict;
[0068] Furthermore, as a fault-tolerant measure, if the difference value is exactly equal to the threshold, the system can be classified into the linear mapping branch to avoid sudden jumps in the suppression coefficient at the threshold boundary; if the difference value fluctuates drastically in a short time due to noise, the difference values of multiple consecutive windows can be averaged and then compared with the threshold; if the suppression coefficient is too close to 1 after nonlinear activation, causing subsequent features to be almost completely suppressed, the system can set an upper limit, such as 0.95, to retain a minimum information pathway.
[0069] During the mid-recovery phase of the patient, the microphone recorded a continuous, sharp wheezing sound, while the image side did not show obvious changes in the chest wall due to the patient being covered by a blanket. At this time, the difference value of the breathing abnormality-related group may exceed the threshold. The system rapidly increases the corresponding inhibition coefficient through nonlinear activation, prioritizing the suppression of unreliable conflict channels. Conversely, when the patient slightly turns their head, causing changes in the brightness of the perioral area, only a slight inconsistency appears between the image and audio. The system uses linear mapping and only performs smooth attenuation.
[0070] The purpose of this step is to enable the system to employ different suppression strategies for strong and weak conflicts, thereby improving the stability and fine-grained adaptability of the fusion process.
[0071] Furthermore, based on the dynamic weight suppression coefficient, the initial modal feature set is subjected to feature dimensionality reduction processing to determine the intermediate modal feature set, including: based on the dynamic weight suppression coefficient, the first modal feature is subjected to channel-level weight attenuation processing to determine the first dimensionality-reduced feature; based on the dynamic weight suppression coefficient, the second modal feature is subjected to channel-level weight attenuation processing to determine the second dimensionality-reduced feature; and the first and second dimensionality-reduced features are concatenated to determine the intermediate modal feature set.
[0072] This embodiment provides a channel-level weight attenuation and dimensionality reduction step; specifically, the suppression coefficient alone is not enough to improve the fusion quality because the conflict information still remains in the original vector; therefore, this embodiment directly applies the suppression coefficient to each channel on the image side and the audio side, attenuates the high-conflict channels, and outputs the first dimensionality reduction feature and the second dimensionality reduction feature.
[0073] Specifically, continuing with the aforementioned example data structure, let's assume the image vector is... The audio vector is The dynamic weight suppression coefficient is The system can implement channel-level weight attenuation by using a retention coefficient = 1 - suppression coefficient, thus the image side becomes... The audio side becomes The system can further remove channels below a preset retention threshold, such as removing elements below 0.05, thereby obtaining a more compact first dimensionality reduction feature. Second dimensionality reduction features Then, these are spliced together to form an intermediate modal feature set;
[0074] The reason for using channel-level processing is that in postoperative monitoring, different channels often correspond to different physiological phenomena. If compression is only performed uniformly at the overall vector level, the effective channels and redundant channels related to the task will be weakened together, making it difficult to reflect the requirement that the attenuation should be targeted at the conflict site. Channel-level attenuation enables the system to accurately suppress local interferences such as occlusion motion in the image and instrument collision sounds in the audio.
[0075] As an anomaly tolerance mechanism, if the number of remaining effective channels for a certain modality is too small after attenuation, such as only one channel remaining on the image side, the system can activate the minimum dimension retention mechanism to replenish the unretained channel with the highest contribution from the original features, thus preventing the modality from completely losing its feature representation in the current window; if the total number of dimensions after concatenating two modalities exceeds the design limit of the subsequent mapping module, the system can retain the first few dimensions according to their contribution, and the remaining dimensions are assigned to a low-priority cache and do not participate in the main fusion of this window;
[0076] When the head shaking amplitude exceeds the preset displacement threshold shortly after the patient coughs, the posture channel in the image and the blast noise channel in the audio will be given a high suppression coefficient. After channel-level weight attenuation, the continuous chest wall compensation channel and high-frequency wheezing channel that are truly related to airway obstruction are preserved, forming a more compact set of intermediate modal features.
[0077] The purpose of this step is to translate the previously identified conflict information into executable feature shrinking actions, thereby reducing redundant dimensions and alleviating the feature dimension redundancy problem in subsequent fusion.
[0078] Furthermore, a nonlinear mapping process is performed on the intermediate modal feature set to determine the cross-modal mutual information gain rate, including: performing a fully connected layer dimension mapping process on the intermediate modal feature set to determine the cross-modal complementarity features; and determining the cross-modal mutual information gain rate based on the cross-modal complementarity features and the intermediate modal feature set.
[0079] This embodiment provides a cross-modal complementarity enhancement step. Specifically, the previous step has already compressed conflicting and redundant features, but if we only focus on the stitching result, the image and audio are still just parallel entities and the complementarity is not truly explored. Therefore, this embodiment performs dimensional mapping on the intermediate modal feature set through a fully connected layer to generate cross-modal complementary features, and then calculates the cross-modal mutual information gain rate.
[0080] Specifically, assuming the intermediate modal feature set is The system inputs this vector into a fully connected layer and outputs a set of lower-dimensional but more combinatorial vectors, for example... Among them, the first dimension can mainly represent the combined pattern of chest wall compensation and high-frequency wheezing, the second dimension can mainly represent the non-dangerous combination of mild cough and postural changes, and the third dimension can mainly represent the synergistic abnormality of rhythm disorder and local cyanosis. Unlike simple splicing, this step re-encodes the effective combinations that were originally scattered in different modalities into cross-modal complementary features through parameter learning.
[0081] Based on this, the system compares the effectiveness of information before and after mapping to determine the cross-modal mutual information gain rate. This can be calculated using the discrimination improvement magnitude, the proportion of discrimination improvement, or the difference in information content before and after fusion, all related to the target task. For example, if the discrimination score of the intermediate modal feature for the current risk level is 0.58 before mapping, and the discrimination score of the complementary feature is 0.74 after mapping, then the mutual information gain rate can be recorded as the proportion corresponding to the improvement. This value serves as the basis for weighted fusion; the higher the gain rate, the more worthwhile the cross-modal joint relationship within that window should be strengthened.
[0082] Furthermore, to avoid the concept of mutual information gain rate being too abstract, in this embodiment it can be understood as an engineering gain index, that is, used to characterize the effective information improvement of cross-modal joint representation after nonlinear mapping relative to intermediate modal features before mapping on the target task, rather than being limited to strict information theory closed-form solutions;
[0083] Preferably, the cross-modal mutual information gain rate The discrimination score of cross-modal complementary features on the same discriminant classifier head can be calculated. Baseline discrimination scores of intermediate modality feature sets on the current task The difference between them, divided by the baseline discrimination score. With a smoothing term to prevent the denominator from being too small We can obtain it by summing the results, that is... ,in, This is a minimal positive real constant used to prevent the denominator from being zero; thus, when Higher than hour, A positive value and a larger value indicate a more significant gain; when and When approaching, A value close to 0 indicates that the current window lacks significant new cross-modal information.
[0084] Furthermore, baseline discrimination score With discrimination score It is advisable to obtain the results under the same scoring caliber; for example, the same discriminant classification head can be used to output risk discrimination scores for the features before and after mapping, respectively, or the same verification rule can be used to calculate the degree of matching between the two and the preset label; this can ensure that the gain rate reflects the increment brought about by the mapping, rather than the apparent fluctuations brought about by the change in the scoring system.
[0085] Furthermore, the fully connected layer dimension mapping process can include four steps: input dimension alignment, linear transformation, activation processing, and output dimension constraint. Input dimension alignment ensures that the intermediate modal feature sets of different windows meet a uniform length requirement; linear transformation establishes cross-modal combination relationships; activation processing enhances nonlinear expressive power; and output dimension constraint ensures that cross-modal complementary features meet the input specifications for subsequent mutual information gain rate calculation. With this structure, the input, intermediate processing quantities, and output objects of the nonlinear mapping process are more clearly defined.
[0086] Without this step, even if dimensionality reduction has been performed previously, only compressed parallel features can be obtained, which is still difficult to solve the real cross-modal collaborative representation problem. Therefore, this embodiment performs joint mapping encoding after completing conflict suppression, so that the information flow first reduces the impact of feature conflict and then forms complementary representation.
[0087] Furthermore, if the complementary features output by the fully connected layer do not bring significant improvement compared to the intermediate modal features of the input, for example, if the gain rate is lower than the preset lower limit, the system can consider that the current window lacks a stable cross-modal complementary relationship. In this case, no additional enhancement is performed, but a conservative fusion strategy is adopted. If a certain dimension is saturated and close to a fixed value for a long time after mapping, a sparse constraint can be applied to that dimension or its parameters can be reset during the training period to prevent invalid channels from occupying the representation space.
[0088] If the baseline discrimination score S1 is lower than the preset baseline threshold, causing the gain rate to interfere with the judgment of fluctuations in amplitude below the preset noise threshold, then the boost amount can be directly used. and proportional value The joint threshold is used for judgment to avoid extreme windows being misjudged as high-gain windows;
[0089] When patients experience early inspiratory obstruction, the mild perioral cyanosis in the image and the faint wheezing in the audio are not particularly prominent when viewed individually. However, after mapping by a fully connected layer, these two types of signals are combined into a highly sensitive complementary feature, and its corresponding gain rate is significantly higher than that of the single-modal discrimination result, thereby improving the early recognition capability.
[0090] The purpose of this step is to further reconstruct the intermediate features that have already undergone dimensionality reduction and deduplication into cross-modal collaborative representations, and to quantify the effective information growth brought about by this collaboration.
[0091] Furthermore, based on the cross-modal mutual information gain rate, the intermediate modal feature set is subjected to feature weighting fusion processing to determine the target multimodal fusion features, including: based on the cross-modal mutual information gain rate, the intermediate modal feature set is subjected to feature reconstruction processing to determine the weighted feature set; the weighted feature set is then merged to determine the target multimodal fusion features.
[0092] This embodiment provides a feature weighted fusion step based on mutual information gain rate. Specifically, after obtaining the cross-modal mutual information gain rate in the previous step, the system already knows how high the joint value of the image and audio in the current window is. However, if fixed weight merging is still used, the dynamic differences between windows cannot be reflected. Therefore, this embodiment reconstructs the intermediate modal feature set according to the mutual information gain rate to obtain a weighted feature set, and then merges them to form the target multimodal fusion feature.
[0093] Specifically, assuming a window has a high mutual information gain ratio, corresponding to a value of 0.8, it indicates that the joint relationship between the image and audio is crucial for risk identification. The system can amplify the dimensions of intermediate modality features that are closely related to complementary features, while keeping the weakly related dimensions unchanged or slightly weakening them. For example, the intermediate modality feature set... After reconstruction, it can become If the gain of another window is only 0.2, the reconstruction amplitude is smaller, indicating that the window should be biased towards robust and conservative fusion.
[0094] This process addresses a potential shortcoming of the previous stage: although the complementary relationship has been calculated, if the relationship is not written back into the original intermediate features, the final output may still be diluted by several low-value dimensions; through feature reconstruction, high-value collaborative dimensions can obtain higher expression intensity.
[0095] As an anomaly tolerance mechanism, if the mutual information gain ratio is greater than the preset upper gain threshold but the corresponding window confidence is lower than the preset confidence threshold (e.g., the image is blurry and the audio is affected by instantaneous spike interference), the system can combine the confidence threshold to prune the gain ratio to avoid erroneous enhancement. If the gain ratio is lower than the preset lower gain threshold, no features will be forcibly amplified. Instead, an approximately balanced reconstruction strategy will be adopted to reduce the risk of feature distortion. If the mutual information gain ratio is greater than or equal to the preset lower gain threshold and less than or equal to the preset upper gain threshold, the intermediate modal feature set will be reconstructed normally according to the cross-modal mutual information gain ratio.
[0096] As patients gradually develop signs of laryngeal edema, the neck compensatory movements in the image and the inspiratory sharp sounds in the audio significantly co-occur in the same window, resulting in an increased mutual information gain rate. After the system reconstructs the features based on this gain rate, the abnormal respiratory combination pattern occupies a higher weight in the output, thereby making the subsequent warning results closer to the true pathological state.
[0097] The purpose of this step is to transform cross-modal joint information from evaluation results into actual fusion weights, so that the output features truly reflect the collaborative value of the current window.
[0098] Furthermore, the weighted feature set is merged to determine the target multimodal fusion feature, including: dividing the weighted feature set into image sub-vectors and audio sub-vectors according to the modality source; calculating the inner product of the image sub-vectors and audio sub-vectors to determine the orthogonality index of the feature space; and merging the weighted feature set according to the orthogonality index of the feature space to determine the target multimodal fusion feature.
[0099] This embodiment provides a step for merging using the orthogonality index of feature space. Specifically, after the weighted feature set is generated, there may still be a residual problem: although some dimensions are amplified, they express similar content, resulting in increased redundancy in the feature space after fusion. To further alleviate redundant coupling, this embodiment first calculates the vector inner product between different sub-vectors in the weighted feature set to obtain the orthogonality index of feature space, and then completes the merging based on it.
[0100] Specifically, assuming the weighted feature set can be divided into image sub-vectors With audio subvectors The system calculates the inner product of the two components. If the result is lower than the set inner product threshold, it indicates that the two components are independent in the current feature space and their complementarity meets the preset requirements. If the result is greater than or equal to the inner product threshold, it indicates that there is expression overlap.
[0101] For ease of explanation, the result can be normalized to obtain the orthogonality index. For example, if the orthogonality index after inner product reduction is 0.82, it means that the image and audio are related but also have high independence in this window, making them suitable for more thorough merging. If the index is only 0.35, it means that multiple dimensions are highly similar, and the repetitive components should be appropriately suppressed during merging.
[0102] Furthermore, to avoid the misunderstanding that a larger inner product necessarily equates to a higher orthogonality index, the optimal orthogonality index is not obtained by directly taking the original inner product value. Instead, the lengths of the two sub-vectors are first normalized, and then the similarity is obtained by reverse mapping. In other words, the system first calculates the normalized similarity. The similarity A value closer to 1 indicates a higher similarity and lower orthogonality between the two subvectors; the orthogonality index... Set as similarity Monotonically opposite quantities, for example, using a relational expression or The orthogonality index A value closer to 1 indicates greater orthogonality and less redundancy. Therefore, the engineering implications of a smaller inner product indicating higher orthogonality and a larger inner product indicating lower orthogonality are established, and subsequent merging steps can directly use this orthogonality index. As a basis for determining the strength of the retained complementary information;
[0103] Furthermore, in actual deployment, image sub-vectors and audio sub-vectors can be principal sub-vectors after the weighted feature set is divided according to modality, or multiple sub-vector pairs after being divided according to preset feature groups. If there are multiple sub-vector pairs, the system can calculate their respective orthogonality indices separately, and then take the average, weighted average, or confidence-weighted result as the final orthogonality index of the current window to avoid overall judgment bias caused by accidental distortion of a single sub-vector pair.
[0104] By first calculating the orthogonality index, the lack of redundancy constraints in the preceding processing can be compensated; because the mutual information gain rate emphasizes the joint effectiveness, but does not directly constrain whether there is high redundancy after jointing; this embodiment further checks the separability of the fusion space, thereby avoiding the repeated superposition of multiple anomaly descriptions with similar content.
[0105] Furthermore, to ensure that the merging of weighted feature sets based on the orthogonality index of the feature space has a clear computational object, the system can first split the weighted feature set into sub-vectors to be compared according to the source identifier or preset grouping rules, then calculate the orthogonality index for each pair of sub-vectors, and determine the retention ratio or merging ratio of the corresponding sub-vectors based on the orthogonality index; in this way, a clear data processing link is formed between vector inner product calculation, orthogonality index generation and subsequent merging actions;
[0106] Furthermore, if there are all zeros or nearly all zeros in the sub-vectors, the inner product result may be distorted. In this case, the system can directly set the orthogonality exponent to the default median or skip the orthogonality correction of this window. If the lengths of the two sub-vectors are inconsistent, they are first adjusted to the same dimension through unified mapping before calculating the inner product to prevent the calculation results from being incomparable. If the magnitude of a certain sub-vector is abnormally high and dominates the similarity calculation, magnitude pruning can be added before normalization to prevent a single peak dimension from masking the overall structural relationship.
[0107] Furthermore, to ensure consistency with the merging process in subsequent implementations, image sub-vectors and audio sub-vectors are preferably obtained by dividing them according to the modal source in the weighted feature set, rather than arbitrarily segmenting them based on numerical values. If the weighted feature set retains channel source identifiers, they can be directly split based on the source identifiers. If dimensional rearrangement has occurred after the previous reconstruction, the rearrangement index table should be retained simultaneously so that when calculating the vector inner product, it is still clear which dimensions come from the first dimensionality reduction feature and which dimensions come from the second dimensionality reduction feature. In this way, the orthogonality index of the feature space always corresponds to a clear technical object, avoiding the drift of the referential relationship of the same index in different windows.
[0108] When a patient has persistent wheezing, although the chest wall compensation in the image and the high-frequency abnormality in the audio both point to an increase in respiratory load, the information they carry is not completely overlapping. After the system calculates a higher orthogonality index, it can retain more information from both sides when merging. In the window of the patient's violent cough, the image and audio are strongly driven by the same action, and the orthogonality index will decrease. The system avoids redundant abnormalities caused by repeated and enhanced cough.
[0109] The purpose of this step is to further quantify the degree of feature redundancy before the final merging, so that the fusion result retains complementarity without excessively stacking similar information.
[0110] Furthermore, based on the orthogonality index of the feature space, the weighted feature set is merged to determine the target multimodal fusion feature, including: multiplying the orthogonality index of the feature space and the weighted feature set to determine the temporary fusion result; and performing global average pooling and normalization on the temporary fusion result to obtain the target multimodal fusion feature.
[0111] This embodiment provides a specific merging step based on the orthogonality index. Specifically, after obtaining the orthogonality index of the feature space, the system does not use direct summation or hard concatenation, but first uses the index to uniformly modulate the weighted feature set to obtain a temporary fusion result, and then outputs the target multimodal fusion feature through global average pooling and normalization. Specifically, continuing the previous example, if the weighted feature set is:
[0112]
[0113] With an orthogonality index of 0.82, the product yields a temporary fusion result:
[0114]
[0115] The system can perform global average pooling according to preset groups, for example, treating the first 3 dimensions as one group and the last 3 dimensions as another group, to obtain... Then, normalization is performed to bring it into a uniform range, ultimately forming a target multimodal fusion feature that can be directly fed into the discrimination and classification head; the result obtained in this way is more compact and easier for downstream models to use stably.
[0116] The reason for adopting this approach is that if the decision to merge is made solely based on the orthogonality index without specifying the merging action, the output is still not standardized enough. This embodiment further clarifies how the index participates in numerical calculations and how pooling and normalization constrain the results of different windows to a comparable scale.
[0117] As an anomaly tolerance mechanism, if the orthogonality index is too low, for example, below a preset lower limit, the system can set a minimum retention value to prevent all fused features from being over-compressed; if the number of elements in a certain group is zero during global average pooling, for example, if a certain mode is completely eliminated in the current window, the average value of that group can be filled by another average value or historical smoothed value; if the denominator of the normalization is zero or close to zero, the unnormalized result is directly output and an anomaly flag is attached to prevent numerical distortion.
[0118] It should be further noted that the distinction between treating the first three dimensions as one group and the last three dimensions as another in this embodiment is merely a specific example illustrating the length of the aforementioned example vector, and is not a fixed limitation on the global average pooling processing in the embodiment. In actual deployment, the pooling grouping is preferably determined based on the source identifier or preset feature group boundary in the weighted feature set.
[0119] When the weighted feature set is arranged continuously according to modality, the dimensions corresponding to the first dimensionality reduction feature can be grouped into one group and the dimensions corresponding to the second dimensionality reduction feature can be grouped into another group. When the weighted feature set is arranged in an alternating manner according to feature groups, the same feature groups are merged and then averaged separately. This ensures that the global average pooling process always has consistent data objects under different window and different dimension configurations, and will not introduce ambiguity due to the dimension partitioning method in the example.
[0120] Furthermore, the normalization process preferably adopts a fixed-caliber range compression method to ensure that the target multimodal fusion features obtained from different windows are at a comparable scale; if minimum-maximum value normalization is used, the reference boundary should be fixed in the same deployment version; if vector norm normalization is used, it is preferable to perform it on the pooled result as a whole, rather than using different rules for each dimension, so as to avoid the same normalization process referring to different operations in the text.
[0121] In the later stages of patient recovery, both images and audio showed a stabilizing state with the orthogonality index at a moderate level. The system obtained a smooth, low-risk fusion feature through multiplication, pooling, and normalization. However, when the patient suddenly experienced persistent inspiratory distress, the fusion feature obtained through the same steps significantly increased in the risk dimension and remained stable in scale, facilitating threshold judgment and trend tracking.
[0122] The purpose of this step is to translate the aforementioned orthogonality assessment into a unified, comparable, and directly usable fusion vector output form, thereby improving the stability during deployment.
[0123] Furthermore, the method also includes: obtaining preset labels; calculating model loss values based on target multimodal fusion features and preset labels; and updating parameters of the preset feature extraction network through backpropagation based on the model loss values.
[0124] This embodiment provides a training update step; specifically, after the target multimodal fusion feature output is completed in the aforementioned implementation methods, the system can also calculate the model loss value in combination with preset labels during the training phase, and update the preset feature extraction network through backpropagation, so that the whole process can not only perform inference, but also continuously learn the real pattern of postoperative airway complications in patients.
[0125] Specifically, the preset labels can come from clinical records, manual annotations, or postoperative review results; for example, each time window can be labeled as one of stable, mild, abnormal, moderate, or high risk. The system inputs the target multimodal fusion features into the discriminant classification head to obtain the prediction results, and then compares them with the corresponding labels to calculate the model loss value. For ease of explanation, if the true label of a certain window is moderate risk, but the model outputs a stable high probability, then the loss value of that window is large; if the prediction is consistent with the truth, then the loss value is small. The system propagates the loss value backward along the fusion module, mapping module, and dimensionality reduction module to the image convolutional network and audio acoustic model, updating the parameters of each layer.
[0126] This training mechanism addresses the following technical shortcomings: Although conflict suppression, complementary mapping, and orthogonality merging have been defined in the preceding steps, these mechanisms are difficult to adaptively optimize in different wards, different equipment, and different patient groups without continuous data-driven correction; Through backpropagation, the system can progressively optimize the weight allocation of which image features and which audio features should be strengthened or attenuated in actual clinical practice.
[0127] As an anomaly tolerance mechanism, if the label distribution in a batch of training samples is severely imbalanced, for example, there are far more stable samples than high-risk samples, the system can increase the loss weight for minority class windows to prevent the model from only outputting low-risk data; if a window has been marked as bimodal low confidence during the acquisition phase, it can be removed from parameter updates to prevent abnormal data from disturbing training; if the gradient of a branch is close to zero for a long time after backpropagation, it indicates that the learning of that branch has stagnated, and the system can enable a small-scale reinitialization or adjustment of the learning rate.
[0128] After a month of trial operation in the hospital, the system accumulated a large amount of convalescent monitoring data with doctor review labels; through periodic training and updates, the image branch gradually became more accurate in identifying perioral cyanosis and chest wall compensation, the audio branch gradually became more accurate in distinguishing wheezing from environmental noise, and the fusion branch gradually optimized its robust ability to distinguish between nursing procedures, coughs, and real airway obstruction.
[0129] The purpose of this step is to enable the entire multimodal fusion process to be trainable and iterative, thereby improving its long-term adaptability to target medical scenarios and recognition accuracy.
[0130] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims
1. A method for feature fusion and engineering processing of multimodal data, characterized in that, The method includes: The initial multimodal data of the target object is obtained according to a preset time window. The initial multimodal data includes image data collected by an image sensor and audio data collected by an audio acquisition device within the preset time window. Based on a preset feature extraction network, feature extraction processing is performed on the initial multimodal data to determine an initial modal feature set. The initial modal feature set includes a first modal feature corresponding to the image data and a second modal feature corresponding to the audio data. Feature distribution difference analysis processing is performed on the initial modal feature set to determine the dynamic weight suppression coefficient. Based on the dynamic weight suppression coefficient, the initial modal feature set is subjected to feature dimensionality reduction processing to determine the intermediate modal feature set, wherein the intermediate modal feature set includes a first dimensionality reduction feature corresponding to the first modal feature and a second dimensionality reduction feature corresponding to the second modal feature; the intermediate modal feature set is subjected to nonlinear mapping processing to determine the cross-modal mutual information gain rate; Based on the cross-modal mutual information gain rate, the intermediate modal feature set is subjected to feature weighted fusion processing to determine the target multimodal fusion feature.
2. The feature fusion and engineering processing method for multimodal data according to claim 1, characterized in that, The step of performing feature extraction processing on the initial multimodal data based on a preset feature extraction network to determine an initial modal feature set includes: extracting the first modal feature from the image data using a convolutional neural network in the preset feature extraction network, and extracting the second modal feature from the audio data using an acoustic feature extraction model in the preset feature extraction network.
3. The feature fusion and engineering processing method for multimodal data according to claim 1, characterized in that, The step of performing feature distribution difference analysis on the initial modal feature set to determine the dynamic weight suppression coefficient includes: calculating the feature distribution difference of the initial modal feature set to determine the degree of feature distribution difference and the corresponding conflict feature subset; and determining the dynamic weight suppression coefficient based on the conflict feature subset.
4. The feature fusion and engineering processing method for multimodal data according to claim 3, characterized in that, The step of determining the dynamic weight suppression coefficient based on the conflict feature subset includes: if the degree of difference corresponding to the conflict feature subset is greater than a preset conflict threshold, then the dynamic weight suppression coefficient is obtained by performing a nonlinear activation function calculation on the conflict feature subset; if the degree of difference corresponding to the conflict feature subset is less than or equal to the preset conflict threshold, then the dynamic weight suppression coefficient is obtained by performing a linear mapping calculation on the conflict feature subset.
5. The feature fusion and engineering processing method for multimodal data according to claim 1, characterized in that, The step of performing feature dimensionality reduction processing on the initial modal feature set according to the dynamic weight suppression coefficient to determine the intermediate modal feature set includes: performing channel-level weight attenuation processing on the first modal feature according to the dynamic weight suppression coefficient to determine the first dimensionality-reduced feature; performing channel-level weight attenuation processing on the second modal feature according to the dynamic weight suppression coefficient to determine the second dimensionality-reduced feature; and concatenating the first dimensionality-reduced feature and the second dimensionality-reduced feature to determine the intermediate modal feature set.
6. The feature fusion and engineering processing method for multimodal data according to claim 1, characterized in that, The step of performing nonlinear mapping processing on the intermediate modal feature set to determine the cross-modal mutual information gain rate includes: performing fully connected layer dimension mapping processing on the intermediate modal feature set to determine cross-modal complementarity features; and determining the cross-modal mutual information gain rate based on the cross-modal complementarity features and the intermediate modal feature set.
7. The feature fusion and engineering processing method for multimodal data according to claim 1, characterized in that, The step of performing feature weighting and fusion processing on the intermediate modal feature set based on the cross-modal mutual information gain rate to determine the target multimodal fusion feature includes: performing feature reconstruction processing on the intermediate modal feature set based on the cross-modal mutual information gain rate to determine a weighted feature set; and performing merging processing on the weighted feature set to determine the target multimodal fusion feature.
8. The feature fusion and engineering processing method for multimodal data according to claim 7, characterized in that, The step of merging the weighted feature set to determine the target multimodal fusion feature includes: dividing the weighted feature set into image sub-vectors and audio sub-vectors according to the modality source; calculating the inner product of the image sub-vectors and the audio sub-vectors to determine the orthogonality index of the feature space; and merging the weighted feature set according to the orthogonality index of the feature space to determine the target multimodal fusion feature.
9. The feature fusion and engineering processing method for multimodal data according to claim 8, characterized in that, The step of merging the weighted feature set according to the orthogonality index of the feature space to determine the target multimodal fusion feature includes: multiplying the orthogonality index of the feature space and the weighted feature set to determine a temporary fusion result; and performing global average pooling and normalization on the temporary fusion result to obtain the target multimodal fusion feature.
10. The feature fusion and engineering processing method for multimodal data according to claim 1, characterized in that, The method further includes: obtaining preset labels; calculating model loss values based on the target multimodal fusion features and the preset labels; and updating the parameters of the preset feature extraction network through backpropagation based on the model loss values.