Multimodal emotion recognition method and device, electronic equipment and storage medium

Abstract

This invention relates to a multimodal emotion recognition method, apparatus, electronic device, and storage medium, belonging to the field of emotion recognition technology. The method includes: calculating the quality index of each modality sample in the multimodal working condition samples corresponding to each effective time window; selecting a target time window based on the matching degree between each effective time window and the target crew member, and performing alignment, offset compensation, gating enhancement processing, and position embedding processing on each modality sample corresponding to the target time window to obtain a unified representation feature for each modality; performing weighted fusion based on the quality index of each modality sample and the unified representation feature of each modality to obtain a multimodal fusion feature; and obtaining the predicted emotion of the target crew member based on the multimodal fusion feature and a large language model. This invention addresses the problems of large fluctuations in modal samples, mixed emotion sources, asynchronous cross-modal expression, and low emotion prediction accuracy in ocean environments.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition technology, and in particular to a multimodal emotion recognition method, device, electronic device, and storage medium. Background Technology

[0002] During long voyages on ocean-going vessels, crew members are typically in confined spaces with long work cycles, limited social contact, and disrupted circadian rhythms, making them prone to fatigue, anxiety, depression, and irritability. Crew members' emotional state not only affects their individual work efficiency and communication quality but also further impacts shipboard collaborative operations, safety watchkeeping, and emergency response capabilities. Therefore, developing an emotion recognition method adapted to the actual working conditions of ocean-going vessels has significant engineering application value.

[0003] In existing technologies, emotion recognition methods typically analyze a single modality from text, speech, or images. While these methods can achieve certain results in relatively stable environments with high-quality data, in high-ocean environments, factors such as continuous mechanical noise, low-light illumination, partial occlusion, close-range communication among multiple people, and discontinuous data acquisition can lead to distortion, loss, or mixed sources of single-modal information, resulting in insufficient stability and poor robustness of the recognition results.

[0004] To improve recognition performance, existing technologies have proposed multimodal emotion recognition methods. These methods typically extract text, speech, and image features separately and fuse them through weighting, voting, concatenation, or unified representation. While these methods can utilize complementary multimodal information to some extent, most solutions primarily focus on multimodal association modeling in ordinary scenarios, lacking specific consideration for the complex conditions at sea. Especially in ocean scenarios, crew members' emotional expressions are often restrained, implicit, and phased, easily leading to cross-modal asynchronous phenomena such as inconsistencies between voice fluctuations and facial expressions, and subtle textual expressions with obvious vocal variations. Furthermore, multiple crew members may simultaneously be in the same cabin or within the same acquisition range, increasing the risk of mixed emotion sources from different modalities. If the general fusion methods used in ordinary scenarios are still applied, it is often difficult to stably establish deep associations between different modalities, and it is also difficult to effectively suppress the interference of low-quality modalities affected by noise, low light, occlusion, or uncertain sources on the recognition results.

[0005] On the other hand, with the development of large-scale pre-trained models, they have shown great potential in long-term semantic modeling and complex association analysis. However, in the high-noise, low-light, and modal heterogeneous environments of the open ocean, if text, speech, and image features that have not undergone scene adaptation are directly input into subsequent discrimination models, the differences in feature dimensions, time scales, and expression densities among different modalities can be further amplified, thereby affecting the effective integration of emotional cues and the stability of recognition.

[0006] Therefore, there is an urgent need to propose a multimodal emotion recognition method for crew members under the complex working conditions of ocean-going vessels, in order to solve problems such as large fluctuations in modal quality, easy mixing of emotion sources, easy asynchronous cross-modal expression, and insufficient adaptability of existing general fusion methods in the ocean environment. Summary of the Invention

[0007] In view of this, it is necessary to provide a multimodal emotion recognition method, device, electronic device and storage medium to solve the problems of large fluctuations in modal samples, mixed emotion sources, asynchronous cross-modal expression and low emotion prediction accuracy in the ocean environment.

[0008] To address the above problems, embodiments of the present invention provide a multimodal emotion recognition method, including: S1, calculate the quality index of each modal sample in the multimodal working condition sample based on the multimodal working condition sample corresponding to each effective time window; S2, based on the matching degree between each effective time window and the target crew member, the target time window is selected, and the modal samples corresponding to the target time window are aligned, offset compensated, gating enhancement and position embedding are performed respectively to obtain the unified representation features of each modality; S3, based on the quality indicators of each modality sample and the unified representation features of each modality, weighted fusion is performed to obtain multimodal fusion features; S4. Based on the multimodal fusion features and the large language model, the predicted sentiment of the target crew member is obtained.

[0009] Further, step S2 includes: S21, calculate the matching degree between each effective time window and the target crew member, and take the effective time window with a matching degree greater than a preset matching threshold as the target time window; S22, map the modal samples corresponding to the target time window to a unified dimension to obtain the alignment features of each modality; S23, perform time offset compensation on the alignment features of each mode to obtain the compensation features of each mode; S24. Based on the alignment features and compensation features of each modality, gating enhancement is performed to obtain the enhancement features of each modality. S25, the enhanced features of each modality are jointly encoded with modality embedding, location embedding, person embedding, and time embedding to obtain the unified representation features of each modality.

[0010] Furthermore, step S24 is achieved through the following formula: ; in, Indicates the target time window intermediate mode Enhanced features, Indicates the target time window intermediate mode The gating coefficients generated from the quality indicators, Indicates the target time window intermediate mode Alignment features, Indicates the target time window intermediate mode The compensation characteristics.

[0011] Further, step S3 includes: S31, input the unified representation features of each modality into the cross-modal fusion coding network to obtain the fusion features of each modality; S32, Calculate the reliability of each mode based on the quality indicators of each mode and the ocean-going operating condition samples in the target time window; S33, normalize the reliability of each mode to obtain the weight of each mode; S34. Based on the fusion features of each modality and the weights of each modality, weighted fusion is performed to obtain the multimodal fusion features.

[0012] Furthermore, the calculation formula for step S32 is as follows: ; in, Indicates the target time window intermediate mode Quality indicators This represents the activation function. Representing modes The first learnable parameter, Representing modes The second learnable parameter, Indicates the target time window The ocean-going operating condition sample in China Indicates the target time window intermediate mode The source status or missing mask information, The value can be 0 or 1.

[0013] Further, step S1 includes: S11, Collect multimodal working condition information within a preset sampling period. The multimodal working condition information includes multimodal emotion information and ocean working condition information corresponding to the target area. The multimodality includes text, voice and image. S12, the multimodal operating condition information is divided according to a preset time window to obtain a multimodal operating condition sample corresponding to each preset time window; S13, calculate the quality index of each mode in each multimodal working condition sample; S14, remove the preset time window where the quality index of each modality is less than the preset quality threshold to obtain the effective time window.

[0014] Further, step S4 includes: S41, Input the multimodal fusion features into the large language model to obtain hidden state features; S42, The hidden state features are processed by linear mapping and normalization to obtain the predicted probability corresponding to each emotion category; S43, based on the predicted probability corresponding to each emotion category, obtain the predicted emotion of the target crew member.

[0015] This invention provides a multimodal emotion recognition device, comprising: The acquisition module is used to calculate the quality index of each modal sample in the multimodal operating condition sample based on the multimodal operating condition sample corresponding to each effective time window. The characterization module is used to filter out target time windows based on the matching degree between each effective time window and the target crew member, and to perform alignment, offset compensation, gating enhancement processing and position embedding processing on each modal sample corresponding to the target time window to obtain unified characterization features for each modality. The fusion module is used to perform weighted fusion based on the quality indicators of each modality sample and the unified representation features of each modality to obtain multimodal fusion features; The prediction module is used to obtain the predicted sentiment of the target crew member based on the multimodal fusion features and the large language model.

[0016] This invention provides an electronic device, including a memory and a processor. The memory is used to store programs or instructions, and the processor is used to execute the programs or instructions stored in the memory to implement the multimodal emotion recognition method described above.

[0017] This invention provides a computer-readable storage medium for storing a computer-readable program or instruction, which, when executed by a processor, can implement the multimodal emotion recognition method described above.

[0018] The beneficial effects of this invention are: (1) Construct a unified multimodal representation feature for complex ocean-going conditions. In the feature construction stage, introduce working condition parameters and quality indicators of each modality. Through unified mapping, location embedding, person embedding and time embedding, reduce the differences in scale, time sequence and expression of different modalities.

[0019] (2) In view of the characteristics of multiple people collecting data in the same domain and asynchronous emotion expression in ocean, a target crew source constraint and time offset compensation mechanism are introduced, and adaptive collaborative fusion is carried out in combination with the reliability of each modality under ocean working conditions to enhance the ability to distinguish the source of emotion and reduce the interference of mechanical noise, weak light occlusion, information loss and source uncertainty on the recognition results.

[0020] (3) The highly robust multimodal fusion features are input into the decoding large language model. The long temporal semantic modeling and cross-modal association understanding capabilities of the large model are used to complete the identification of the target crew's emotional state, which improves the accuracy and robustness of the identification. Attached Figure Description

[0021] Figure 1 A flowchart illustrating a multimodal emotion recognition method provided in an embodiment of the present invention; Figure 2 This is a comparison chart of ablation experiment results provided in an embodiment of the present invention; Figure 3 This is a structural diagram of a multimodal emotion recognition device provided in an embodiment of the present invention. Detailed Implementation

[0022] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0023] Ocean voyages are characterized by prolonged periods of isolation, sustained high pressure, and complex operating conditions. The emotional state of crew members not only affects their individual physical and mental health but also directly impacts watchkeeping judgment, collaborative operations, and ship navigation safety. Therefore, how to stably, accurately, and deployably identify crew members' emotions in the ocean environment is a pressing technical problem that needs to be solved.

[0024] Although various emotion perception methods already exist, they still reveal significant shortcomings in application under specific extreme conditions such as ocean-going operations.

[0025] First, text, voice, and image data in the open ocean environment are subject to continuous mechanical noise, low light illumination, partial occlusion, limited shooting angle, and discontinuous acquisition, which can easily lead to distortion, loss, or quality fluctuation of single-modal emotional cues, making it difficult for single-modal recognition methods to maintain stable performance.

[0026] Furthermore, the emotional expression of ocean-going sailors is usually characterized by restraint, implicitness, and stages. This often manifests as implicit textual expression, obvious vocal fluctuations with weak facial expression changes, or relatively calm facial expressions but tense tone, among other cross-modal asynchronous phenomena. When ordinary multimodal methods use simple splicing, weighting, or voting, it is difficult to effectively model the deep connections and differences between different modalities.

[0027] Furthermore, in ocean-going cabins or collaborative work environments, multiple crew members may be within the same visual or auditory acquisition range at the same time, making it easy for the sources of emotions to become mixed.

[0028] Finally, shipborne terminals have relatively limited computing resources, and directly adopting a multi-model parallel post-processing approach is often not conducive to practical deployment.

[0029] Furthermore, if multi-source heterogeneous features that have not been uniformly aligned and screened are directly input into the subsequent discrimination model, the natural differences between different modalities in terms of representation dimension, time scale and expression density will be further amplified. Low-quality modalities will also interfere with the overall sentiment judgment, thereby affecting the accuracy and robustness of the recognition results.

[0030] Therefore, the problem to be solved by this invention is: how to fuse emotional cues in text, speech and images in the context of continuous mechanical noise, low light illumination, restrained expression, multi-person simultaneous acquisition and cross-modal asynchronous processing in ocean-going vessel scenarios, thereby improving the accuracy, stability, robustness and engineering applicability of crew emotion recognition in complex ocean-going environments.

[0031] To address these challenges, this invention constructs a multimodal unified representation space to accommodate the characteristics of continuous mechanical noise, low-light illumination, restrained expression, multi-person simultaneous data collection, and changes in operational phases in ocean-going scenarios. It introduces person embedding and temporal embedding, and combines modal reliability for adaptive collaborative fusion. Finally, the fused, highly robust features are input into a decoding-based large language model to determine the emotional state of the crew members.

[0032] Figure 1 A flowchart of a multimodal emotion recognition method provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes: S1, calculate the quality index of each modal sample in the multimodal working condition sample based on the multimodal working condition sample corresponding to each effective time window; In the context of ocean-going vessels, emotion recognition input not only includes multimodal information such as text, speech, and images, but is also affected by complex operating conditions such as continuous mechanical noise, low light illumination, partial occlusion, multiple people operating in the same area, and changes during long voyages. Unlike ordinary office or teaching scenarios, emotional cues in the ocean environment are often not stable, continuous, or from a single source, but rather exhibit characteristics such as large fluctuations in modal quality, easily mixed sources, and discontinuous collection fragments. To ensure that subsequent feature extraction, source binding, and reliability constraint fusion are based on a unified and computable input, this invention first performs ocean-going operating condition perception and target sample construction.

[0033] Collect multimodal operating condition samples corresponding to each effective time window, and calculate the quality index of each modal sample in the multimodal operating condition sample.

[0034] S2, based on the matching degree between each effective time window and the target crew member, the target time window is selected, and the modal samples corresponding to the target time window are aligned, offset compensated, gating enhancement and position embedding are performed respectively to obtain the unified representation features of each modality; Since multiple people often enter the same data collection area at the same time in ocean-going cabins, duty areas, and collaborative work areas, voice and facial expressions are not synchronized, and there are fluctuations in the quality of local modalities, this step first performs unified mapping and alignment of each modality, and then combines the source constraints of the target crew members to complete the binding of personnel identifiers, asynchronous compensation, and robust representation generation, thereby constructing a unified representation feature adapted to ocean-going scenarios.

[0035] S3, based on the quality indicators of each modality sample and the unified representation features of each modality, weighted fusion is performed to obtain multimodal fusion features; The unified representation sequence of each modality obtained in step S2 is input into the fusion module to complete the deep interaction modeling between text, speech and image modalities. Dynamic fusion weights are generated by combining the reliability of each modality under ocean conditions to obtain the final multimodal fusion features.

[0036] This step further considers factors such as continuous mechanical noise, weak lighting, local occlusion, uncertain sources, and modal missingness in the ocean environment, and performs a constraint evaluation on the effectiveness of each modality in the current sample, thereby suppressing the interference of low-quality modalities on the recognition results.

[0037] S4. Based on the multimodal fusion features and the large language model, the predicted sentiment of the target crew member is obtained.

[0038] The multimodal fusion features obtained in step S3 are input into the decoding-based large language model. Leveraging its long-term temporal semantic modeling and complex relational understanding capabilities, the model determines the emotional state of the target crew members within the target time window. After the aforementioned processing steps, the multimodal fusion features have undergone unified representation, source constraints, and reliability weighting, thus enabling them to be fed into the subsequent discrimination model in a more stable form. This avoids the feature fragmentation problem that occurs when a general-purpose large model directly processes heterogeneous, low-quality features.

[0039] This invention provides an emotion recognition method for ocean-going scenarios based on large language models and multimodal adaptive fusion. It solves the problems of crew emotional cues being easily masked in long-term closed, high-pressure operations and multi-source noise environments on ocean-going vessels, asynchronous expression of different modalities, mixed sources of emotions in multi-person collaborative environments, and feature fragmentation when general large models directly process heterogeneous features.

[0040] In some embodiments, step S1 includes: S11, Collect multimodal working condition information within a preset sampling period. The multimodal working condition information includes multimodal emotion information and ocean working condition information corresponding to the target area. The multimodality includes text, voice and image. The parameters for ocean-going operations include at least: ambient noise intensity, cabin illuminance, data collection timestamp, watch status, navigation stage, and data collection area identifier. These can be determined based on actual conditions, and this embodiment of the invention does not impose specific limitations on them.

[0041] Multimodal emotional information includes three modalities: text information, voice information, and image information.

[0042] In this embodiment of the invention, by unifying the association between multimodal emotion information and ocean-going operational condition information, an input basis can be provided for subsequent modal reliability calculation and stage prior modeling.

[0043] Let the multimodal operating condition information corresponding to the k-th preset sampling period be: (1) in, Indicates the first Text information for a preset sampling period, Indicates the first Voice information for a preset sampling period, Indicates the first Image information for a preset sampling period can be either an image or a sequence of video frames. Indicates the first Ocean-going operating condition information for a preset sampling period.

[0044] S12, the multimodal operating condition information is divided according to a preset time window to obtain a multimodal operating condition sample corresponding to each preset time window; To reduce the impact of inconsistent sampling frequencies for different modes, the continuously acquired multimodal operating condition information is distributed according to a unified preset time window. The data is segmented to obtain multimodal operating condition samples: (2) Each multimodal operating condition sample corresponds to text information, voice information, image information, and ocean operating condition information within the same time range, thereby ensuring that subsequent processing is carried out under the same time-series reference.

[0045] S13, calculate the quality index of each mode in each multimodal working condition sample; In consideration of the phenomena such as extreme noise, low light and short-term occlusion that often exist in ocean scenarios, this invention further extracts the quality indicators of each modality during the sample construction stage to form prior indicators that can be used for subsequent reliability calculations.

[0046] The quality metrics for each modality include those for speech, text, and image. Speech quality metrics include signal-to-noise ratio, effective speech percentage, and transcription confidence. Text quality metrics include text length, completeness, and keyword coverage. Image quality metrics include average illumination, sharpness, and target visibility.

[0047] No. The quality indicators for each modality corresponding to a preset time window can be expressed as: (3) in, Indicates the first The quality index of text within a preset time window. Indicates the first The quality indicators of speech within a preset time window. Indicates the first Quality metrics for images within a preset time window.

[0048] S14, remove the preset time window where the quality index of each modality is less than the preset quality threshold to obtain the effective time window.

[0049] Based on this, a preliminary validity screening is performed on the multimodal operating condition samples. If all three modes are below a preset quality threshold within a preset time window, or if at least one core mode is missing within a preset time window and the degree of absence exceeds a preset proportion, the preset time window is marked as an invalid window and discarded; otherwise, it is considered a valid time window and the multimodal operating condition sample is retained. The valid window determination function can be expressed as: (4) Through the above processing, a target sample set can be obtained that includes multimodal samples, as well as information on ocean-going operating conditions and prior quality. (5) Therefore, subsequent steps no longer directly process the unfiltered raw heterogeneous data, but instead perform source binding, robust feature extraction, unified representation construction, and reliability constraint fusion on the multimodal working condition samples corresponding to the retained effective time window, thereby improving the stability and deployability of the entire emotion recognition process under complex ocean working conditions.

[0050] In some embodiments, step S2 includes: S21, calculate the matching degree between each effective time window and the target crew member, and take the effective time window with a matching degree greater than a preset matching threshold as the target time window; In this embodiment of the invention, by combining visual identity features, speaker features, and the overlap relationship between text timestamps and voice timestamps, the modal segments within each effective time window and the target crew member are calculated. Source match score: (6) in, and These represent the visual identity features and speaker features in the current window, respectively. and These represent the identity templates of the target crew member in the visual and speech domains, respectively. Represents the similarity function. The time overlap function is represented. These are the weighting coefficients.

[0051] Based on the matching degree of each valid time window, it is determined whether the multimodal fragments within that valid time window belong to the target crew member. If the threshold condition is met, the valid time window is taken as the target time window and assigned a corresponding person identifier; otherwise, the valid time window and its corresponding modal samples are deleted and marked as placeholders for unknown identities. The determination method is as follows: (7) in, A preset matching threshold is set. In this way, the person embedding is no longer just a static identity number, but a target identification information based on the consistency constraints of visual, speech and text cross-modal sources, thereby being able to distinguish the source of emotions of different crew members within the same collection range.

[0052] S22, map the modal samples corresponding to the target time window to a unified dimension to obtain the alignment features of each modality; The modal samples corresponding to each effective time window obtained in step S1 are length-normalized and projected onto a unified representation dimension through linear mapping to obtain the alignment features of each modality. Each modal sample includes text samples, speech samples, and image samples; the alignment features of each modality include text alignment features, speech alignment features, and image alignment features.

[0053] Let the first Modal within a valid time window The sample is ,in A unified mapping is performed, and the alignment features of each modality are as follows: (8) in, Indicates the first One effective time window mode Alignment features, Representing modes The slope, Representing modes The intercept.

[0054] Based on a unified mapping, in order to solve the problem of mixed emotion sources caused by multiple people collecting data in the same domain at sea, this invention introduces target crew source constraints for visual samples, voice samples, and text samples.

[0055] S23, perform time offset compensation on the alignment features of each mode to obtain the compensation features of each mode; In this embodiment of the invention, considering that the emotional expression of ocean-going crew members often manifests as "voice fluctuations first, followed by facial expression changes" or "text recording lags behind voice expression," the present invention performs time offset compensation on the alignment features of each modality within the target time window.

[0056] Let the mode With mode The alignment features are respectively and The optimal offset between the two can be expressed as: (9) Using this offset, the mode is obtained. With mode The compensation characteristics are then obtained. Using the same method, the compensation characteristics for each mode are also obtained.

[0057] The compensation features for each modality include text compensation features, speech compensation features, and image compensation features.

[0058] S24. Based on the alignment features and compensation features of each modality, gating enhancement is performed to obtain the enhancement features of each modality. After completing source binding and asynchronous compensation, to suppress local distortion caused by factors such as low light, occlusion, mechanical noise, and incomplete text, this invention combines the quality indices of each modality obtained in step S1 and performs gated enhancement on the alignment features of each modality within the target time window to obtain the enhanced features of each modality: (10) in, Indicates the target time window intermediate mode Enhanced features, Indicates the target time window intermediate mode The gating coefficients generated from the quality indicators, Indicates the target time window intermediate mode Alignment features, Indicates the target time window intermediate mode The compensation feature is generated based on the adjacent context or homologous neighboring segments.

[0059] This formula indicates that when a certain mode has high quality, the original features of the current mode are mainly retained; when the mode is greatly disturbed by the ocean environment, the proportion of context compensation components is increased to enhance the stability of the representation.

[0060] S25, the enhanced features of each modality are jointly encoded with modality embedding, location embedding, person embedding, and time embedding to obtain the unified representation features of each modality.

[0061] After completing the target crew source binding, cross-modal asynchronous compensation, and single-modal robust representation generation, the enhanced features of each modality are jointly encoded with modality embedding, location embedding, person embedding, and time embedding to obtain a unified representation vector for each modality: (11) in, Indicates the target time window intermediate mode A unified representation vector, Representing modes Modal embedding parameters, Indicates the target time window Position embedding parameters, Indicates the target time window Internal target crew Character embedding parameters, Indicates the target time window The time embedding parameter, Indicates the embedding weight of the person. This represents the temporal embedding weight.

[0062] The unified representation features for each modality include unified representation features for text, unified representation features for speech, and unified representation features for images.

[0063] Furthermore, the unified representation features of text, speech, and image are combined to form the target time window. The unified representation sequence: (12) This unified representation sequence As output, it is input to the subsequent modal reliability constraint fusion module.

[0064] In some embodiments, step S3 includes: S31, input the unified representation features of each modality into the cross-modal fusion coding network to obtain the fusion features of each modality; The unified representation sequence of each modality output from step S2 is input into the cross-modal fusion coding network to obtain the fusion features of each modality. Among them, the fusion features of each modality include text fusion features, speech fusion features, and image fusion features.

[0065] Let the target time window be... The unified representation sequence for each mode is as follows The fusion coding result can be expressed as: (13) S32, Calculate the reliability of each mode based on the quality indicators of each mode and the ocean-going operating condition samples in the target time window; Subsequently, combining the quality indicators of each mode extracted in step S1, the ocean-going operating condition samples, and the source status information in step S2, the reliability score of each mode within the target time window is calculated. The reliability of a mode can be expressed as: ; in, Indicates the target time window intermediate mode Quality indicators This represents the activation function. Representing modes The first learnable parameter, Representing modes The second learnable parameter, Indicates the target time window The ocean-going operating condition sample in China Indicates the target time window intermediate mode The source status or missing mask information, The value can be 0 or 1.

[0066] This formula allows for the reflection of enhanced mechanical noise in speech, low-light occlusion in vision, and incomplete expressions in text in the reliability assessment results.

[0067] S33, normalize the reliability of each mode to obtain the weight of each mode; After obtaining the reliability of each mode, it is normalized to generate the weights of each mode within the target time window: (15) in, Indicates the target time window intermediate mode The weight, Indicates the target time window intermediate mode Quality indicators.

[0068] The weights for each modality include text weights, speech weights, and image weights.

[0069] S34. Based on the fusion features of each modality and the weights of each modality, weighted fusion is performed to obtain the multimodal fusion features.

[0070] The fusion features of each modality are weighted and fused to obtain the final multimodal fusion features: (16) in, Indicates the target time window The corresponding multimodal fusion features.

[0071] Through the above processing, when a certain mode is significantly disturbed or its source is uncertain under the current ocean-going conditions, its weight will be automatically reduced; when a certain mode has higher quality and a clearer source, its contribution to the final fusion result will be increased accordingly.

[0072] Therefore, the fusion features of each mode output in step S3 It not only preserves the discrimination information of text, speech and visual modalities, but also can adaptively adjust the fusion results according to modal reliability under complex conditions at sea, thus providing a more stable and scene-adaptive input for the emotion discrimination of subsequent decoding-based large language models.

[0073] In some embodiments, step S4 includes: S41: Input the multimodal fusion features into the large language model to obtain the hidden state features; Multimodal fusion features As a prefix feature sequence or conditional embedding input to a decoding-based large language model, the corresponding hidden state features are obtained: (17) in, This represents the reasoning process of a decoding-based large language model. Represents the target time window of the model output. The hidden state features.

[0074] S42: The hidden state features are linearly mapped and normalized to obtain the predicted probabilities corresponding to each emotion category; Hidden state features After linear mapping and normalization, the predicted probabilities for each emotion category are obtained: (18) in, and For output layer parameters, Indicates the target time window The corresponding probability distribution of emotion categories.

[0075] S43: Based on the predicted probability corresponding to each emotion category, obtain the predicted emotion of the target crew member.

[0076] The emotion category with the highest predicted probability is taken as the predicted emotion for the target crew member.

[0077] Finally, the emotion category with the highest probability is taken as the emotion recognition result for the target time window; in one embodiment, the emotion risk level can be further output according to the probability distribution for emotion monitoring and early warning in ocean scenarios.

[0078] Preferably, the decoding-based large language model adopts a Transformer decoder-based structure; in resource-constrained shipboard deployment environments, rotational position encoding, grouped query attention, parameter quantization, or pruning strategies can also be introduced to enhance long sequence modeling capabilities and reduce inference overhead.

[0079] Through the above method, step S4 can further utilize the high-level semantic inference capability of the large language model to achieve stable recognition of the emotional state of crew members under complex ocean-going conditions, based on the ocean-going scenario adaptation processing completed in steps S1 to S3.

[0080] To verify the effectiveness of the method of the present invention, this embodiment conducted a systematic verification experiment based on a real ocean-going vessel scenario.

[0081] (1) Experimental dataset and evaluation metrics The dataset used in this embodiment of the invention originates from multimodal records of ocean-going crew members during real long-haul voyages and high-pressure operations. It contains a large amount of heterogeneous data with ambient noise from the ocean, specifically approximately 50,000 text entries, 1,300 video clips, and 1,500 audio recordings. This data covers the language semantics, facial expressions, and intonation / prosodic variations of the crew members in different mission scenarios. The dataset is divided into training, testing, and validation sets.

[0082] This invention uses the large language model LLaMA2-70B as the core benchmark model and selects four core indicators—accuracy, precision, recall, and F1 score—to quantitatively evaluate the performance in discriminating sentiment states.

[0083] (2) Validation and performance evaluation To verify the overall recognition performance of multimodal feature fusion in the high-noise and complex environment of the open ocean, this experiment conducted a comparative verification of the effectiveness of single-modal (pure text, pure speech, pure image) and trimodal fusion emotion recognition on the same evaluation dataset. The results are shown in Table 1.

[0084] Table 1

[0085] Experimental results show that traditional unimodal emotion recognition capabilities have significant limitations in ocean environments. Text and speech performed relatively well, with accuracies of 72.4% and 75.6%, respectively, while the visual modality, limited by the lighting conditions in the ocean cabin and the depth of feature extraction, achieved an accuracy of only 68.3%. The most significant improvement came from trimodal fusion, with the fused accuracy reaching 85.7%, an improvement of over 10% compared to the highest value of unimodal fusion (75.6% for audio). In terms of precision, recall, and F1 score, trimodal fusion also achieved optimal performance at 84.3%, 83.6%, and 83.9%, respectively. This preliminarily indicates that the multimodal approach of this invention can simultaneously utilize semantic, acoustic, and facial information, enhancing the discriminative power of emotional features while maintaining contextual consistency, significantly improving overall emotion recognition capability and robustness.

[0086] (3) Modular ablation experiment and results analysis After verifying the overall effectiveness of multimodal fusion in the embodiments of the present invention, in order to further verify the specific contribution of the multimodal fusion strategy proposed in this invention to performance improvement, an ablation experiment was designed in the embodiments of the present invention. The experimental results are as follows: Figure 2 .

[0087] Ablation experiments show that when three modalities are input simultaneously, the accuracy of the model is only 81.5% without the adaptive weight fusion strategy of this invention (i.e., simple feature concatenation). However, after introducing the multimodal fusion strategy of this invention, the accuracy jumps to 85.7%, significantly better than all control groups.

[0088] This real performance leap proves that the feature layer collaborative fusion mechanism of the present invention can effectively overcome the modal quality fluctuations caused by the strong noise environment in the ocean, enabling multimodal features to interact effectively at the deep semantic level, thereby significantly improving the accuracy of emotion recognition.

[0089] Figure 3 This is a schematic diagram of the structure of a multimodal emotion recognition device provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the device includes: The acquisition module 310 is used to calculate the quality index of each modal sample in the multimodal working condition sample based on the multimodal working condition sample corresponding to each effective time window. The characterization module 320 is used to filter out target time windows based on the matching degree between each effective time window and the target crew member, and to perform alignment, offset compensation, gating enhancement processing and position embedding processing on each modal sample corresponding to the target time window to obtain unified characterization features of each modality. The fusion module 330 is used to perform weighted fusion based on the quality indicators of each modality sample and the unified representation features of each modality to obtain multimodal fusion features; The prediction module 340 is used to obtain the predicted sentiment of the target crew member based on the multimodal fusion features and the large language model.

[0090] This embodiment is a device embodiment corresponding to the above method embodiment. Its specific embodiment is the same as the above method embodiment. For details, please refer to the above method embodiment. This device embodiment will not be described in detail here.

[0091] In one embodiment, the present invention also provides an electronic device. The electronic device includes a memory and a processor. In some embodiments, the memory may be an internal storage unit of the electronic device, such as a hard disk or RAM. In other embodiments, the memory may be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc.

[0092] Furthermore, the memory can include both internal storage units and external storage devices of the electronic device, and the memory is used to install the application software and various types of data of the electronic device.

[0093] In some embodiments, the processor may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, the processor may be local or remote. In some embodiments, the processor may be implemented on a cloud platform. In some embodiments, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, internal cloud, multi-cloud, or any combination thereof.

[0094] Furthermore, when the processor executes the multimodal emotion recognition program in memory, the following steps can be implemented: S1, calculate the quality index of each modal sample in the multimodal working condition sample based on the multimodal working condition sample corresponding to each effective time window; S2, based on the matching degree between each effective time window and the target crew member, the target time window is selected, and the modal samples corresponding to the target time window are aligned, offset compensated, gating enhancement and position embedding are performed respectively to obtain the unified representation features of each modality; S3, based on the quality indicators of each modality sample and the unified representation features of each modality, weighted fusion is performed to obtain multimodal fusion features; S4. Based on the multimodal fusion features and the large language model, the predicted sentiment of the target crew member is obtained.

[0095] It should be understood that when the processor executes the program of the multimodal emotion recognition method in memory, in addition to the functions mentioned above, it can also implement other functions, as can be found in the description of the corresponding method embodiments above.

[0096] Accordingly, embodiments of the present invention also provide a computer storage medium storing a computer program, which, when executed by a processor, implements the steps of a multimodal emotion recognition method described above. Alternatively, when executed by a processor, the computer program implements the functions of each module / unit in this embodiment of a multimodal emotion recognition device.

[0097] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0098] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal emotion recognition method, characterized in that, include: S1, calculate the quality index of each modal sample in the multimodal working condition sample based on the multimodal working condition sample corresponding to each effective time window; S2, based on the matching degree between each effective time window and the target crew member, the target time window is selected, and the modal samples corresponding to the target time window are aligned, offset compensated, gating enhancement and position embedding are performed respectively to obtain the unified representation features of each modality; S3, based on the quality indicators of each modality sample and the unified representation features of each modality, weighted fusion is performed to obtain multimodal fusion features; S4. Based on the multimodal fusion features and the large language model, the predicted sentiment of the target crew member is obtained.

2. The multimodal emotion recognition method according to claim 1, characterized in that, Step S2 includes: S21, calculate the matching degree between each effective time window and the target crew member, and take the effective time window with a matching degree greater than a preset matching threshold as the target time window; S22, map the modal samples corresponding to the target time window to a unified dimension to obtain the alignment features of each modality; S23, perform time offset compensation on the alignment features of each mode to obtain the compensation features of each mode; S24. Based on the alignment features and compensation features of each modality, gating enhancement is performed to obtain the enhancement features of each modality. S25, the enhanced features of each modality are jointly encoded with modality embedding, location embedding, person embedding, and time embedding to obtain the unified representation features of each modality.

3. The multimodal emotion recognition method according to claim 2, characterized in that, Step S24 is achieved through the following formula: ； in, Indicates the target time window intermediate mode Enhanced features, Indicates the target time window intermediate mode The gating coefficients generated from the quality indicators, Indicates the target time window intermediate mode Alignment features, Indicates the target time window intermediate mode The compensation characteristics.

4. The multimodal emotion recognition method according to claim 1, characterized in that, Step S3 includes: S31, input the unified representation features of each modality into the cross-modal fusion coding network to obtain the fusion features of each modality; S32, Calculate the reliability of each mode based on the quality indicators of each mode and the ocean-going operating condition samples in the target time window; S33, normalize the reliability of each mode to obtain the weight of each mode; S34. Based on the fusion features of each modality and the weights of each modality, weighted fusion is performed to obtain the multimodal fusion features.

5. The multimodal emotion recognition method according to claim 4, characterized in that, The calculation formula for step S32 is as follows: ； in, Indicates the target time window intermediate mode Quality indicators This represents the activation function. Representing modes The first learnable parameter, Representing modes The second learnable parameter, Indicates the target time window The ocean-going operating condition sample in China Indicates the target time window intermediate mode The source status or missing mask information, The value can be 0 or 1.

6. The multimodal emotion recognition method according to claim 1, characterized in that, Step S1 includes: S11, Collect multimodal working condition information within a preset sampling period. The multimodal working condition information includes multimodal emotion information and ocean working condition information corresponding to the target area. The multimodality includes text, voice and image. S12, the multimodal operating condition information is divided according to a preset time window to obtain a multimodal operating condition sample corresponding to each preset time window; S13, calculate the quality index of each mode in each multimodal working condition sample; S14, remove the preset time window where the quality index of each modality is less than the preset quality threshold to obtain the effective time window.

7. The multimodal emotion recognition method according to claim 1, characterized in that, Step S4 includes: S41: Input the multimodal fusion features into the large language model to obtain the hidden state features; S42: The hidden state features are linearly mapped and normalized to obtain the predicted probabilities corresponding to each emotion category; S43: Based on the predicted probability corresponding to each emotion category, obtain the predicted emotion of the target crew member.

8. A multimodal emotion recognition device, characterized in that, include: The acquisition module is used to calculate the quality index of each modal sample in the multimodal operating condition sample based on the multimodal operating condition sample corresponding to each effective time window. The characterization module is used to filter out target time windows based on the matching degree between each effective time window and the target crew member, and to perform alignment, offset compensation, gating enhancement processing and position embedding processing on each modal sample corresponding to the target time window to obtain unified characterization features for each modality. The fusion module is used to perform weighted fusion based on the quality indicators of each modality sample and the unified representation features of each modality to obtain multimodal fusion features; The prediction module is used to obtain the predicted sentiment of the target crew member based on the multimodal fusion features and the large language model.

9. An electronic device, characterized in that, The method includes a memory and a processor, wherein the memory is used to store programs or instructions, and the processor is used to execute the programs or instructions stored in the memory to implement the multimodal emotion recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the multimodal emotion recognition method according to any one of claims 1 to 7.

Images