Model training method, multi-modal depression screening method and electronic equipment

Abstract

The invention provides a model training method, a multi-modal depression screening method and electronic equipment, and belongs to the technical field of computers. The model training method comprises the following steps: dividing a video sample into a plurality of fragments; performing feature extraction on each segment to obtain a plurality of to-be-processed features; fusing the mth visual feature and the nth audio feature in the to-be-processed features to obtain a first fused feature; fusing the kth text feature and the nth audio feature in the to-be-processed features to obtain a second fused feature; processing the first fusion feature and the second fusion feature by using a cross-modal model to obtain a cross-modal fusion feature; processing the cross-modal fusion features by using a behavior pattern analysis model to obtain a depression screening result; and training a cross-modal model and a behavior pattern analysis model according to the depression screening result and the annotation information. According to the method, the cross-modal model is trained by using the cross-modal features, and depression screening is performed by using the cross-modal model, so that the accuracy of depression screening can be effectively improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a model training method, a multimodal depression screening method, and an electronic device. Background Technology

[0002] Post-stroke depression (PSD) is a common emotional complication of stroke that can significantly interfere with the rehabilitation process of stroke patients. Therefore, timely identification and intervention of PSD is of great clinical significance for improving patients' rehabilitation outcomes and long-term prognosis.

[0003] Traditional assessment methods using scales suffer from drawbacks such as high subjectivity and low efficiency. With the rapid development of artificial intelligence technology, machine learning methods have been applied to depression screening tasks. Summary of the Invention

[0004] The inventors noted that in related technologies, depression screening research based on multimodal models mainly focuses on visual and textual modalities, comprehensively assessing an individual's mental state through facial expressions, eye movements, and interview texts. However, the acoustic features contained in speech are highly sensitive to depressive moods, and current research lacks attention to the audio modality. There is an urgent need to construct a multimodal collaborative architecture that integrates visual, text, and audio to improve the effectiveness of depression screening.

[0005] Accordingly, this disclosure provides a model training method that trains a cross-modal model by fusing visual modal features, text modal features, and audio modal features of samples, and uses this cross-modal model for depression screening, which can effectively improve the accuracy of depression screening results.

[0006] In a first aspect of this disclosure, a model training method is provided, comprising: detecting each frame in a video sample to identify frames that reflect depression-specific features as valid frames and frames that do not reflect the depression-specific features as invalid frames; dividing the video sample into multiple segments of predetermined duration, wherein the proportion of valid frames in each segment is greater than a proportion threshold; extracting features from each segment to obtain multiple features to be processed, wherein each feature to be processed is a visual modal feature, an audio modal feature, or a text modal feature; fusing the m-th visual modal feature and the n-th audio modal feature among the features to be processed to obtain a first fused feature. M represents the total number of visual modal features. N represents the total number of audio modal features; the k-th text modal feature and the n-th audio modal feature in the features to be processed are fused to obtain the second fused feature. K represents the total number of visual modal features; the first fusion feature and the second fusion feature are processed using a cross-modal model with a mutual attention mechanism to obtain cross-modal fusion features; the cross-modal fusion features are processed using a behavioral pattern analysis model to obtain depression screening results; a loss value is determined based on the depression screening results and video sample annotation information; the cross-modal model and the behavioral pattern analysis model are trained using the loss value.

[0007] In some embodiments, the mutual attention mechanism includes a first sub-mechanism of querying corresponding visual modal features and text modal features using audio modal features, a second sub-mechanism of querying corresponding audio modal features using visual modal features and text modal features, a third sub-mechanism of querying corresponding text modal features using visual modal features, and a fourth sub-mechanism of querying corresponding visual modal features using text modal features.

[0008] In some embodiments, fusing the m-th visual modal feature and the n-th audio modal feature in the features to be processed to obtain a first fused feature includes: encoding the m-th visual modal feature using a visual encoder to obtain a visual encoded feature; encoding the n-th audio modal feature using an audio encoder to obtain an audio encoded feature; performing linear projection processing on the visual encoded feature to obtain a projection feature; performing pooling processing on the audio encoded feature to obtain a pooled feature; and obtaining the first fused feature based on the projection feature and the pooled feature.

[0009] In some embodiments, obtaining the first fusion feature based on the projection feature and the pooling feature includes: calculating a weighted sum of the projection feature and the pooling feature to obtain the first fusion feature.

[0010] In some embodiments, fusing the k-th text modal feature and the n-th audio modal feature in the features to be processed to obtain a second fused feature includes: encoding the k-th text modal feature using a text encoder to obtain a text-encoded feature; encoding the n-th audio modal feature using an audio encoder to obtain an audio-encoded feature; and fusing the text-encoded feature and the audio-encoded feature to obtain the second fused feature.

[0011] In some embodiments, detecting each frame in the video sample includes: detecting the gaze features, neuromotor features, behavioral synchronization features of the target subject in the i-th frame, and the fundamental frequency mutation rate of the audio stream corresponding to the i-th frame, wherein... I represents the total number of frames in the video sample; the scores for the gaze feature, the neuromotor feature, the behavioral synchronization feature, and the fundamental frequency mutation rate are calculated respectively to obtain the total clinical information score; if the total clinical information score is greater than or equal to the scoring threshold, the i-th frame is determined to be a valid frame; if the total clinical information score is less than the scoring threshold, the i-th frame is determined to be an invalid frame.

[0012] In some embodiments, the score of the gaze feature is positively correlated with the gaze direction deviation of the target subject; the score of the neuromotor feature is positively correlated with the activity intensity of a specified muscle; the score of the behavioral synchronization feature is positively correlated with the degree of synchronization of mouth movements or facial expressions when the speech rhythm of the target subject is disordered; and the score of the fundamental frequency mutation rate is determined by the relationship between the fundamental frequency mutation rate and a predetermined mutation rate threshold.

[0013] In some embodiments, the designated muscle includes at least one of the orbicularis oculi, zygomaticus major, and mandibular muscles of the target body.

[0014] In some embodiments, if at least one of the gaze feature, the neuromotor feature, and the behavioral synchronization feature of the target subject in the i-th frame cannot be detected, then the i-th frame is determined to be an invalid frame.

[0015] In some embodiments, determining the i-th frame as an invalid frame if the total score of the clinical information is less than the score threshold includes: if the total score of the clinical information is less than the score threshold, detecting whether there is speech rhythm disorder in the audio stream corresponding to the i-th frame; if there is no speech rhythm disorder in the audio stream corresponding to the i-th frame, then determining the i-th frame as an invalid frame.

[0016] In some embodiments, if there is speech rhythm disorder in the audio stream corresponding to the i-th frame, then the i-th frame is determined to be a valid frame.

[0017] In some embodiments, feature extraction for each segment includes: extracting multiple semantic features with different modalities from each segment; mapping each semantic feature to a specified semantic space to obtain multiple mapped features; aligning the multiple mapped features to obtain a position embedding feature for each mapped feature; and modifying each mapped feature according to its modality conditional embedding feature and position embedding feature to obtain multiple features to be processed.

[0018] In some embodiments, aligning the plurality of mapping features includes: in each segment, detecting the number of frames corresponding to each mapping feature to obtain a plurality of frame numbers; using the maximum number of frames among the plurality of frame numbers as a reference frame number; determining the mapping ratio of each mapping feature based on the ratio of the reference frame number to the number of frames corresponding to each mapping feature; and generating a position embedding feature of each mapping feature based on the mapping ratio of each mapping feature and the frame index of each mapping feature.

[0019] In some embodiments, modifying each mapping feature based on its modal conditional embedding feature and positional embedding feature includes: adding the j-th mapping feature, its modal conditional embedding feature, and its positional embedding feature to obtain the j-th feature to be processed. J represents the total number of mapping features.

[0020] In some embodiments, processing the cross-modal fusion features using a behavioral pattern analysis model to obtain depression screening results includes: masking the cross-modal fusion features using a preset mask matrix to obtain masked fusion features; and processing the masked fusion features using the behavioral pattern analysis model to obtain the depression screening results.

[0021] In a second aspect of this disclosure, a multimodal depression screening method is provided, comprising: detecting each frame in a video to be tested, such that frames reflecting depression-specific characteristics are considered valid frames, and frames not reflecting the depression-specific characteristics are considered invalid frames; dividing the video to be tested into multiple segments of predetermined duration, wherein the proportion of valid frames in each segment is greater than a proportion threshold; extracting features from each segment to obtain multiple features to be processed, wherein each feature to be processed is a visual modal feature, an audio modal feature, or a text modal feature; fusing the m-th visual modal feature and the n-th audio modal feature among the features to be processed to obtain a first fused feature. M represents the total number of visual modal features. N represents the total number of audio modal features; the k-th text modal feature and the n-th audio modal feature in the features to be processed are fused to obtain the second fused feature. K represents the total number of visual modal features; the first fusion feature and the second fusion feature are processed using a cross-modal model with a mutual attention mechanism to obtain cross-modal fusion features, wherein the cross-modal model is trained using the model training method involved in any of the above embodiments; the cross-modal fusion features are processed using a behavioral pattern analysis model to obtain depression screening results, wherein the behavioral pattern analysis model is trained using the model training method involved in any of the above embodiments.

[0022] In a third aspect of this disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute instructions stored in the memory to implement the methods involved in any of the above embodiments.

[0023] In a fourth aspect of this disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any of the above embodiments.

[0024] In a fifth aspect of this disclosure, a computer program product is provided, including computer instructions, wherein the computer instructions, when executed by a processor, implement the method as described in any of the above embodiments.

[0025] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic flowchart of a model training method according to an embodiment of the present disclosure;

[0028] Figure 2 This is a schematic diagram of the structure of multimodal feature alignment according to an embodiment of the present disclosure;

[0029] Figure 3 This is a schematic diagram of the structure of a cross-modal model according to an embodiment of the present disclosure;

[0030] Figure 4 This is a flowchart illustrating a multimodal depression screening method according to an embodiment of the present disclosure;

[0031] Figure 5 This is a flowchart illustrating another embodiment of the multimodal depression screening method disclosed herein;

[0032] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0033] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0034] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0035] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0036] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0037] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0038] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0039] The inventors noted that post-stroke depression is a common affective complication of stroke, characterized by persistent depressed mood and low emotional state. Patients in the acute phase of stroke often experience neurological deficits, reduced work capacity, and changes in family and social status. Furthermore, prolonged social isolation or insufficient psychological support during the stroke recovery period can accelerate the development of post-stroke depression and worsen existing depressive symptoms. The combined effect of these factors can lead to decreased patient adherence to rehabilitation, impaired neurological recovery, significantly reduced quality of life, and even an increased risk of recurrent stroke and death. Therefore, timely identification and intervention of post-stroke depression is of significant clinical importance for improving patient rehabilitation outcomes and long-term prognosis.

[0040] Psychomotor changes are a key characteristic of depression, often intertwined with cognitive and emotional symptoms. Depressed patients exhibit a significant bias in attention allocation, showing increased attention to negative emotional stimuli and decreased attention to positive emotional stimuli. Based on this attentional bias, dynamically assessing the attention allocation of depressed patients at the levels of emotional facial expressions and other facial features has the potential to become a novel method for early warning and accurate screening of depression.

[0041] Depression not only manifests as low mood but also causes changes in neurophysiology and cognitive function, thereby affecting an individual's language and behavioral characteristics. Depressed patients typically exhibit features such as reduced speech, monotonous intonation, and slowed responses. Among these, speech characteristics such as Mel-frequency cepstral coefficients (MFCC), fundamental frequency variation, speech spectrum, formants, and vocal amplitude fremitus are significantly correlated with an individual's psychological state. Furthermore, speech signal-based auxiliary diagnostic methods offer advantages such as non-invasiveness and privacy protection, increasing patients' willingness to participate in the diagnosis. Therefore, effective identification and evaluation of speech features can improve the accuracy of depression screening.

[0042] In addition to acoustic features, interview texts, as semantic carriers of speech information, can supplement emotional and cognitive information that cannot be directly reflected in speech signals. Depressed patients exhibit typical psychological characteristics in their language expression in interview texts. For example, their vocabulary tends to be self-focused, expressing negative emotions and cognitive closure; their syntactic structures are relatively simple, and they lack vocabulary reflecting positive emotions. Therefore, combining speech features with semantic features of the text holds promise for further improving the accuracy and interpretability of depression screening.

[0043] However, current research on depression screening based on multimodal models mainly focuses on visual and textual modalities, comprehensively assessing an individual's mental state through facial expressions, eye movements, and interview texts, while lacking attention to audio modalities.

[0044] Accordingly, this disclosure provides a model training method that integrates and analyzes multimodal data of visual, text, and audio through a cross-modal model, achieving a comprehensive characterization and accurate identification of depressive states from multiple levels of cognition, emotion, and behavior, thereby providing a new technical path for the early screening and intervention of depression.

[0045] Figure 1 This is a schematic flowchart illustrating a model training method according to an embodiment of the present disclosure. In some embodiments, the following model training method is performed by an electronic device, including steps 11-19.

[0046] In step 11, each frame in the video sample is detected so that frames that reflect depression-specific characteristics are considered valid frames, and frames that do not reflect depression-specific characteristics are considered invalid frames.

[0047] It should be noted that depression-specific characteristics include gaze patterns, neuromotor features, and behavioral synchronization features. Invalid frames include frames where the target subject is missing, the target subject is severely occluded, the image is blurred, or there is interference from non-target areas.

[0048] For example, the aforementioned depression-specific features include PSD-specific features, and the aforementioned gaze features include stroke-associated gaze features.

[0049] In some embodiments, the method for detecting each frame in a video sample includes the following steps S11-S14.

[0050] In step S11, the gaze characteristics, neuromotor characteristics, behavioral synchronization characteristics, and fundamental frequency mutation rate of the target subject in the i-th frame are detected. I represents the total number of frames in the video sample.

[0051] It should be noted that gaze characteristics include the gaze direction of the target subject. Neuromotor characteristics include the activity of specified muscles, such as involuntary contraction of the orbicularis oculi muscle, abnormal activity of the zygomaticus major muscle, and fluctuations in the tone of the mandibular muscle. Behavioral synchronization characteristics include mouth movements or facial expressions corresponding to speech rhythm disturbances. Simultaneously, the audio stream corresponding to the i-th frame can be extracted from the video samples to detect the fundamental frequency mutation rate of the audio stream corresponding to the i-th frame.

[0052] In some embodiments, if at least one of the gaze characteristics, neuromotor characteristics, and behavioral synchronization characteristics of the target subject in the i-th frame cannot be detected, the i-th frame is determined to be an invalid frame.

[0053] It should be noted that if gaze features, neuromotor features, and behavioral synchronization features cannot be detected simultaneously in the i-th frame, then the i-th frame cannot fully reflect the depression-specific features, thus determining the i-th frame as an invalid frame.

[0054] In step S12, the scores for gaze features, neuromotor features, behavioral synchronization features, and fundamental frequency mutation rate are calculated to obtain the total clinical information score.

[0055] In some embodiments, the gaze feature score is positively correlated with the gaze direction deviation of the target subject. The neuromotor feature score is positively correlated with the activity intensity of a specified muscle. The behavioral synchronization feature score is positively correlated with the degree of synchronization of mouth movements or facial expressions when the target subject's speech rhythm is disordered. The fundamental frequency mutation rate score is determined by the relationship between the fundamental frequency mutation rate and a predetermined mutation rate threshold.

[0056] In some embodiments, the designated muscles include at least one of the orbicularis oculi, zygomaticus major, and mandibular muscles of the target subject.

[0057] For example, if the deviation of the target subject's gaze direction from the horizontal or vertical direction is greater than a gaze direction deviation threshold, then the gaze feature score is determined to be a high gaze feature score. The gaze direction deviation threshold can be set to... The high score for the gaze feature can be set to 0.4.

[0058] For example, by performing image gradient magnitude quantization on the i-th frame, the activity intensity of a specified muscle in the i-th frame is obtained. The muscle activity intensity is then divided into multiple levels according to preset rules, such as high intensity, medium intensity, and low intensity. Based on the level of the specified muscle activity intensity in the i-th frame, the neuromotor feature score is determined to be either a high, medium, or low score. For example, if the specified muscle activity intensity is high, the neuromotor feature score is determined to be high; if the specified muscle activity intensity is medium, the neuromotor feature score is determined to be medium; and if the specified muscle activity intensity is low, the neuromotor feature score is determined to be low. The high, medium, and low scores can be set to 0.3, 0.2, and 0.1 points, respectively.

[0059] For example, a predetermined mutation rate threshold can be set. If the fundamental frequency mutation rate is greater than the predetermined mutation rate threshold, then the score of the fundamental frequency mutation rate is determined as a high mutation rate score. The predetermined mutation rate threshold can be set to 20%, and the high mutation rate score can be set to 0.3 points.

[0060] In some embodiments, the scores for gaze features, neuromotor features, behavioral synchronization features, and fundamental frequency mutation rate are summed to obtain a total clinical information score.

[0061] In step S13, if the total score of clinical information is greater than or equal to the score threshold, then the i-th frame is determined to be a valid frame.

[0062] For example, the scoring threshold can be set to 0.6 points.

[0063] In step S14, if the total score of clinical information is less than the score threshold, the i-th frame is determined to be an invalid frame.

[0064] It should be noted that if the total score of clinical information is less than the scoring threshold, the i-th frame cannot significantly reflect the depression-specific characteristics, thus the i-th frame is determined to be an invalid frame.

[0065] In some embodiments, if the total score of clinical information is less than a score threshold, it is detected whether there is speech rhythm disorder in the audio stream corresponding to the i-th frame. If there is no speech rhythm disorder in the audio stream corresponding to the i-th frame, the i-th frame is determined to be an invalid frame.

[0066] In some embodiments, if there is speech rhythm disorder in the audio stream corresponding to the i-th frame, then the i-th frame is determined to be a valid frame.

[0067] It's important to note that depression-specific features exhibit temporal correlation between the audio and visual modalities. For instance, when a subject exhibits speech abnormalities, their facial muscle activity often simultaneously displays depression-specific features. Therefore, even if depression-specific features cannot be reflected in the visual modality, if speech rhythm disorder is detected in the audio stream corresponding to the i-th frame in the audio modality, the i-th frame can still be identified as a valid frame.

[0068] The method described above achieves cross-modal collaborative verification through temporal correlation between audio and visual modalities. This ensures that even when video quality fluctuates (e.g., low-light environments affect the recognition of depression features), the visual modal can still be compensated by specific features in the audio modal, thereby improving the robustness of depression-specific feature recognition and increasing the signal-to-noise ratio of video samples.

[0069] In step 12, the video sample is divided into multiple segments with a predetermined duration, wherein the proportion of valid frames in each segment is greater than the proportion threshold.

[0070] For example, the first in the dataset One video sample is ,in Let A be the total number of video samples in the dataset. (The video samples are then...) Divided into multiple segments with a predetermined duration ,in This is the start time index. The percentage threshold can be set to 50%.

[0071] For example, for each segment Calculate the percentage of valid frames. If the percentage of valid frames is lower than the percentage threshold, the start time index can be adjusted using a sliding window. Continue until the percentage threshold requirement is met.

[0072] It should be noted that dividing the video samples into multiple segments with predetermined durations provides standardized processing units for model training. Simultaneously, by controlling the proportion of valid frames in each segment, the information density of the segment can be increased. The method described in the above embodiments effectively avoids computational redundancy caused by excessively long video durations, frame rate differences, and an excessive proportion of invalid frames.

[0073] In step 13, feature extraction is performed on each segment to obtain multiple features to be processed, where each feature to be processed is a visual modal feature, an audio modal feature, or a text modal feature.

[0074] In some embodiments, the method for feature extraction of each segment includes the following steps S21-S24.

[0075] In step S21, multiple semantic features with different modalities are extracted from each segment.

[0076] It should be noted that semantic features of the visual modality include facial emotion embeddings, gaze information, and blink information; semantic features of the text modality include language understanding embeddings; and semantic features of the audio modality include audio-speech embeddings. These semantic features can be extracted from each segment using a frozen pre-trained model.

[0077] In step S22, each semantic feature is mapped to a specified semantic space to obtain multiple mapped features.

[0078] It should be noted that different semantic features have different feature dimensions. By mapping each semantic feature to a specified semantic space, multiple mapped features are obtained, thereby unifying the feature dimensions of different mapped features.

[0079] For example, using the first A learnable feature encoder For fragments After processing, the first result is obtained. One mapping feature As shown in formula (1).

[0080] (1)

[0081] in, , To extract from fragments The total number of mapped features obtained, the first A learnable feature encoder The feature dimension of the mapped features can be unified to d, the first... One mapping feature , For the first The number of frames for each mapping feature.

[0082] In step S23, multiple mapping features are aligned to obtain the position embedding feature of each mapping feature.

[0083] In some embodiments, the method for aligning multiple mapping features includes the following steps S231-S234.

[0084] In step S231, in each segment, the number of frames corresponding to each mapping feature is detected to obtain multiple frame numbers.

[0085] It should be noted here that the first One mapping feature Frame count With the One mapping feature Sampling rate and segment of the mode It is related to the duration.

[0086] For example, suppose fragment The duration is 10 seconds, and the first mapping feature Belonging to the visual modality, with a sampling rate of 30fps, the first mapped feature... Frame count For 300 frames. Second mapping feature. Belonging to the audio modality, with a sampling rate of 120fps, the second mapping feature... Frame count It is 1200 frames.

[0087] In step S232, the maximum number of frames among multiple frames is taken as the reference number of frames.

[0088] For example, suppose the first One mapping feature The number of frames is Then the base frame number As shown in formula (2).

[0089] (2)

[0090] in, It is a function for maximizing the value.

[0091] In step S233, the mapping ratio of each mapping feature is determined based on the ratio of the base frame number to the frame number corresponding to each mapping feature.

[0092] For example, assuming the base frame number is , No. One mapping feature The number of frames is Then the first One mapping feature mapping ratio As shown in formula (3).

[0093] (3)

[0094] For example, assuming the baseline frame count is 1200 frames, the first mapping feature Frame count For 300 frames, the second mapping feature Frame count If the number of frames is 1200, then the first mapping feature The mapping ratio is 4, and the second mapping feature The mapping ratio is 1.

[0095] In step S234, a position embedding feature for each mapping feature is generated based on the mapping ratio of each mapping feature and the frame index of each mapping feature.

[0096] It should be noted here that the first One mapping feature Position embedding features are ,in Location embedding features Each row vector in the vector represents the positional code for a time step.

[0097] For example, suppose the first mapping feature The frame index is The first mapping feature If the mapping ratio is 4, then the first mapping feature generated... Location embedding features The row vectors in the vector represent positions 0, 4, 8, 12, ..., 1196, respectively. .

[0098] For example, suppose the second mapping feature The frame index is The second mapping feature If the mapping ratio is 1, then the second mapping feature is generated. Location embedding features The row vectors in the diagram represent positions 0, 1, 2, ..., 1199, respectively. .

[0099] Figure 2 This is a schematic diagram of the structure for multimodal feature alignment according to an embodiment of this disclosure. Figure 2 As shown, the first mapping feature Belongs to the visual modality, the second mapping feature It belongs to the audio modality.

[0100] It should be noted that the method involved in the above embodiments actually takes the mode with the highest sampling rate as the reference mode and maps the frame index of each mapping feature to the time axis of the reference mode according to the mapping ratio.

[0101] like Figure 2 As shown, the sampling rate for the visual modality is 30fps, and the sampling rate for the audio modality is 120fps. Therefore, each frame in the visual modality corresponds to four consecutive frames in the audio modality. The position code for the 0th frame of the visual modality and the 0th frame of the audio modality is 0; the position code for the 1st frame of the visual modality and the 4th frame of the audio modality is 4; and the position code for the 2nd frame of the visual modality and the 8th frame of the audio modality is 8.

[0102] The multimodal feature alignment described in the above embodiments enables the alignment of features from different modalities at time steps.

[0103] In some embodiments, the Fractional Position Embedding (FPE) method is used to generate position embedding features for each mapped feature.

[0104] For example, a positional embedding feature for each mapped feature can be generated using a Transformer-based sinusoidal positional encoding.

[0105] It should be noted that fractional position embedding is achieved through sinusoidal position coding. This process does not require training, and frames from different modalities at the same time point share the same position coding value, which can solve the cross-modal time synchronization problem.

[0106] In step S24, each mapping feature is modified according to its modal conditional embedding feature and positional embedding feature to obtain multiple features to be processed.

[0107] In some embodiments, the first The mapping feature, the first The modal conditional embedding features of the mapping feature, and the 1st mapping feature. The position embedding features of the mapped feature are added together to obtain the first mapped feature. There are 10 features to be processed. , This represents the total number of mapping features.

[0108] For example, suppose the first The mapping features are , No. One mapping feature The modal conditional embedding features are , No. One mapping feature Position embedding features are Then the first One feature to be processed As shown in formula (4).

[0109] (4)

[0110] The method described in the above embodiments modally conditionally embedding features and positionally embedding features of each mapping feature are used to modify each mapping feature to obtain a feature to be processed, so as to add modal information and temporal positional information to the feature to be processed.

[0111] In step 14, the m-th visual modality feature and the n-th audio modality feature in the features to be processed are fused to obtain the first fused feature. Wherein, M represents the total number of visual modal features. N represents the total number of audio modal features.

[0112] It should be noted that, depending on the modality, the multiple features to be processed can be categorized into visual modal features, audio modal features, and text modal features. In some embodiments, the multiple features to be processed may also include features from other modalities.

[0113] It should also be noted that current research on depression screening based on multimodal models mainly focuses on the visual and text modalities. LXMERT (Learning Cross-Modality Encoder Representations from Transformers), a commonly used multimodal task, can only process features from the visual-text modal and cannot process features from the audio modal. Therefore, this disclosure proposes a cross-modal model, EXT-LXMERT, which can process features from the visual, text, and audio modalities.

[0114] Figure 3 This is a schematic diagram of the structure of a cross-modal model according to an embodiment of this disclosure. Figure 3 As shown, the cross-modal model 30 includes a visual encoder 31, an audio encoder 32, a first fusion module 33, a text encoder 34, a second fusion module 35, and a cross-modal encoder 36.

[0115] In some embodiments, the method for obtaining the first fusion feature includes the following steps S31-S35.

[0116] In step S31, the m-th visual modality feature is encoded using a visual encoder to obtain the visual encoded feature.

[0117] For example, suppose the first One feature to be processed It is a visual modal feature, utilizing Figure 3 The visual encoder 31 shown is paired with the first... One feature to be processed Encode the corresponding visual encoded features. .

[0118] It should be noted that the visual encoder 31 can be an object relation encoder, which can perform self-attention interaction on image region features to generate image representations with spatial context association.

[0119] In step S32, the audio encoder is used to encode the nth audio modal feature to obtain the audio encoded feature.

[0120] For example, suppose the first One feature to be processed It is an audio modal feature, utilizing Figure 3 The audio encoder 32 shown is paired with the first... One feature to be processed Encode the audio to obtain the corresponding audio coding features. .

[0121] It should be noted that the audio encoder 32 can be constructed based on a pre-trained LSTM-AudioEncoder (Long Short-Term Memory Audio Encoder). By using the long short-term memory network to perform temporal modeling of long sequence acoustic features, it can effectively capture the prosody and emotional features in speech, and is suitable for depression screening tasks.

[0122] In step S33, the visual coding features are subjected to linear projection processing to obtain the projected features.

[0123] In step S34, the audio coding features are pooled to obtain pooled features.

[0124] In step S35, the first fusion feature is obtained based on the projection feature and the pooling feature.

[0125] In some embodiments, a weighted sum of the projection features and pooling features is calculated to obtain a first fused feature.

[0126] For example, using Figure 3 The first fusion module 33 shown pairs visual encoded features and audio coding features To integrate.

[0127] For example, suppose a visual encoder is used to encode the m-th visual modality feature, resulting in the following visual encoded feature: The audio encoder is used to encode the nth audio modal feature, resulting in the audio encoded feature. Then the first fusion feature As shown in formula (5).

[0128] (5)

[0129] in, and To encode visual features Learnable parameters for linear projection. It is a learnable scaling factor. This is the global average pooling function.

[0130] It's important to note that the visual encoded features output by the visual encoder and the audio encoded features output by the audio encoder have different dimensions. Therefore, a linear projection layer is used to linearly project the visual encoded features to facilitate feature dimension matching with the audio encoded features. Pooling the audio encoded features facilitates the extraction of the global sentiment context, ensuring semantic alignment between the sentiment in the visual encoded features and the audio features.

[0131] By using the methods described in the above embodiments, a first fusion feature is obtained, thereby achieving the fusion of visual modal features and audio modal features, which improves the model's understanding ability, robustness, and generalization ability.

[0132] In step 15, the k-th text modality feature and the n-th audio modality feature in the features to be processed are fused to obtain the second fused feature. K represents the total number of visual modal features.

[0133] In some embodiments, the method for obtaining the second fusion feature includes the following steps S41-S43.

[0134] In step S41, the k-th text modal feature is encoded using a text encoder to obtain the text encoded feature.

[0135] For example, suppose the first One feature to be processed It is a text modal feature, utilizing Figure 3 The text encoder 34 shown is paired with the first... One feature to be processed Encode the text to obtain the corresponding text encoding features. .

[0136] It should be noted that the text encoder is capable of performing self-attention modeling on the input sequence and outputting a context-dependent semantic representation.

[0137] In step S42, the audio encoder is used to encode the nth audio modal feature to obtain the audio encoded feature.

[0138] For example, suppose the first One feature to be processed It is an audio modal feature, utilizing Figure 3 The audio encoder 32 shown is paired with the first... One feature to be processed Encode the audio to obtain the corresponding audio coding features. .

[0139] In step S43, the text encoding features and audio encoding features are fused to obtain the second fused feature.

[0140] For example, using Figure 3 The second fusion module 35 shown pairs text encoding features. and audio coding features The fusion is performed. The second fusion module 35 is an adapter based on the Transformer architecture.

[0141] For example, suppose a text encoder is used to encode the k-th text modal feature, resulting in the following text encoded feature: The audio encoder is used to encode the nth audio modal feature, resulting in the audio encoded feature. Then the second fusion feature As shown in formula (6).

[0142] (6)

[0143] in, It is a lightweight cross-modal adapter based on the Transformer architecture.

[0144] It's important to note that audio contains paralinguistic information beyond the transcribed text, such as intonation and emotion. A cross-modal adapter is used to process the concatenated features of the text-encoded features and audio-encoded features, ensuring semantic alignment between the text-encoded features output by the text encoder and the audio-encoded features output by the audio encoder.

[0145] The method designed in the above embodiments yields a second fusion feature, which integrates audio modal features and text modal features, thereby improving the model's understanding, robustness, and generalization ability.

[0146] In step 16, the first fusion feature and the second fusion feature are processed using a cross-modal model with mutual attention mechanism to obtain cross-modal fusion features.

[0147] In some embodiments, the mutual attention mechanism includes a first sub-mechanism of querying corresponding visual modal features and text modal features using audio modal features, a second sub-mechanism of querying corresponding audio modal features using visual modal features and text modal features, a third sub-mechanism of querying corresponding text modal features using visual modal features, and a fourth sub-mechanism of querying corresponding visual modal features using text modal features.

[0148] In some embodiments, a cross-modal encoder in a cross-modal model with a mutual attention mechanism is used to process the first fused feature and the second fused feature to obtain the cross-modal fused feature.

[0149] For example, using Figure 3 The cross-modal encoder 36 shown processes the first fusion feature and the second fusion feature.

[0150] It's important to note that the original LXMERT model's cross-modal encoder only includes a bidirectional interactive attention mechanism between visual and language modes. In contrast, the proposed cross-modal model, EXT-LXMERT, supports deep fusion of visual, text, and audio modalities. The cross-modal model adds audio-visual / text attention and visual / text-audio attention. In audio-visual / text attention, audio features are used as queries to retrieve relevant visual and text features. In visual / text-audio attention, visual and text features are used as queries to retrieve relevant audio features, achieving bidirectional cross-attention interaction among the three modalities. Within the cross-modal model, visual, text, and audio features engage in multi-level, iterative information exchange, thereby achieving cross-modal semantic alignment and deep fusion.

[0151] It should also be noted that cross-modal models can fuse any number of features from multiple modalities.

[0152] In some embodiments, a cross-modal model with a mutual attention mechanism is used to process the first fused feature and the second fused feature to obtain multiple cross-modal fused sub-features. These multiple cross-modal fused sub-features are then concatenated to obtain the cross-modal fused feature.

[0153] For example, suppose the t-th cross-modal fusion sub-feature is ,in Then cross-modal fusion features As shown in formula (7).

[0154] (7)

[0155] For example, suppose the t-th cross-modal fusion sub-feature The number of frames is Then cross-modal fusion features Frame count As shown in formula (8).

[0156] (8)

[0157] It should be noted here that cross-modal fusion features Frame count The core scale used to determine the input of the behavioral pattern analysis model. The behavioral pattern analysis model will process a length of... sequence.

[0158] In step 17, the cross-modal fusion features are processed using a behavioral pattern analysis model to obtain the depression screening results.

[0159] In some embodiments, a preset mask matrix is ​​used to mask the cross-modal fusion features to obtain masked fusion features. A behavioral pattern analysis model is then used to process the masked fusion features to obtain depression screening results.

[0160] In some embodiments, the masks of each cross-modal fusion sub-feature in a plurality of cross-modal fusion sub-features are concatenated to obtain a preset mask matrix.

[0161] For example, suppose the t-th cross-modal fusion sub-feature The mask is Then the preset mask matrix As shown in formula (9).

[0162] (9)

[0163] It should be noted here that the t-th cross-modal fusion sub-feature mask It can be a binary mask.

[0164] For example, suppose fragment The cross-modal fusion features are The preset mask matrix is Then the fragment Depression screening results As shown in formula (10).

[0165] (10)

[0166] in, This is the set of learnable parameters for the model, including weights in all modal encoders, modal conditional embeddings, and attention weights and feedforward network weights in the model. This is a behavioral pattern analysis model.

[0167] It should be noted that the preset mask matrix is ​​used to mark the validity of each modality in the frame.

[0168] In step 18, the loss value is determined based on the depression screening results and video sample annotation information.

[0169] In step 19, the cross-modal model and the behavior pattern analysis model are trained using the loss value.

[0170] In some embodiments, the model parameters are optimized by minimizing the cross-entropy loss function.

[0171] For example, suppose fragment The cross-modal fusion features are The preset mask matrix is The video sample annotation information is as follows The optimized model parameters As shown in formula (11).

[0172] (11)

[0173] in, The loss function of the model. Let be the expected function. This is the function to be minimized.

[0174] It should be noted that formula (11) means that by traversing A videos in the dataset and all segments of each video, the expected value of the loss function is calculated, and the optimal model parameters are found by minimizing the expected value of the loss function.

[0175] It's also worth noting that the dataset used for model training included the E-DAIC-WOZ (Extended Distress Analysis Interview Corpus - Wizard of Oz) dataset, which contained 66 samples from depressed patients and 209 samples from healthy controls. Due to severe class imbalance in the E-DAIC-WOZ dataset, it was integrated with the AVEC2014 dataset to create an expanded dataset. The AVEC2014 dataset included 73 samples from depressed patients. The expanded dataset included 139 samples from depressed patients and 209 samples from healthy controls, alleviating the class imbalance problem. Furthermore, the AVEC2014 dataset contained various task scenarios, such as free reading and emotional stimulation, enhancing data diversity. After expanding the dataset, compared to using only the E-DAIC-WOZ dataset, the recall rate increased from 0.55 to 0.68, and... The value showed a significant improvement, indicating that the model significantly reduced the false negative rate for patients with depression. The hybrid training strategy exposed the model to more diverse nonverbal behavior patterns, thereby enhancing its adaptability to real-world scenarios.

[0176] The model training method described in the above embodiments detects each frame in the video sample, identifying each frame as a valid or invalid frame. This divides the video sample into multiple segments with a predetermined duration, where the proportion of valid frames in each segment is greater than a threshold, improving the signal-to-noise ratio of the input data and thus increasing the efficiency of model training. Simultaneously, feature extraction is performed on each segment. Visual modal features are fused with audio modal features to obtain a first fused feature, and text modal features are fused with audio modal features to obtain a second fused feature. A cross-modal model with a mutual attention mechanism is used to process the first and second fused features to obtain a cross-modal fused feature. A behavioral pattern analysis model is then used to process the cross-modal fused feature to obtain the depression screening result. This allows the system to integrate information from the visual, text, and audio modalities, thereby improving the accuracy of the depression screening result.

[0177] Figure 4 This is a schematic flowchart of a multimodal depression screening method according to an embodiment of the present disclosure. In some embodiments, the following multimodal depression screening method is performed by an electronic device, including steps 41-47.

[0178] In step 41, each frame in the video to be detected is detected so that frames that reflect specific characteristics of depression are considered valid frames, and frames that do not reflect specific characteristics of depression are considered invalid frames.

[0179] In step 42, the video to be detected is divided into multiple segments with a predetermined duration, wherein the percentage of valid frames in each segment is greater than the percentage threshold.

[0180] In step 43, feature extraction is performed on each segment to obtain multiple features to be processed. Each feature to be processed is a visual modal feature, an audio modal feature, or a text modal feature.

[0181] In step 44, the m-th visual modality feature and the n-th audio modality feature in the features to be processed are fused to obtain the first fused feature. Wherein, M represents the total number of visual modal features. N represents the total number of audio modal features.

[0182] In step 45, the k-th text modal feature and the n-th audio modal feature in the features to be processed are fused to obtain the second fused feature. Wherein, K represents the total number of visual modal features.

[0183] In step 46, the first and second fused features are processed using a cross-modal model with a mutual attention mechanism to obtain the cross-modal fused features. The cross-modal model utilizes... Figure 1 The model is trained using the model training method involved in any of the embodiments.

[0184] In step 47, the cross-modal fusion features are processed using a behavioral pattern analysis model to obtain the depression screening results. The behavioral pattern analysis model utilizes... Figure 1 The model is trained using the model training method involved in any of the embodiments.

[0185] The multimodal depression screening method described in the above embodiments detects each frame in the video to be tested, identifying each frame as a valid or invalid frame. This divides the video into multiple segments of predetermined duration, where the proportion of valid frames in each segment exceeds a threshold, improving the signal-to-noise ratio of the input data and thus increasing the efficiency of depression screening. Simultaneously, feature extraction is performed on each segment. Visual modal features are fused with audio modal features to obtain a first fused feature, and text modal features are fused with audio modal features to obtain a second fused feature. A cross-modal model with a mutual attention mechanism is used to process the first and second fused features to obtain a cross-modal fused feature. A behavioral pattern analysis model is then used to process the cross-modal fused feature to obtain the depression screening result. This method integrates information from the visual, text, and audio modalities, thereby improving the accuracy of the depression screening results.

[0186] Figure 5 This is a flowchart illustrating a multimodal depression screening method according to another embodiment of the present disclosure, including steps 501-512.

[0187] In step 501, a video segmenter is used to detect each frame in the video to be detected, so that frames that can reflect the specific characteristics of depression are considered as valid frames and frames that cannot reflect the specific characteristics of depression are considered as invalid frames. The video to be detected is divided into multiple segments with a predetermined duration, wherein the proportion of valid frames in each segment is greater than the proportion threshold.

[0188] In step 502, a visual feature extractor is used to extract multiple semantic features of the visual modality from each segment.

[0189] In step 503, an audio feature extractor is used to extract multiple semantic features of the audio modality from each segment.

[0190] In step 504, a text feature extractor is used to extract multiple semantic features of the text modality from each segment.

[0191] In step 505, each visual semantic feature is mapped to a specified semantic space using a visual feature encoder to obtain multiple visual mapping features.

[0192] In step 506, each audio semantic feature is mapped to a specified semantic space using an audio feature encoder to obtain multiple audio mapping features.

[0193] In step 507, each text semantic feature is mapped to a specified semantic space using a text feature encoder to obtain multiple text mapping features.

[0194] In step 508, multiple visual mapping features are aligned to obtain the position embedding feature of each visual mapping feature. Based on the modal conditional embedding feature and position embedding feature of each visual mapping feature, each visual mapping feature is corrected to obtain multiple visual modal features.

[0195] In step 509, multiple audio mapping features are aligned to obtain the position embedding feature of each audio mapping feature. Based on the modal conditional embedding feature and position embedding feature of each audio mapping feature, each audio mapping feature is corrected to obtain multiple audio modal features.

[0196] In step 510, multiple text mapping features are aligned to obtain the position embedding feature of each text mapping feature. Based on the modal conditional embedding feature and position embedding feature of each text mapping feature, each text mapping feature is corrected to obtain multiple text modal features.

[0197] In step 511, the visual modal features and audio modal features are fused using a cross-modal model to obtain a first fused feature, and the text modal features and audio modal features are fused to obtain a second fused feature. The first fused feature and the second fused feature are processed to obtain the cross-modal fused feature.

[0198] In step 512, the cross-modal fusion features are processed using a behavioral pattern analysis model to obtain the depression screening results.

[0199] The multimodal depression screening method described in the above embodiments can improve the signal-to-noise ratio of the input data, thereby increasing the efficiency of depression screening. Simultaneously, it can fuse visual, textual, and audio information, thereby improving the accuracy of depression screening results.

[0200] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Figure 6 As shown, electronic device 60 can be represented in the form of a general computing device. Electronic device 60 includes memory 61, processor 62, and bus 63 connecting different system components.

[0201] The memory 61 may include, for example, system memory, non-volatile storage media, etc. System memory may store, for example, an operating system, applications, a boot loader, and other programs. System memory may include volatile storage media, such as random access memory (RAM) and / or cache memory. Non-volatile storage media may store, for example, instructions for corresponding embodiments of at least one of the following methods: a model training method and a multimodal depression screening method. Non-volatile storage media include, but are not limited to, disk storage, optical storage, flash memory, etc.

[0202] The processor 62 can be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete hardware components such as discrete gates or transistors. Accordingly, each module, such as the acquisition module, the calculation module, and the adjustment module, can be implemented by executing instructions in the central processing unit (CPU) running memory to perform the corresponding steps, or by implementing dedicated circuitry to perform the corresponding steps.

[0203] For example, processor 62 is configured for memory-based instruction execution implementation such as Figure 1 , Figure 4 and Figure 5 The method involved in any of the embodiments.

[0204] Bus 63 can use any of the various bus architectures. For example, bus architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, and the Peripheral Component Interconnect (PCI) bus.

[0205] The interfaces 64, 65, and 66 of the electronic device 60, as well as the memory 61 and processor 62, can be connected via bus 63. Input / output interface 64 provides a connection interface for input / output devices such as monitors, mice, and keyboards. Network interface 65 provides a connection interface for various networked devices. Storage interface 66 provides a connection interface for external storage devices such as floppy disks, USB flash drives, and SD cards.

[0206] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations thereof, can be implemented by computer-readable program instructions.

[0207] These computer-readable program instructions are provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable device to produce a machine, such that execution of the instructions by the processor produces means for implementing the functions specified in one or more boxes of the flowchart and / or block diagram.

[0208] These computer-readable program instructions may also be stored in a computer-readable storage medium. These instructions cause a computer to work in a particular manner to produce an article of manufacture, including instructions that implement the functions specified in one or more boxes in a flowchart and / or block diagram.

[0209] This disclosure may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.

[0210] This disclosure also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement... Figure 1 , Figure 4 and Figure 5 The method involved in any of the embodiments.

[0211] This disclosure also provides a computer program product, including computer instructions, wherein the computer instructions, when executed by a processor, implement as follows: Figure 1 , Figure 4 and Figure 5 The method involved in any of the embodiments.

[0212] The beneficial effects obtained by implementing the above embodiments of this disclosure are as follows:

[0213] (1) By detecting each frame of the video to be detected, valid frames and invalid frames are determined, and the video to be detected is divided into multiple segments, so that the proportion of valid frames in each segment is greater than the proportion threshold, which can improve the signal-to-noise ratio of the input data. This design enables the model to still extract effective features stably in scenarios where stroke patients have stiff facial expressions and unclear speech.

[0214] (2) The fractional positional embedding method is used to generate positional embedding features for each mapping feature, which solves the problem of temporal resolution mismatch caused by the difference in sampling rate of different modal data, thereby ensuring accurate synchronization of multimodal data. At the same time, the mapping features are corrected by modal conditional embedding features and positional embedding features, so that the model can distinguish features of different modalities and capture cross-modal spatiotemporal correlations.

[0215] (3) The model trained using the model training method provided in this disclosure, in the multimodal depression screening task The value can reach 0.74, while the baseline method The value was 0.53, which significantly improved the depression screening effect and verified the model's efficient modeling ability for complex multimodal associations.

[0216] In some embodiments, the functional units described above may be implemented as general-purpose processors, programmable logic controllers (PLCs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any suitable combination thereof for performing the functions described herein.

[0217] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0218] The description in this disclosure is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the disclosure to its forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of this disclosure and to enable those skilled in the art to understand this disclosure and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A model training method, comprising: Each frame in the video sample is detected so that frames that reflect depression-specific characteristics are considered valid frames, and frames that do not reflect the depression-specific characteristics are considered invalid frames. The video sample is divided into multiple segments with a predetermined duration, wherein the proportion of effective frames in each segment is greater than a proportion threshold. Feature extraction is performed on each segment to obtain multiple features to be processed, wherein each feature to be processed is a visual modal feature, an audio modal feature, or a text modal feature; The m-th visual modal feature and the n-th audio modal feature in the features to be processed are fused to obtain the first fused feature. M represents the total number of visual modal features. N is the total number of audio modal features; The k-th text modal feature and the n-th audio modal feature in the features to be processed are fused to obtain the second fused feature. K represents the total number of visual modal features; The first fused feature and the second fused feature are processed using a cross-modal model with a mutual attention mechanism to obtain cross-modal fused features; The cross-modal fusion features were processed using a behavioral pattern analysis model to obtain depression screening results; The loss value was determined based on the depression screening results and video sample annotation information. The cross-modal model and the behavior pattern analysis model are trained using the loss value.

2. The model training method according to claim 1, wherein, The mutual attention mechanism includes a first sub-mechanism that uses audio modal features to query corresponding visual modal features and text modal features, a second sub-mechanism that uses visual modal features and text modal features to query corresponding audio modal features, a third sub-mechanism that uses visual modal features to query corresponding text modal features, and a fourth sub-mechanism that uses text modal features to query corresponding visual modal features.

3. The model training method according to claim 1, wherein, The step of fusing the m-th visual modal feature and the n-th audio modal feature from the features to be processed to obtain the first fused feature includes: The m-th visual modality feature is encoded using a visual encoder to obtain the visual encoded feature; The nth audio modal feature is encoded using an audio encoder to obtain the audio encoded feature; The visual encoded features are subjected to linear projection processing to obtain projected features; The audio coding features are pooled to obtain pooled features; The first fusion feature is obtained based on the projection feature and the pooling feature.

4. The model training method according to claim 3, wherein, The step of obtaining the first fusion feature based on the projection feature and the pooling feature includes: The first fused feature is obtained by calculating the weighted sum of the projection feature and the pooling feature.

5. The model training method according to claim 1, wherein, The step of fusing the k-th text modal feature and the n-th audio modal feature in the features to be processed to obtain the second fused feature includes: The k-th text modal feature is encoded using a text encoder to obtain the text encoded feature; The nth audio modal feature is encoded using an audio encoder to obtain the audio encoded feature; The text encoding features and the audio encoding features are fused together to obtain the second fused feature.

6. The model training method according to claim 1, wherein, The detection of each frame in the video sample includes: The gaze characteristics, neuromotor characteristics, behavioral synchronization characteristics, and fundamental frequency abrupt change rate of the target subject in the i-th frame are detected, as well as the fundamental frequency change rate of the audio stream corresponding to the i-th frame. I is the total number of frames in the video sample; The scores for the gaze feature, the neuromotor feature, the behavioral synchronization feature, and the fundamental frequency mutation rate are calculated separately to obtain the total clinical information score. If the total score of the clinical information is greater than or equal to the score threshold, then the i-th frame is determined to be a valid frame; If the total score of the clinical information is less than the score threshold, then the i-th frame is determined to be an invalid frame.

7. The model training method according to claim 6, wherein, The score of the gaze feature is positively correlated with the gaze direction deviation of the target subject; The scores of the neuromotor characteristics are positively correlated with the activity intensity of the specified muscles; The score of the behavioral synchronization feature is positively correlated with the degree of synchronization of the mouth movements or facial expressions when the target subject's speech rhythm is disordered. The score of the fundamental frequency mutation rate is determined by the relationship between the fundamental frequency mutation rate and a predetermined mutation rate threshold.

8. The model training method according to claim 7, wherein, The designated muscles include at least one of the orbicularis oculi, zygomaticus major, and mandibular muscles of the target body.

9. The model training method according to claim 6 further includes: If at least one of the gaze feature, neuromotor feature, and behavioral synchronization feature of the target subject in the i-th frame cannot be detected, then the i-th frame is determined to be an invalid frame.

10. The model training method according to claim 6, wherein, The step of determining the i-th frame as an invalid frame if the total score of the clinical information is less than the score threshold includes: If the total score of the clinical information is less than the score threshold, detect whether there is speech rhythm disorder in the audio stream corresponding to the i-th frame; If there is no speech rhythm disorder in the audio stream corresponding to the i-th frame, then the i-th frame is determined to be an invalid frame.

11. The model training method according to claim 10, further comprising: If there is speech rhythm disorder in the audio stream corresponding to the i-th frame, then the i-th frame is determined to be a valid frame.

12. The model training method according to claim 1, wherein, The feature extraction for each segment includes: Extract multiple semantic features with different modalities from each of the segments; Each semantic feature is mapped to a specified semantic space, resulting in multiple mapped features; Align the multiple mapping features to obtain the position embedding feature of each mapping feature; Based on the modal conditional embedding feature and positional embedding feature of each mapping feature, each mapping feature is modified to obtain multiple features to be processed.

13. The model training method according to claim 12, wherein, The step of aligning the plurality of mapping features includes: In each segment, the number of frames corresponding to each mapping feature is detected to obtain multiple frame counts; The maximum number of frames among the plurality of frames is used as the base number of frames; The mapping ratio of each mapping feature is determined based on the ratio of the base frame number to the frame number corresponding to each mapping feature. Based on the mapping ratio of each mapping feature and the frame index of each mapping feature, a position embedding feature for each mapping feature is generated.

14. The model training method according to claim 12, wherein, The step of correcting each mapping feature based on its modality conditional embedding feature and positional embedding feature includes: The j-th mapping feature, its modal conditional embedding feature, and its positional embedding feature are added together to obtain the j-th feature to be processed. J represents the total number of mapping features.

15. The model training method according to any one of claims 1-14, wherein, The process of using a behavioral pattern analysis model to process the cross-modal fusion features to obtain depression screening results includes: The cross-modal fusion features are masked using a preset mask matrix to obtain masked fusion features; The mask fusion features are processed using the behavioral pattern analysis model to obtain the depression screening results.

16. A multimodal depression screening method, comprising: Each frame in the video to be tested is detected so that frames that reflect the specific characteristics of depression are considered valid frames, and frames that do not reflect the specific characteristics of depression are considered invalid frames. The video to be detected is divided into multiple segments with a predetermined duration, wherein the percentage of effective frames in each segment is greater than a percentage threshold. Feature extraction is performed on each segment to obtain multiple features to be processed, wherein each feature to be processed is a visual modal feature, an audio modal feature, or a text modal feature; The m-th visual modal feature and the n-th audio modal feature in the features to be processed are fused to obtain the first fused feature. M represents the total number of visual modal features. N is the total number of audio modal features; The k-th text modal feature and the n-th audio modal feature in the features to be processed are fused to obtain the second fused feature. K represents the total number of visual modal features; The first fused feature and the second fused feature are processed using a cross-modal model with a mutual attention mechanism to obtain cross-modal fused features, wherein the cross-modal model is trained using the model training method of any one of claims 1-15; The cross-modal fusion features are processed using a behavioral pattern analysis model to obtain depression screening results, wherein the behavioral pattern analysis model is trained using the model training method described in any one of claims 1-15.

17. An electronic device comprising: Memory; A processor, coupled to a memory, configured to implement the method as described in any one of claims 1-16 based on memory-stored instruction execution.

18. A computer-readable storage medium, wherein, A computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-16.

19. A computer program product comprising computer instructions, wherein the computer instructions, when executed by a processor, implement the method as described in any one of claims 1-16.

Images