Depressive emotion recognition method, device, computer equipment and storage medium

Abstract

The present application provides a method, device, computer equipment and storage medium for identifying depression, which relates to the field of machine learning technology. The method extracts face images of each frame in first video data in sample data; determines the target length of a voice segment corresponding to the face image based on the size of the face image, the acquisition rate and acquisition accuracy of first voice data; determines the voice image corresponding to the voice segment of the target length; fuses the voice image and the face image to determine a first fused image, realizes the fusion of different modal data, and inputs the first fused image into a preset depression recognition model for training. Compared with the existing method of first extracting features of different modal data and then fusing these features, the present application fuses different modal data and then extracts features in the fused image. The obtained features can simultaneously retain the complementary information between the modalities and the consistency information between the modalities, thereby improving the accuracy of identifying depression.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a method, apparatus, computer device and storage medium for identifying depression. Background Art

[0002] With the development of machine learning technology, depressive mood can be identified through behavioral characteristics such as voice signals and facial expressions. Depressive mood refers to the depression, sadness and low mood caused by different factors.

[0003] In related technologies, features corresponding to each modal data are usually extracted from different modal data (such as voice data, image data, etc.), and then the features corresponding to each modal data are fused according to weighted average, feature concatenation, stacking or other fusion strategies to determine the fusion features corresponding to the multimodal data; and depressive emotions are identified through the fusion features.

[0004] However, in the process of fusing the features corresponding to each modal data, due to the differences between the features corresponding to different modal data, some feature information (for example, detail information, timing information, etc.) in some modalities will be lost, resulting in low accuracy in identifying depressive emotions. Summary of the invention

[0005] The present application provides a method, apparatus, computer device and storage medium for identifying depression, which can solve the technical problem of low accuracy in identifying depression due to differences between features corresponding to different modal data during the process of fusing features corresponding to each modal data.

[0006] The embodiment of the present application is implemented as follows:

[0007] In a first aspect, the present invention provides a method for identifying depression, comprising the following steps:

[0008] Acquire sample data, the sample data including first voice data of each subject and first video data corresponding to the first voice data;

[0009] Extracting a face image from each frame of the first video data;

[0010] Determining a target length of a voice segment corresponding to the face image based on the size of the face image, the acquisition rate and the acquisition accuracy of the first voice data, where the voice segment is segment data in the first voice data;

[0011] Determine a speech image corresponding to a speech segment of a target length, wherein the size of the speech image is the same as the size of the face image;

[0012] Fusing the voice image and the face image to determine a first fused image;

[0013] Inputting the first fused image into a preset depression emotion recognition model to determine a trained depression emotion recognition model;

[0014] Determine a second fused image of the subject according to the second voice data of the subject and the second video data corresponding to the second voice data, wherein the second fused image is determined in the same manner as the first fused image;

[0015] Based on the second fused image and the trained depression emotion recognition model, a depression emotion recognition result of the subject is determined, and the depression emotion recognition result is used to characterize the corresponding level of the depression emotion of the subject.

[0016] In some possible implementations, before determining the target length of the speech segment corresponding to the face image, the method further includes:

[0017] Establishing a first relationship between the size of the face image and the acquisition rate, acquisition accuracy and target length of the voice segment of the first voice data;

[0018] Determine the target length of the speech segment corresponding to the face image, including:

[0019] Based on the first relationship, the size of the face image, and the acquisition rate and acquisition accuracy of the first voice data, a target length of the voice segment corresponding to the face image is determined.

[0020] In some possible implementations, establishing the first relationship includes:

[0021] Determine the number of pixels in the voice digital data corresponding to the voice segment, and the number of pixels is calculated by the following formula:

[0022] PD=t*sr*(sa / 8)

[0023] Where PD is the number of pixels, t is the target length, sr is the acquisition rate of the first speech data, and sa is the acquisition accuracy of the first speech data;

[0024] Establish a similar relationship between the number of pixels and the size of the face image,

[0025] PD≈W*H

[0026] Where PD is the number of pixels, W*H is the size of the face image, where W is the width of the face image and H is the length of the face image;

[0027] The first relationship is established by:

[0028] t*sr*(sa / 8)≈W*H.

[0029] In some possible implementations, determining a speech image corresponding to a speech segment of a target length includes:

[0030] Establishing a voice-expression synchronization relationship, which is used to characterize the relationship between the time synchronization and feature synchronization of the voice segment and the face image;

[0031] Determining an initial overlap length in the speech segment based on a speech-expression synchronization relationship and a preset number of face images;

[0032] Determine each speech segment in the first speech data based on the target length and the initial overlap length;

[0033] The voice digital data corresponding to the voice segment are arranged into a pixel matrix, the pixel matrix is ​​a voice image, and the elements in the pixel matrix correspond to the voice digital data.

[0034] In some possible implementations, establishing a voice expression synchronization relationship includes:

[0035] m*H*W≈[(1000-t) / (ts)+1]t*sr*(sa / 8)

[0036] Wherein, t is the target length, s is the initial overlapping length in the speech segment, sr is the acquisition rate of the first speech data, sa is the acquisition accuracy of the first speech data, m is the preset number of face images per second, and W*H is the size of the face image, where W is the width of the face image and H is the length of the face image.

[0037] In some possible implementations, inputting the first fused image into a preset depression emotion recognition model to determine a trained depression emotion recognition model includes:

[0038] Determine the evaluation indicators of the depression emotion recognition model, which at least include accuracy, precision, and recall;

[0039] Based on the evaluation indicators, determine the trained depression emotion recognition model;

[0040] Based on the trained depression emotion recognition model, the target overlap length is screened out from the initial overlap length.

[0041] In some possible implementations, after obtaining the sample data, the following steps are further included:

[0042] Slice the first voice data by different numbers of slices, and determine voice slices corresponding to the slices, wherein the length of the voice slices is greater than the length of the voice segment;

[0043] Preprocessing the voice slices to determine the preprocessed voice slices, where the preprocessing at least includes filtering, and the filtering is used to filter the voice slices;

[0044] Determine the speech image corresponding to the speech segment of the target length, including:

[0045] Based on the speech slices, the speech image corresponding to the speech segment of the target length is determined.

[0046] In a second aspect, the embodiment of the present application provides a depression recognition device, including a data acquisition module, a data processing module, a model training module and a testing module; wherein:

[0047] A data acquisition module, used to acquire sample data, the sample data including first voice data of each subject and first video data corresponding to the first voice data;

[0048] A data processing module, used for extracting a face image from each frame of the first video data;

[0049] The data processing module is further used to determine a target length of a voice segment corresponding to the face image based on the size of the face image, the acquisition rate and acquisition accuracy of the first voice data, where the voice segment is segment data in the first voice data;

[0050] The data processing module is further used to determine a speech image corresponding to a speech segment of a target length, wherein the size of the speech image is the same as the size of the face image;

[0051] The data processing module is further used to fuse the voice image and the face image to determine a first fused image;

[0052] A model training module, used for inputting the first fused image into a preset depression emotion recognition model to determine a trained depression emotion recognition model;

[0053] A testing module, used to determine a second fused image of the subject to be tested according to second voice data of the subject to be tested and second video data corresponding to the second voice data, wherein the second fused image is determined in the same manner as the first fused image;

[0054] The testing module is used to input the second fused image and the second fused image into the trained depression emotion recognition model to determine the recognition result of the subject to be tested, and the recognition result is used to characterize the corresponding level of the depression emotion of the subject to be tested.

[0055] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the depressive emotion recognition methods in the first aspect are implemented.

[0056] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any one of the depressive emotion recognition methods in the first aspect are implemented.

[0057] The embodiments of the present application provide a method, apparatus, computer device and storage medium for identifying depression. Through the scheme provided in the embodiments of the present application, a facial image in each frame image in the first video data in the sample data is extracted; and based on the size of the facial image, the acquisition rate and acquisition accuracy of the first voice data, a target length of the voice segment corresponding to the facial image is determined; a voice image corresponding to the voice segment of the target length is determined; the voice image and the facial image are fused to determine a first fused image, the first fused image is input into a preset depression recognition model, a trained depression recognition model is determined, and then a second fused image corresponding to the second voice data and the second video data of the subject is determined, and then, based on the second fused image and the trained depression recognition model, a recognition result of the subject is determined.

[0058] Through the solution provided in the embodiment of the present application, before training the depression recognition model, a first fused image is determined based on the first voice data and the first video data to achieve the fusion of different modal data (voice data and image data), the depression recognition model is trained, and the features corresponding to the first fused image are extracted. The complementary information between the modalities and the consistency information between the modalities can be retained at the same time. Compared with the existing method of extracting features of different modal data through a model and then fusing these features, the solution of the present application improves the accuracy of multi-modal data in identifying depression. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 A schematic diagram of a process for identifying depression provided in an embodiment of the present application is shown;

[0061] Figure 2 A schematic diagram showing a method for identifying depression emotions provided in an embodiment of the present application for determining a corresponding voice image for a voice segment is shown;

[0062] Figure 3 A schematic diagram of a process for determining a voice image in a method for identifying depression provided in an embodiment of the present application is shown;

[0063] Figure 4 A schematic diagram showing a first fused image determined by a speech segment in a method for identifying depression provided by an embodiment of the present application is shown;

[0064] Figure 5 A schematic diagram of a depression emotion recognition model based on a ResNet-18 network in an embodiment of the present application is shown;

[0065] Figure 6 A schematic diagram of a V-net network framework in an embodiment of the present application is shown;

[0066] Figure 7 A schematic diagram showing the principle of a method for identifying depression provided in an embodiment of the present application is shown;

[0067] Figure 8 A depressive emotion recognition device also provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0068] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0069] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0070] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0071] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0072] With the development of machine learning technology, depression can be identified through behavioral characteristics such as voice signals, facial expressions, and movements. Depression refers to the depression, sadness, and low emotions caused by different factors.

[0073] In related technologies, features corresponding to each modal data are usually extracted from different modal data (such as voice data, image data, etc.), and then the features corresponding to each modal data are fused according to weighted average, feature concatenation, stacking or other fusion strategies to determine the fusion features corresponding to the multimodal data; and depressive emotions are identified through the fusion features.

[0074] In the process of fusing the features corresponding to each modal data, due to the differences between the features corresponding to different modal data, some feature information in some modalities will be lost, resulting in low accuracy in identifying depressive emotions; among them, some feature information such as detail information, time series information, etc.

[0075] The feature extraction process of each modal data may have different properties and characteristics. Fusing the features of each modal data after independent processing may result in the loss of details and correlation information unique to some modal data. Since the interaction and collaborative information between the modal data are not effectively integrated in the feature extraction stage, the subsequent fusion may not be able to fully utilize the complementarity and consistency between the modalities.

[0076] To solve the above problems, the embodiments of the present application provide a method, apparatus, computer device and storage medium for identifying depression. Through the scheme provided in the embodiments of the present application, a facial image in each frame image in the first video data in the sample data is extracted; and based on the size of the facial image, the acquisition rate and acquisition accuracy of the first voice data, the target length of the voice segment corresponding to the facial image is determined; the voice image corresponding to the voice segment of the target length is determined; the voice image and the facial image are fused to determine a first fused image, the first fused image is input into a preset depression recognition model, the trained depression recognition model is determined, and then after determining a second fused image corresponding to the second voice data and the second video data of the subject, the recognition result of the subject is determined based on the second fused image and the trained depression recognition model.

[0077] Through the solution provided in the embodiment of the present application, before training the depression recognition model, a first fused image is determined based on the first voice data and the first video data to achieve the fusion of different modal data (voice data and image data), the depression recognition model is trained, and the features corresponding to the first fused image are extracted. The complementary information between the modalities and the consistency information between the modalities can be retained at the same time. Compared with the existing method of extracting features of different modal data through a model and then fusing these features, the solution of the present application improves the accuracy of multi-modal data in identifying depression.

[0078] In addition, since the present application applies speech image and face image fusion before training the depression emotion recognition model when performing depression emotion recognition, the depression emotion recognition method disclosed in the present application can also be considered as a depression emotion recognition method that can be based on speech image signal fusion.

[0079] The following is a detailed description of the depression emotion recognition method and system of the embodiment of the present application in conjunction with the accompanying drawings.

[0080] Figure 1 A schematic diagram of a method for identifying depression provided in an embodiment of the present application is shown. Figure 1 As shown, a method for identifying depression emotions is provided in an embodiment of the present application.

[0081] The depression recognition method comprises the following steps:

[0082] S110 , obtaining sample data, where the sample data includes first voice data of each subject and first video data corresponding to the first voice data.

[0083] The first video data is data corresponding to the first voice data, that is, the video data and the voice data are acquired in time synchronization.

[0084] In some embodiments, whether the difference in time between each frame image in the video data and the corresponding voice data satisfies a preset difference is used to determine whether the video data and the voice data correspond to each other, wherein the preset difference may be data at the nanosecond level or microsecond level.

[0085] It should be understood that the sample data can be data from a public depression dataset; it can also be video data and voice data recorded by professional analysts through shooting and recording equipment during the actual process of depression emotion recognition. The sample data includes a depression group (as a positive example) and a control group (as a negative example).

[0086] In some embodiments, sample data is data obtained through specific stimulus materials in the form of questions, answers and interviews; stimulus materials refer to materials used to induce specific reactions or behaviors in subjects in psychology, neuroscience and other experimental studies. Stimulus materials can be different types of substances or information such as images, videos, text, audio, taste, smell, etc., or virtual environments such as some simulation scenes and virtual reality.

[0087] If the sample data is data in a public depression data set, the corresponding first voice data and first video data are corresponding. If the sample data is shot and recorded data, the synchronization of the first voice data and the first video data can be achieved through existing technologies.

[0088] In some embodiments, the sample data also includes the collection rate and collection accuracy of the first voice data.

[0089] In some embodiments, the sample data can also be effectively screened through a preset screening strategy. The preset screening strategy can be determined by whether the content in the first voice data is data related to depressive emotions. For example, the data obtained through questions and answers and interviews with specific stimulus materials may contain conversation data that eases the atmosphere. These data are invalid data and can be excluded, thereby effectively screening the sample data.

[0090] In some embodiments, the first voice data can be segmented and processed by different numbers of slices to determine the voice slices corresponding to each segmentation process, and the voice slices can be preprocessed to determine the preprocessed voice slices. The preprocessing at least includes filtering, and filtering is used to filter the voice slices.

[0091] Splitting and processing each first voice data by different numbers of slices means that each first voice data is split and processed multiple times, and each time the splitting and processing is performed by a different preset time length, that is, the first voice data is split into a different number of voice slices each time, wherein the preset time length can be 2 seconds to 6 seconds.

[0092] In the embodiment of the present application, the voice segment in the first voice data is fused with the image in the first video data, and therefore, the length of the voice slice is greater than the length of the voice segment.

[0093] The speech slices are preprocessed to determine the preprocessed speech slices, wherein the preprocessing at least includes filtering, and the filtering is used to filter the speech slices. The audio signals in the speech slices that are outside a preset range can be filtered, and the frequency range related to the evaluation of depressive mood is retained, so that the intensity of the high-frequency signal and the low-frequency signal components will not differ too much, so that all relevant information can be applied in subsequent analysis.

[0094] In some embodiments, preprocessing may be performed before segmentation, that is, preprocessing is performed on the first voice data to determine the preprocessed first voice data.

[0095] In some embodiments, preprocessing also includes endpoint detection, pre-emphasis, etc., wherein endpoint detection is to first determine the intersection of the silent segment and the voiced segment in the first voice data, and then segment the first voice data according to the intersection, retain the voiced segment, and remove the silent segment. For the features in the first voice data, which only appear in the voiced segment, endpoint detection can be used to remove the silent segment and improve the accuracy of the data to be trained; pre-emphasis is to improve the resolution of the high-frequency signal in the first voice data. For the first voice data, the consonants contain more information, and the frequency of the consonants is higher. By amplifying the high-frequency signal, more information can be obtained.

[0096] It should be understood that the voice slice and the first voice data have the same true label, and the true label is used to characterize the true depressive emotion score of the sample data.

[0097] S120: extracting a face image from each frame of the first video data.

[0098] The face image extracted from each frame of the first video data can be extracted using the OpenFace neural network face recognition toolkit to extract the face image containing face information.

[0099] In some embodiments, the first video data can be segmented into video segments of the same length as the voice slices. For each video segment, the image is extracted frame by frame through the OpenCV tool library, and then face detection is performed to determine the most important facial part in the image as a facial image. The size of the facial image needs to meet a preset image size, for example, 112*112.

[0100] It should be understood that the face image and the first video data have the same true label, and the true labels of the first video data and the first voice data are the same, that is, both are true labels of the sample data.

[0101] S130: Determine a target length of the voice segment corresponding to the face image based on the size of the face image, the acquisition rate and the acquisition accuracy of the first voice data.

[0102] The voice segment is segment data in the first voice data.

[0103] The target length of the voice segment corresponding to the face image can be determined by the size of the face image, the acquisition rate and the acquisition accuracy of the first voice data. After the target length is determined, the voice segment corresponding to the face image can be obtained through the target length to achieve the correspondence between image and voice.

[0104] In order to achieve the fusion of image data and voice data, it is necessary to first determine the target length of the voice segment corresponding to the face image.

[0105] In some embodiments, before step 130, a first relationship is established between the size of the facial image and the acquisition rate, acquisition accuracy and target length of the voice segment of the first voice data; then, based on the first relationship, the size of the facial image, the acquisition rate and acquisition accuracy of the first voice data, the target length of the voice segment corresponding to the facial image is determined.

[0106] The establishment of the first relationship includes the following steps:

[0107] First, determine the number of pixels in the voice digital data corresponding to the voice segment. The number of pixels is calculated by the following formula:

[0108] PD=t*sr*(sa / 8)

[0109] Wherein, PD is the number of pixels, t is the target length, sr is the acquisition rate of the first speech data, and sa is the acquisition accuracy of the first speech data.

[0110] Because the speech image to be determined is a grayscale image, the value range of each pixel in the grayscale image is 0-255, that is, the element in the pixel matrix corresponding to each pixel is 8 bits, which can satisfy the value range of the image pixel point is 0-255. In the above formula, 8 corresponds to 8 bits.

[0111] Secondly, establish a similar relationship between the number of pixels and the size of the face image, as follows:

[0112] PD≈W*H

[0113] Where PD is the number of pixels, W*H is the size of the face image, where W is the width of the face image and H is the length of the face image.

[0114] Furthermore, combining the above two formulas, the first relationship is established by the following formula:

[0115] t*sr*(sa / 8)≈W*H.

[0116] By modifying the above formula, the target length t of the speech segment corresponding to the face image can be obtained. For the value of the target length, if the initial result of the target length is not an integer, the rounded result can be used as the calculation result of the target length.

[0117] In some embodiments, if the initial result of the target length is not an integer, some decimals in the initial result may be retained, and the progress may be maintained to the nanosecond level.

[0118] Based on the first relationship, the size of the face image, the acquisition rate and acquisition accuracy of the first voice data, the target length of the voice segment corresponding to the face image is determined. As an example, the size of the face image is 112*112, the acquisition rate of the first voice data is 44.1KHZ, and the acquisition accuracy is 16bit, then the corresponding first relationship can be expressed as:

[0119]

[0120] The corresponding target length t≈142ms can be determined.

[0121] S140: Determine the speech image corresponding to the speech segment of the target length.

[0122] The size of the voice image is the same as that of the face image, so that the voice image can be integrated with the face image. It should be understood that the voice image is determined by the voice data corresponding to the face image in time.

[0123] In some embodiments, if the first voice data is not segmented, a voice image corresponding to a voice segment of a target length is determined from the first voice data. In some embodiments, if the first voice data is segmented into voice slices, a voice image corresponding to a voice segment of a target length is determined from each voice slice.

[0124] In some embodiments, determining the voice image corresponding to the voice segment of the target length includes converting the voice segment of the target length into the corresponding voice digital data, and determining the voice image corresponding to the voice digital data, that is, converting the voice segment (analog signal) into voice digital data, and converting the voice digital data into the corresponding voice image. The voice segment is converted into voice digital data, which can be achieved by A / D conversion coding. The A / D conversion process can be achieved in many ways, which is not limited in this application.

[0125] In some embodiments, the first voice data or voice slice may be A / D converted, and then the corresponding voice digital data may be determined from the A / D converted data according to the target length, and then the voice image corresponding to the voice digital data may be determined.

[0126] Figure 2 A schematic diagram of determining a corresponding voice image for a voice segment in a method for identifying depression provided in an embodiment of the present application is shown. Figure 2 As shown, taking the above example, a 142ms voice segment is converted into corresponding voice digital data, the voice digital data is arranged into a pixel matrix, and then the voice digital data is converted into a voice image, and the size of the voice image is also 112*112.

[0127] In some embodiments, the process of converting the voice segment into voice digital data may be performed in step 130 .

[0128] In some embodiments, during the audio and video fusion process, the voice clip and the face image need to be synchronized. In order to maintain the time synchronization and information synchronization between the voice clip and the face image, Figure 3 FIG. 4 is a flow chart showing a method for identifying a speech image in a depression emotion recognition method provided in an embodiment of the present application. Figure 3 As shown, determining the speech image corresponding to the speech segment of the target length includes:

[0129] S141. Establishing a voice-expression synchronization relationship, where the voice-expression synchronization relationship is used to characterize the relationship between the time synchronization and feature synchronization of the voice segment and the face image.

[0130] Among them, m (preset number) face images are selected per second in the first video data, and the length of the voice segment in the corresponding 1 second is t ms, and the initial overlapping length in the voice segment is sms, and the voice expression synchronization relationship is established, including:

[0131] m*H*W≈[(1000-t) / (ts)+1]t*sr*(sa / 8)

[0132] Wherein, t is the target length, s is the initial overlapping length in the speech segment, sr is the acquisition rate of the first speech data, sa is the acquisition accuracy of the first speech data, m is the preset number of face images per second, and W*H is the size of the face image, where W is the width of the face image and H is the length of the face image.

[0133] It should be understood that the unit of t and s in the above formula is ms, and the corresponding formula is (1000-t) / (ts)+1. If the unit of t and s in the above formula is s, the corresponding formula (1000-t) / (ts)+1 should be expressed as (1-t) / (ts)+1.

[0134] S142: Determine an initial overlapping length in the voice segment based on the voice expression synchronization relationship, the preset number of face images and the first relationship.

[0135] By modifying the formula in step 141, the initial overlap length s in the speech segment can be obtained. For the value of the initial overlap length, if the initial result of the initial overlap length is not an integer, the rounded result can be used as the calculation result of the initial overlap length.

[0136] In some embodiments, if the initial result of the initial overlap length is not an integer, some decimals in the initial result may be retained, and the progress may be maintained to the nanosecond level.

[0137] Based on the first relationship and the preset number of face images, the voice expression synchronization relationship can be modified, and then the initial overlap length in the voice segment can be determined. As an example, the acquisition frame rate of the first video data is 30FPS, that is, 30 frames of images are acquired per second. If the preset number m=10, and when the target length t≈142ms, the initial overlap length s≈48ms in the voice segment.

[0138] S143: Determine each speech segment in the first speech data based on the target length and the initial overlapping length.

[0139] It should be understood that based on the target length and the initial overlapping length, it is determined whether there are multiple voice segments. For example, the voice segment includes a first voice segment and a second voice segment. There is an overlapping part between the first voice segment and the second voice segment, that is, there is an overlap between the tail of the first voice segment and the front of the second voice segment. For example, when the target length t≈142ms, the initial overlapping length s≈48ms in the voice segment, the first voice segment is data from X to X+142ms, and the second voice segment is data from X+142ms-48ms to X+142ms-48ms+142ms.

[0140] S144, arranging the voice digital data corresponding to the voice segment into a pixel matrix, where the pixel matrix is ​​a voice image, and elements in the pixel matrix correspond to the voice digital data.

[0141] like Figure 2 As shown, the voice digital data corresponding to the voice segment are arranged into a pixel matrix, and the pixel matrix is ​​a voice image. It should be understood that the voice image is a pixel matrix of one channel.

[0142] like Figure 1 As shown, the method further includes: S150, fusing the voice image and the face image to determine a first fused image.

[0143] The speech image and the face image are fused to generate virtual data, namely, a first fused image, which is used as input data for model training.

[0144] In some embodiments, the face image is an RGB three-channel image, and fusing the synchronized voice image and the face image is equivalent to stacking a pixel matrix of one channel and the three-channel image to determine a first fused image, which is virtual 4D data.

[0145] Figure 4 FIG. 1 is a schematic diagram showing a method for identifying depression emotions provided in an embodiment of the present application, wherein a voice segment determines a first fused image. Figure 4 As shown, the RGB three-channel image of the face image and the one-channel matrix of the voice image are fused to obtain virtual 4D data.

[0146] After step 150, the pre-trained depression emotion recognition model can be used to initialize the parameters of the network through the weight parameters pre-trained on the large-scale data set. The pre-trained weight parameters refer to the weight parameters trained on the large-scale data set, which are usually obtained by supervised learning methods on the large-scale data set. These pre-trained weight parameters can be used to initialize the weight parameters of the neural network, thereby accelerating the training process of the neural network and improving the accuracy of the model. Among them, the large-scale data set is a data set known in the prior art.

[0147] For example, the ImageNet dataset, ImageNet is a large-scale image database containing more than 14 million images, each of which is annotated with at least one category label. ImageNet was created to help advance the fields of computer vision and machine learning, especially in tasks such as image classification, object detection, and image segmentation.

[0148] For example, a depression recognition model based on the ResNet-18 network is pre-trained with the ImageNet dataset. Some parameters are fine-tuned during the training process. The training set labels in the ImageNet dataset are classified according to the depression assessment indicator PHQ-8 scores, which are divided into five categories: minimal [0-4], mild [5-9], moderate [10-14], moderately severe [15-19] and severe [20-24], which can be divided into five levels: no depression, mild depression, moderate depression, moderate to severe depression, and severe depression. A depression recognition model based on the ResNet-18 network is used to train a 5-category model, and ten-fold cross validation can be used in the training process.

[0149] The embodiment of the present application improves the speed of model training and the accuracy of the model in identifying depressive emotions by utilizing the generalization ability of the pre-trained ResNet-18-based network model.

[0150] S160: Input the first fused image into a preset depression emotion recognition model to determine a trained depression emotion recognition model.

[0151] The depression recognition model can be an existing neural network model, or can be adjusted based on an existing neural network model; for example, the depression recognition model is a depression recognition model based on the ResNet-18 network. Due to the excellent performance of the ResNet-18 network in classification tasks, the recognition of depression can be adapted by fine-tuning the fully connected layer and classification head of the ResNet-18 network. The first fused image is used as the input of the depression recognition model based on the ResNet-18 network, and the first fused image has the true label of the sample data.

[0152] In some embodiments, Figure 5 A schematic diagram of a depression emotion recognition model based on a ResNet-18 network in an embodiment of the present application is shown. Figure 5 As shown, this is a depression emotion recognition model based on the ResNet-18 network.

[0153] In some embodiments, pre-information fusion is performed on the first fused image, and the obtained first fused image is pre-information fused through a V-net network to obtain pre-fused data, which is then input into a depression emotion recognition model to enable analysis of depression emotion recognition.

[0154] in, Figure 6 A schematic diagram of the V-net network framework in an embodiment of the present application is shown, Figure 6 As shown, the V-net network framework includes 1*1 convolution and channel convolution, and the first fused image is pre-fused to obtain pre-fused data.

[0155] The first fused image unifies two completely different signals, speech and image. That is, the first fused image carries a synchronized speech segment related to the face image, thereby realizing the fusion of speech and image. The speech information is expressed in the form of an image as the input format for training the depression recognition model. The first fused image is processed through a series of operations such as convolution and pooling, so that the depression recognition model learns the features.

[0156] In some embodiments, step 160 also includes: determining evaluation indicators of the depression emotion recognition model, where the evaluation indicators at least include accuracy, precision, and recall.

[0157] The predicted label of the prediction result output by the model is depression as a positive example, and the predicted label of the prediction result output by the model is normal as a negative example. Among them, TP (True Positive) represents the number of true positive examples, TN (True Negative) represents the number of true negative examples, FP (False Positive) represents the number of false positive examples, and FN (False Negative) represents the number of false negative examples.

[0158] It should be understood that the determination of TP, TN, FP and FN is determined by the following table:

[0159]

[0160] The evaluation indicators include:

[0161] Accuracy is used to indicate the number of samples that are correctly predicted as a percentage of the total number of samples. The formula is as follows:

[0162] Accuracy=(TP+TN) / (TP+FN+FP+TN);

[0163] Precision is used to predict the proportion of samples that are actually positive among the positive samples. The formula is as follows:

[0164] Precision = TP / (TP+FP);

[0165] Recall is the ratio of the number of samples predicted as positive to the total number of positive samples. The formula is as follows:

[0166] Recall = TP / (TP+FN);

[0167] The F1 score is the weighted harmonic mean of precision and recall, and the formula is as follows:

[0168] F1=2*(Precision*Recall) / (Precision+Recall).

[0169] Furthermore, based on the evaluation indicators, a trained depression emotion recognition model is determined; based on the trained depression emotion recognition model, a target overlapping length is screened out from each initial overlapping length, wherein multiple initial overlapping lengths can be obtained through multiple preset numbers, and a target overlapping length is screened out from each initial overlapping length, as well as a preset number corresponding to the target overlapping length.

[0170] S170 , determining a second fused image of the subject according to second voice data of the subject and second video data corresponding to the second voice data.

[0171] Among them, the method for determining the second fused image is the same as the method for determining the first fused image. For the second voice data and the second video data of the subject, the same method as determining the first fused image in the above steps 120 to 150 can be used to determine the corresponding second fused image based on the second voice data and the second video data. The second fused image is a fusion of voice and picture, and the method for determining the second fused image will not be repeated.

[0172] S180: Determine a depression emotion recognition result of the subject based on the second fused image and the trained depression emotion recognition model.

[0173] Among them, the depression emotion recognition result is used to characterize the corresponding level of the depression emotion of the subject.

[0174] In some embodiments, the second fused image is input into a trained depression emotion recognition model, and the recognition result of the subject is determined through the trained depression emotion recognition model based on the target parameters of the depression emotion recognition model (for example, target length, preset number of facial images, target overlapping length, etc.).

[0175] In some embodiments, the depression scores corresponding to the second voice data and the second video data of the subject correspond to the depression assessment index (PHQ-8) scores, and the PHQ-8 score ranges from 0 to 24 points. Every increase of 5 points increases one depression level, indicating no depression, mild depression, moderate depression, moderate to severe depression, and severe depression, respectively.

[0176] The above steps 170 to 180 are the prediction part of the depressive emotion recognition method. The prediction part includes first inputting the second voice data and the second video data, fusing the simultaneous data in the second voice data and the second video data, sending the fused image to the model fusion network for prediction, and outputting the recognition result.

[0177] Figure 7 A schematic diagram showing the principle of a method for identifying depression provided by an embodiment of the present application is shown in FIG. Figure 7As shown, the voice data (including the first voice data and the second voice data) is encoded into digital data and then converted into a one-channel image, and then superimposed with the corresponding face image in the synchronized video data (including the first video data and the second video data), and put into the depression recognition model based on the ResNet-18 network. The depression recognition model based on the ResNet-18 network has a certain generalization ability after pre-training, which improves the speed of network training and the recognition accuracy of the network. This process fuses two completely different data, voice and image, that is, the fused image (including the first fused image and the second fused image) carries the characteristic signals of synchronized voice and portrait, wherein the voice is expressed in the form of an image, the format of the input data of the model is unified, and the fused image is subjected to a series of operations such as convolution and pooling, so that the depression recognition model learns these features, thereby effectively identifying depression. This method can improve the validity of the data and improve the accuracy of the recognition of depression corresponding to the tester's data.

[0178] The embodiments of the present application provide a method, apparatus, computer device and storage medium for identifying depression. Through the scheme provided in the embodiments of the present application, a facial image in each frame image in the first video data in the sample data is extracted; and based on the size of the facial image, the acquisition rate and acquisition accuracy of the first voice data, a target length of the voice segment corresponding to the facial image is determined; a voice image corresponding to the voice segment of the target length is determined; the voice image and the facial image are fused to determine a first fused image, the first fused image is input into a preset depression recognition model, a trained depression recognition model is determined, and then a second fused image corresponding to the second voice data and the second video data of the subject is determined, and then, based on the second fused image and the trained depression recognition model, a recognition result of the subject is determined.

[0179] Through the solution provided in the embodiment of the present application, before training the depression recognition model, a first fused image is determined based on the first voice data and the first video data to achieve the fusion of different modal data (voice data and image data), the depression recognition model is trained, and the features corresponding to the first fused image are extracted. The complementary information between the modalities and the consistency information between the modalities can be retained at the same time. Compared with the existing method of extracting features of different modal data through a model and then fusing these features, the solution of the present application improves the accuracy of multi-modal data in identifying depression.

[0180] Based on the above depression emotion recognition method, the same technical concept is adopted. Figure 8 A depressive emotion recognition device is also provided in the embodiment of the present application. Figure 8As shown, the depression emotion recognition device 800 includes a data acquisition module 810, a data processing module 820, a model training module 830 and a testing module 840, wherein:

[0181] A data acquisition module, used to acquire sample data, the sample data including first voice data of each subject and first video data corresponding to the first voice data;

[0182] The data processing module is used to extract the face image in each frame image in the first video data; further used to determine the target length of the voice segment corresponding to the face image based on the size of the face image, the acquisition rate and acquisition accuracy of the first voice data, where the voice segment is the segment data in the first voice data; further used to determine the voice image corresponding to the voice segment of the target length, where the size of the voice image is the same as the size of the face image; further used to fuse the voice image and the face image to determine a first fused image;

[0183] A model training module, used for inputting the first fused image into a preset depression emotion recognition model to determine a trained depression emotion recognition model;

[0184] The test module is used to determine a second fused image of the subject under test based on the subject's second voice data and the second video data corresponding to the second voice data, wherein the second fused image is determined in the same manner as the first fused image; and is used to input the second fused image and the second fused image into a trained depression emotion recognition model to determine a recognition result of the subject under test, and the recognition result is used to characterize the level of depression of the subject under test.

[0185] In some embodiments, the data processing module also includes a first relationship establishing unit and a target length determining unit, that is, before the target length determining unit determines the target length of the voice segment corresponding to the face image, the first relationship establishing unit is used to establish a first relationship between the size of the face image and the acquisition rate, acquisition accuracy and target length of the voice segment of the first voice data; at this time, the target length determining unit is used to determine the target length of the voice segment corresponding to the face image based on the first relationship, the size of the face image, the acquisition rate and acquisition accuracy of the first voice data.

[0186] In some embodiments, the first relationship establishing unit includes a pixel determination subunit and a first relationship establishing subunit; wherein:

[0187] The pixel determination subunit is used to determine the number of pixels in the voice digital data corresponding to the voice segment. The number of pixels is calculated by the following formula:

[0188] PD=t*sr*(sa / 8)

[0189] Where PD is the number of pixels, t is the target length, sr is the acquisition rate of the first speech data, and sa is the acquisition accuracy of the first speech data;

[0190] The first relationship establishing subunit is used to establish a similarity relationship between the number of pixels and the size of the face image.

[0191] PD≈W*H

[0192] Where PD is the number of pixels, W*H is the size of the face image, where W is the width of the face image and H is the length of the face image;

[0193] The first relationship establishment subunit is also used to establish the first relationship through the following formula:

[0194] t*sr*(sa / 8)≈W*H.

[0195] In some embodiments, the data processing module further includes a voice expression synchronization relationship establishing unit and an execution unit, wherein:

[0196] A voice expression synchronization relationship establishing unit is used to establish a voice expression synchronization relationship, which is used to characterize the relationship between time synchronization and feature synchronization between voice segments and facial images; an execution unit is used to determine the initial overlapping length in the voice segments based on the voice expression synchronization relationship and a preset number of facial images; it is also used to determine each voice segment in the first voice data based on the target length and the initial overlapping length; it is also used to arrange the voice digital data corresponding to the voice segment into a pixel matrix, the pixel matrix is ​​a voice image, and the elements in the pixel matrix correspond to the voice digital data.

[0197] In some embodiments, the voice expression synchronization relationship establishing unit includes a voice expression synchronization relationship establishing sub-unit, and the voice expression synchronization relationship establishing sub-unit is used to establish the voice expression synchronization relationship, including:

[0198] m*H*W≈[(1000-t) / (ts)+1]t*sr*(sa / 8)

[0199] Wherein, t is the target length, s is the initial overlapping length in the speech segment, sr is the acquisition rate of the first speech data, sa is the acquisition accuracy of the first speech data, m is the preset number of face images per second, and W*H is the size of the face image, where W is the width of the face image and H is the length of the face image.

[0200] In some embodiments, the model training module includes a prediction and evaluation unit and a parameter determination unit, wherein the prediction and evaluation unit is used to determine the evaluation indicators of the depression emotion recognition model, and the evaluation indicators at least include accuracy, precision, and recall rate; based on the evaluation indicators, a trained depression emotion recognition model is determined; the parameter determination unit is used to filter out the target overlap length from the initial overlap length based on the trained depression emotion recognition model.

[0201] In some embodiments, the data acquisition module includes a speech processing unit, which is used to segment the first speech data by different numbers of slices, determine the speech slices corresponding to each segmentation process, wherein the length of the speech slice is greater than the length of the speech segment; pre-process the speech slices to determine the pre-processed speech slices, and the pre-processing at least includes filtering, which is used to filter the speech slices; at this time, the corresponding data processing unit is used to determine the speech image corresponding to the speech segment of the target length based on the speech slices.

[0202] It should be understood that the depressive emotion recognition device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example during training and recognition. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0203] Moreover, the depression emotion recognition device and the depression emotion recognition method provided in the above embodiment belong to the same concept, and the specific implementation process thereof is detailed in the above depression emotion recognition method embodiment, which will not be repeated here.

[0204] Based on the above-mentioned depression recognition method and using the same technical concept, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned depression recognition method when executing the computer program.

[0205] Based on the above-mentioned depression recognition method and using the same technical concept, an embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned depression recognition method are implemented.

[0206] The following paragraphs will compare and list the Chinese terms involved in this application specification and their corresponding English terms to facilitate reading and understanding.

[0207] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above discussion in some embodiments is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. According to the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A method for identifying depression, characterized in that: include: Acquire sample data, wherein the sample data includes first voice data of each subject and first video data corresponding to the first voice data; Extracting a face image from each frame of the first video data; Determining a target length of a voice segment corresponding to the face image based on the size of the face image, the acquisition rate and the acquisition accuracy of the first voice data, wherein the voice segment is segment data in the first voice data; Determine a speech image corresponding to the speech segment of the target length, wherein the size of the speech image is the same as the size of the face image; Fusing the voice image and the face image to determine a first fused image; Inputting the first fused image into a preset depression emotion recognition model to determine the trained depression emotion recognition model; Determine a second fused image of the subject according to second voice data of the subject and second video data corresponding to the second voice data, wherein the second fused image is determined in the same manner as the first fused image; Determine a depression emotion recognition result of the subject based on the second fused image and the trained depression emotion recognition model, wherein the depression emotion recognition result is used to characterize a level corresponding to the depression emotion of the subject; Before determining the target length of the voice segment corresponding to the face image, the method further includes: Establishing a first relationship between the size of the face image and the acquisition rate and acquisition accuracy of the first voice data and the target length of the voice segment; The step of determining a target length of a speech segment corresponding to the face image comprises: Determining a target length of a voice segment corresponding to the face image based on the first relationship, the size of the face image, and the acquisition rate and accuracy of the first voice data; The establishment of the first relationship includes: Determine the number of pixels in the voice digital data corresponding to the voice segment, and the number of pixels is calculated by the following formula: PD=t*sr*(sa / 8) Wherein, PD is the number of pixels, t is the target length, sr is the acquisition rate of the first voice data, and sa is the acquisition accuracy of the first voice data; Establishing a similarity relationship between the number of pixels and the size of the face image, PD≈W*H Wherein, PD is the number of pixels, W*H is the size of the face image, where W is the width of the face image, and H is the length of the face image; The first relationship is established by the following formula: t*sr*(sa / 8)≈W*H.

2. The method for identifying depression according to claim 1, characterized in that: The determining of the speech image corresponding to the speech segment of the target length includes: Establishing a voice-expression synchronization relationship, wherein the voice-expression synchronization relationship is used to characterize a relationship between the time synchronization and feature synchronization of the voice segment and the face image; Determining an initial overlapping length in the voice segment based on the voice-expression synchronization relationship, the preset number of face images, and the first relationship; Determining each speech segment in the first speech data based on the target length and the initial overlap length; The voice digital data corresponding to the voice segment are arranged into a pixel matrix, the pixel matrix is ​​the voice image, and the elements in the pixel matrix correspond to the voice digital data.

3. The method for identifying depression according to claim 2, characterized in that: The establishing of the voice expression synchronization relationship includes: m*H*W≈[(1000-t) / (ts)+1]t*sr*(sa / 8) In the formula, t is the target length, s is the initial overlapping length in the voice segment, sr is the acquisition rate of the first voice data, sa is the acquisition accuracy of the first voice data, m is the preset number of face images per second, W*H is the size of the face image, where W is the width of the face image and H is the length of the face image.

4. The method for identifying depression according to claim 2, characterized in that: The step of inputting the first fused image into a preset depression emotion recognition model to determine the trained depression emotion recognition model includes: Determining evaluation indicators of the depression emotion recognition model, wherein the evaluation indicators at least include accuracy, precision, and recall; Based on the evaluation index, determining the trained depression emotion recognition model; Based on the trained depression emotion recognition model, a target overlap length is screened out from the initial overlap lengths.

5. The method for identifying depression according to claim 1, characterized in that: After obtaining the sample data, the method further includes: Slice the first voice data by different numbers of slices, and determine voice slices corresponding to the slices, wherein the length of the voice slices is greater than the length of the voice segment; Preprocessing the voice slices to determine the preprocessed voice slices, wherein the preprocessing at least includes filtering, and the filtering is used to filter the voice slices; The determining of the speech image corresponding to the speech segment of the target length includes: Based on the speech slices, a speech image corresponding to the speech segment of the target length is determined.

6. A depressive emotion recognition device, characterized in that: include: A data acquisition module, used for acquiring sample data, wherein the sample data includes first voice data of each subject and first video data corresponding to the first voice data; A data processing module, used for extracting a face image from each frame of the first video data; The data processing module is further used to determine a target length of a voice segment corresponding to the face image based on the size of the face image, the acquisition rate and the acquisition accuracy of the first voice data, wherein the voice segment is segment data in the first voice data; The data processing module is further used to determine a voice image corresponding to the voice segment of the target length, wherein the size of the voice image is the same as the size of the face image; The data processing module is further used to fuse the voice image and the face image to determine a first fused image; A model training module, used for inputting the first fused image into a preset depression emotion recognition model to determine the trained depression emotion recognition model; A testing module, used for determining a second fused image of the subject according to second voice data of the subject and second video data corresponding to the second voice data, wherein the second fused image is determined in the same manner as the first fused image; The testing module is used to input the second fused image into the trained depression emotion recognition model to determine the recognition result of the subject to be tested, wherein the recognition result is used to characterize the level of depression emotion of the subject to be tested; The data processing module is further used for: Establishing a first relationship between the size of the face image and the acquisition rate and acquisition accuracy of the first voice data and the target length of the voice segment; The step of determining a target length of a speech segment corresponding to the face image comprises: Determining a target length of a voice segment corresponding to the face image based on the first relationship, the size of the face image, and the acquisition rate and accuracy of the first voice data; The establishment of the first relationship includes: Determine the number of pixels in the voice digital data corresponding to the voice segment, and the number of pixels is calculated by the following formula: PD=t*sr*(sa / 8) Wherein, PD is the number of pixels, t is the target length, sr is the acquisition rate of the first voice data, and sa is the acquisition accuracy of the first voice data; Establishing a similarity relationship between the number of pixels and the size of the face image, PD≈W*H Wherein, PD is the number of pixels, W*H is the size of the face image, where W is the width of the face image, and H is the length of the face image; The first relationship is established by the following formula: t*sr*(sa / 8)≈W*H.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the depressive emotion recognition method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the depressive emotion recognition method according to any one of claims 1 to 5 are implemented.

Images