Depressive emotional state data acquisition device, system, method and storage medium

Abstract

The present application relates to a device, system, method and storage medium for acquiring data on depressive emotional state. The device includes: a total voice data acquisition module, used to acquire voice data formed by a target user reading a specified sentence; a total voice feature recognition module, used to input the voice data into a preset total voice feature recognition model and output the total voice feature data of the target user; a first similarity determination module, used to determine a first similarity value, wherein the first similarity value is the similarity value between the total voice feature data of the target user and the total voice feature reference data of the depression user. The use of this device can accurately obtain the depressive emotional state data of the target user and save manual processing time.

Description

Technical Field

[0001] The present application relates to the technical field of computer data processing, and in particular to a device, system, method and storage medium for acquiring data on a depressive mood state. Background Art

[0002] Depression is a common mental illness. According to the "Depression and Other Common Mental Disorders" published by the World Health Organization in 2017, approximately 322 million people worldwide are affected by depression. According to the article "Prevalence of mental disorders in China: a cross-sectional epidemiological study" published by Professor Huang Yueqin and others in 2019, it is estimated that more than 90 million people in my country are currently affected by depression. Depression is the main cause of disability worldwide and the main factor in the global burden of disease. Depression seriously affects the quality of life and mental state of individuals, but in the current environment, the misdiagnosis rate is high, the recurrence rate is high, and there is a lack of sufficient well-trained relevant physicians for diagnosis and treatment.

[0003] Currently, the diagnosis of depression is mainly based on clinical questionnaires, psychologists' inquiries and observations, and subjective descriptions of test takers. It is easily influenced by doctors' bias and clinical experience, which affects the accuracy of the diagnosis. At the same time, it requires psychologists to spend a lot of time on diagnosis, which is inefficient. Summary of the invention

[0004] Based on this, it is necessary to provide a device, system, method and storage medium that can efficiently and accurately obtain the depressive emotional state data of the target user in response to the above technical problems.

[0005] A device for acquiring data on a depressive emotional state comprises: a total voice data acquisition module, used to acquire voice data formed by a target user reading aloud designated sentences, wherein the number of the designated sentences is multiple, and at least two of the designated sentences have different emotional stimulation types; a total voice feature recognition module, used to input the voice data into a preset total voice feature recognition model, and output the total voice feature data of the target user; a first similarity determination module, used to determine a first similarity value based on the total voice feature data of the target user, preset total voice feature reference data of a depression user, and preset total voice feature reference data of a healthy user, wherein the first similarity value is a similarity value between the total voice feature data of the target user and the total voice feature reference data of the depression user, and the depressive emotional state data includes the first similarity value or is determined based on the first similarity value.

[0006] In one embodiment, the device also includes a model training module, which includes: a training data acquisition unit, used to acquire voice data of two users with depression and voice data of two healthy users; a feature prediction unit, used to input the voice data of the two users with depression and the voice data of the two healthy users into the original total voice feature recognition model respectively, and output the corresponding total voice feature prediction results; a calculation unit, used to substitute each of the total voice feature prediction results into a preset first target loss function to obtain a first loss function value; a parameter adjustment unit, used to adjust the model parameters of the original total voice feature recognition model when the first loss function value does not meet the preset training stop condition, and instruct the first training data acquisition unit to re-acquire the voice data of the two users with depression and the voice data of the two healthy users; a training stop unit, used to obtain the total voice feature recognition model when the first loss function value meets the preset training stop condition.

[0007] In one embodiment, the device also includes a first feature reference data acquisition module, which includes: a first total speech processing unit, used to input the speech data of multiple depression users into the total speech feature recognition model respectively, and output the total speech feature data of multiple depression users; a first total mean calculation unit, used to obtain the total speech feature reference data of the depression users by calculating the average value of the total speech feature data of the multiple depression users; a second total speech processing unit, used to input the speech data of multiple healthy users into the total speech feature recognition model respectively, and output the total speech feature data of multiple healthy users; a second total mean calculation unit, used to obtain the total speech feature reference data of the healthy users by calculating the average value of the total speech feature data of the multiple healthy users.

[0008] In one embodiment, the device also includes a second feature reference data acquisition module, which includes: a first vector set determination unit, used to input the voice data of multiple depression users into the total voice feature recognition model respectively, output the total voice feature data of multiple depression users, and use the total voice feature data of the multiple depression users as the first vector set; a first intermediate vector determination unit, used to determine a first standard vector located in the middle of all vectors in the first vector set in the target coordinate system, and use the first standard vector as the total voice feature reference data of the depression users; a second vector set determination unit, used to input the voice data of multiple healthy users into the total voice feature recognition model respectively, output the total voice feature data of multiple healthy users, and use the total voice feature data of the multiple healthy users as the second vector set; a second intermediate vector determination unit, used to determine a second standard vector located in the middle of all vectors in the second vector set in the target coordinate system, and use the second standard vector as the total voice feature reference data of the healthy users.

[0009] In one embodiment, the first similarity determination module is based on Determine the first similarity value; wherein ALLS represents the first similarity value, PatternDA represents the total voice feature reference data of the depression user, PatternHA represents the total voice feature reference data of the healthy user, and PatternS represents the total voice feature data of the target user.

[0010] In one of the embodiments, the emotional stimulation types of the plurality of designated sentences include a sad emotional stimulation type, a neutral emotional stimulation type, and a positive emotional stimulation type.

[0011] In one of the embodiments, the device further includes: a segmented speech feature recognition module, which is used to input the speech data into a corresponding segmented speech feature recognition model according to the emotional stimulation type of the specified sentence, and output the segmented speech feature data of the target user corresponding to each emotional stimulation type; a second similarity determination module, which is used to determine a second similarity value based on the segmented speech feature data of the target user corresponding to the same emotional stimulation type and the segmented speech feature reference data of the depressed user, respectively, the second similarity value being the similarity value between the segmented speech feature data of the target user and the segmented speech feature reference data of the depressed user; a third similarity determination module, which is used to determine a third similarity value based on the segmented speech feature data of the target user corresponding to the same emotional stimulation type and the segmented speech feature reference data of the healthy user, the third similarity value being the similarity value between the segmented speech feature data of the target user and the segmented speech feature reference data of the healthy user; the depressive emotional state data also includes the second similarity value and the third similarity value, or is also determined based on the second similarity value and the third similarity value.

[0012] In one of the embodiments, the feature prediction unit is also used to input the similar voice data of the two depressed users and the similar voice data of the two healthy users into the original sub-speech feature recognition model respectively, and output the corresponding sub-speech feature prediction results; the calculation unit is also used to substitute each of the sub-speech feature prediction results into a preset second target loss function to obtain a second loss function value; when the first loss function value or the second loss function value does not meet the preset training stop condition, the parameter adjustment unit adjusts the model parameters of the original total speech feature recognition model or the model parameters of the original sub-speech feature recognition model, and instructs the second training data acquisition unit to re-acquire the similar voice data of the two depressed users and the similar voice data of the two healthy users; when the first loss function value and the second loss function value both meet the preset training stop condition, the training stop unit obtains the total speech feature recognition model and the sub-speech feature recognition model.

[0013] In one of the embodiments, the device also includes a third feature reference data acquisition module, and the third feature reference data acquisition module includes: a total voice feature acquisition unit for depressed users, which is used to input the voice data of depressed users into the total voice feature recognition model and output the total voice feature data of depressed users; a sub-voice feature acquisition unit for depressed users, which is used to input the voice data of depressed users corresponding to different emotional stimulation types into the corresponding sub-voice feature recognition model and output the sub-voice feature data of depressed users; a total voice feature acquisition unit for healthy users, which is used to input the voice data of healthy users into the total voice feature recognition model and output the total voice feature data of healthy users; a sub-voice feature acquisition unit for healthy users, which is used to input the voice data of healthy users corresponding to different emotional stimulation types into the corresponding sub-voice feature recognition model and output the total voice feature data of healthy users. The speech data of the healthy user is input into the corresponding speech feature recognition model, and the speech feature data of the healthy user is output; the fully connected network model training unit is used to take the total speech feature data of the depressed user, the total speech feature data of the healthy user, the speech feature data of the depressed user and the speech feature data of the healthy user as training input values, train the fully connected network model, and obtain the trained fully connected network model; the model parameter extraction unit is used to obtain the model parameters of the trained fully connected network model, and determine the total speech feature reference data of the depressed user, the total speech feature reference data of the healthy user, the speech feature reference data of the depressed user and the speech feature reference data of the healthy user according to the model parameters.

[0014] A system for acquiring data on a depressive emotional state comprises a server and a terminal; the terminal is used to collect voice data formed by a target user reading aloud designated sentences, and send the voice data to the server, wherein the number of the designated sentences is multiple, and at least two of the designated sentences have different emotional stimulation types; the server is used to acquire the voice data sent by the terminal, input the voice data into a preset total voice feature recognition model, output the total voice feature data of the target user, and determine a first similarity value based on the total voice feature data of the target user, preset total voice feature reference data of a depressed user, and preset total voice feature reference data of a healthy user, wherein the first similarity value is a similarity value between the total voice feature data of the target user and the total voice feature reference data of the depressed user, and the depressive emotional state data includes the first similarity value or is determined based on the first similarity value.

[0015] A method for acquiring depressive emotional state data comprises: acquiring speech data formed by a target user reading aloud designated sentences, wherein the number of the designated sentences is multiple, and at least two of the designated sentences have different emotional stimulation types; inputting the speech data into a preset total speech feature recognition model, and outputting the total speech feature data of the target user; determining a first similarity value based on the total speech feature data of the target user, preset total speech feature reference data of a depression user, and preset total speech feature reference data of a healthy user, wherein the first similarity value is a similarity value between the total speech feature data of the target user and the total speech feature reference data of the depression user, and the depressive emotional state data includes the first similarity value or is determined based on the first similarity value.

[0016] A computer-readable storage medium stores a computer program, which implements the following steps when executed by a processor: obtaining speech data formed by a target user reading a specified sentence, wherein the number of the specified sentences is multiple, and at least two of the specified sentences have different emotional stimulation types; inputting the speech data into a preset total speech feature recognition model, and outputting the total speech feature data of the target user; determining a first similarity value based on the total speech feature data of the target user, total speech feature reference data of a preset first emotional state user, and total speech feature reference data of a preset second emotional state user, wherein the first similarity value is a similarity value between the total speech feature data of the target user and the total speech feature reference data of the first emotional state user, and the emotional state data of the target user includes the first similarity value or is determined based on the first similarity value.

[0017] The above-mentioned depressive emotional state data acquisition device, system, method and storage medium use the voice data of the target user as the input value of the total voice feature recognition model. Since the designated sentences corresponding to the voice data have different and definite emotional stimulation types and have a priori and definite sentence content, the voice data formed by the target user reading the designated sentence can effectively avoid the influence of meaningless voice information and environmental noise on the neural network model during feature extraction. The first similarity value is determined comprehensively based on the total voice feature data of the target user and the total voice feature reference data of the depression user and the total voice feature reference data of the healthy users. Since the data processing process is not affected by the deviation of human judgment experience, the depressive emotional state data of the target user can be accurately obtained, which is beneficial to improving the accuracy of the depression diagnosis results and can save manual processing time. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a schematic diagram of functional modules of a device for acquiring data on a depressive mood state in one embodiment;

[0019] Figure 2 A schematic diagram of a functional unit involved in a model training module in one embodiment;

[0020] Figure 3 is a schematic diagram of functional units involved in a first characteristic reference data acquisition module in an embodiment;

[0021] Figure 4 is a schematic diagram of a subdivided speech feature recognition module in one embodiment;

[0022] Figure 5 is a schematic diagram of functional units involved in a second characteristic reference data acquisition module in one embodiment;

[0023] Figure 6 is a schematic diagram of functional units involved in a third characteristic reference data acquisition module in one embodiment;

[0024] Figure 7 A schematic diagram of a system for acquiring data on a depressive mood state in one embodiment;

[0025] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] The depressive mood state data acquisition device provided in the present application may have the following features: Figure 1 The functional modules shown in Figure 1 In the embodiment, the depression state data acquisition device 100 includes:

[0028] The total voice data acquisition module 101 is used to acquire voice data formed by a target user reading aloud designated sentences, where the number of designated sentences is multiple, and at least two designated sentences have different emotional stimulation types;

[0029] The total voice feature recognition module 102 is used to input the voice data into a preset total voice feature recognition model and output the total voice feature data of the target user;

[0030] The first similarity determination module 103 is used to determine a first similarity value based on the total voice feature data of the target user, the total voice feature reference data of the preset depression user and the total voice feature reference data of the preset healthy user. The first similarity value is the similarity value between the total voice feature data of the target user and the total voice feature reference data of the depression user. The depressive emotional state data includes the first similarity value or is determined based on the first similarity value.

[0031] About target users, depression users and healthy users. Target users refer to users whose depression status data is to be obtained. In some cases, others or the target users themselves may want to use depression status data to understand whether the target users suffer from depression or have depression tendencies. Depression users refer to people who have been diagnosed with depression, and healthy users refer to people who are diagnosed as not having depression.

[0032] About the voice data formed by the target user reading a designated sentence. The voice data corresponds to the designated sentence read aloud by the target user. Generally speaking, after the target user reads a designated sentence, a segment or a copy of audio information can be obtained. In some more specific embodiments, the audio information can be directly used as voice data and input into a preset overall voice feature recognition model; in other embodiments, the audio information can be preprocessed such as background noise removal to obtain the preprocessed audio information as voice data, and then input into the preset overall voice feature recognition model. It should be noted that the reading aloud mentioned in this article refers to the action of conveying the content of a designated sentence through pronunciation.

[0033] About designated sentences and emotional stimulation types. Designated sentences refer to sentences with fixed content and information. These sentences have fixed and corresponding emotional stimulation types. A designated sentence does not necessarily mean that its corresponding text content has only one period, but may also have multiple periods. In some more specific embodiments, a designated sentence may refer to a sentence with complete semantics. Generally speaking, the content of the designated sentence is preset, and the target user can obtain the content of the designated sentence by viewing the interface or listening to the sound. In addition, a designated sentence has an emotional stimulation type. Emotional stimulation types can generally include three types: negative emotional stimulation types, positive emotional stimulation types, and neutral emotional stimulation types. Having different emotional stimulation types generally refers to having two or more of these three types. A designated sentence with a negative emotional stimulation type can produce a negative emotional stimulation effect on ordinary people. For example, when the negative emotional stimulation type is a sad emotional stimulation type, the corresponding designated sentence can produce a sad emotional stimulation effect. When the negative emotional stimulation type is a fear emotional stimulation type, the corresponding designated sentence can produce a fear emotional stimulation effect. A designated sentence with a positive emotion stimulation type can produce a positive emotion stimulation effect on ordinary people. For example, when the positive emotion stimulation type is a happy emotion stimulation type, the corresponding designated sentence can produce a happy emotion stimulation effect. A designated sentence with a neutral emotion stimulation type has neither a negative emotion stimulation effect nor a positive emotion stimulation effect on ordinary people.

[0034] For ease of understanding, a simple example is given here. The content of a designated sentence with a negative emotional stimulus type (such as a sad emotional stimulus type) can be "Just now, he accidentally broke his favorite vase. When he was sorting out the pieces of the vase, he cut his hand. On the way to the flower shop, a thunderstorm suddenly broke out." The content of a designated sentence with a positive emotional stimulus type can be "Today the weather is clear and sunny. I went outing with a few friends. Everyone I met on the road was talking and laughing." The content of a designated sentence with a neutral emotional stimulus type can be "Today I read a book."

[0035] Of course, the emotional stimulation type can also be defined separately according to scientific methods as needed, and the corresponding designated sentence content can also use other designated sentences, which are not particularly limited here.

[0036] The solution of the present application limits the content of the voice data by specifying sentences, so that the duration of the target user's reading can be controlled, and thus the size of the voice data can be controlled; by specifying sentences, the content of the voice data can also be controlled in terms of content quality. Considering that if the target user is asked to answer questions or chat to form voice data, the uncertainty of the user's speech content and duration will have an uncertain impact on the size and quality of the voice data, which will ultimately affect the accuracy of the depressive emotional state data, the solution of the present application adopts the method of letting the target user read aloud specified sentences to form voice data. In addition, the voice data corresponding to the specified sentences with different emotional stimulation types have a lower correlation, which is also conducive to improving the accuracy of obtaining depressive emotional state data.

[0037] About the total speech feature recognition model. The total speech feature recognition model is essentially a neural network model. Its main function is to extract data that can generally represent the speech features of the target user from the input speech data. Its output value is the total speech feature data. Users in different emotional states will have different speech features in terms of speech speed, intonation, etc. By outputting the total speech feature data, it is possible to compare the difference between the target user's speech feature value and the speech feature value of a depressed user or / and the speech feature value of a healthy user, which is helpful for judging the target user's tendency towards a depressive emotional state.

[0038] About the first similarity value. The first similarity value is used to judge the tendency of the target user's emotional state. Through the total voice feature recognition model, the total voice feature data of the target user can be obtained. According to the total voice feature data of the target user and the preset total voice feature reference data of the depression user and the preset total voice feature reference data of the healthy user, the first similarity value can be determined, which can quantify the deviation between the emotional state of the target user and the emotional state of the depression user. Supplementary explanation: the preset total voice feature reference data of the depression user and the preset total voice feature reference data of the healthy user are two sets of reference data, which are used to quantify the emotional state of the depression user and the emotional state of the healthy user, respectively. The sources of these two sets of reference data can be diverse, and can generally be obtained by inputting the voice data of the depression user and the voice data of the healthy user into the total voice feature recognition model, but it does not rule out the situation where the data is obtained by setting and adjusting according to experience.

[0039] Regarding depressive emotional state data. In the present application, the depressive emotional state data is used to quantify or reflect the deviation between the emotional state of the target user and the emotional state of the depressed user. In some more specific embodiments, the first similarity value can be mapped to determine whether the target user has depression; of course, in some embodiments, the degree of depression tendency of the target user can also be judged based on the first similarity.

[0040] The above-mentioned depressive emotional state data acquisition device uses the voice data corresponding to the designated sentences with different emotional stimulation types as the input value of the total voice feature recognition model, so that different input values ​​have a lower correlation, and the designated sentences with different emotional stimulation types make the file size of the input value controllable and the content quality controllable, which is generally beneficial to obtain data that can accurately reflect the voice characteristics of the target user (that is, the total voice feature data of the target user); the first similarity value is determined by the total voice feature data of the target user and the total voice feature reference data of the depression user and the total voice feature reference data of the healthy user. Since the data processing process is not affected by the deviation of human judgment experience, accurate depressive emotional state data can be obtained.

[0041] In some embodiments, Figure 2 As shown, the depression state data acquisition device 100 further includes a model training module 201. The model training module 201 can be used to obtain a total speech feature recognition model. The model training module 201 includes:

[0042] A training data acquisition unit 2011 is used to acquire speech data of two users with depression and speech data of two healthy users;

[0043] The feature prediction unit 2012 is used to input the speech data of the two depression users and the speech data of the two healthy users into the original total speech feature recognition model, and output the corresponding total speech feature prediction results;

[0044] The calculation unit 2013 is used to substitute each total speech feature prediction result into a preset first target loss function to obtain a first loss function value;

[0045] The parameter adjustment unit 2014 is used to adjust the model parameters of the original total speech feature recognition model when the first loss function value does not meet the preset training stop condition, and instruct the training data acquisition unit 2011 to re-acquire the speech data of two depression users and the speech data of two healthy users;

[0046] The training stop unit 2015 is used to obtain the overall speech feature recognition model when the first loss function value meets the preset training stop condition.

[0047] In the above, "the corresponding total speech feature prediction results" refer to the total speech feature prediction results corresponding to users with depression and the total speech feature prediction results corresponding to healthy users.

[0048] The first objective loss function may adopt some loss functions in the prior art, or the loss functions specifically listed below, without conflicting with other technical aspects of the present application.

[0049] The preset training stop condition includes that the first loss function value is greater than a preset first loss threshold. The specific value of the first loss threshold can be determined according to actual needs.

[0050] The following will take the case where the training sentences corresponding to the speech data of two depressed users and the speech data of two healthy users have three types of emotional stimulation as an example to further illustrate the training process of the total speech feature recognition model.

[0051] When the training sentences corresponding to the speech data of two users with depression and the speech data of two healthy users have three types of emotional stimulation, the three types of emotional stimulation may be positive emotional stimulation type, negative emotional stimulation type and neutral emotional stimulation type. At this time, the first loss function value is LA, and the first loss function value can be calculated according to formula (1):

[0052]

[0053] In formula (1), α1 can be -0.5, β1 can be 0.5, and of course α1 and β1 can also be other values ​​as needed. cosine() means calculating the cosine similarity between input values. For example, cosine(ModelA1(Dp1, Dn1, Dz1), ModelA1(Dp2, Dn2, Dz2)) means taking ModelA1(Dp1, Dn1, Dz1) and ModelA1(Dp2, Dn2, Dz2) as input values ​​and calculating the cosine similarity between them. Dp1, Dn1 and Dz1 represent the three voice data formed by the first depression user reading aloud the training sentences with positive emotional stimulation type, the training sentences with negative emotional stimulation type and the training sentences with neutral emotional stimulation type; Dp2, Dn2 and Dz2 represent the three voice data formed by the second depression user reading aloud the training sentences with positive emotional stimulation type, the training sentences with negative emotional stimulation type and the training sentences with neutral emotional stimulation type; Hp1, Hn1 and Hz1 represent the three voice data formed by the first healthy user reading aloud the training sentences with positive emotional stimulation type, the training sentences with negative emotional stimulation type and the training sentences with neutral emotional stimulation type; Hp2, Hn2 and Hz2 represent the three voice data formed by the second healthy user reading aloud the training sentences with positive emotional stimulation type, the training sentences with negative emotional stimulation type and the training sentences with neutral emotional stimulation type. At the end of the training, the original total speech feature recognition model with the latest model parameters is regarded as the total speech feature recognition model. ModelA1() represents the original total speech feature recognition model, and the output value of the model is the corresponding total speech feature prediction results. Specifically, ModelA1(Dp1, Dn1, Dz1) represents the total speech feature prediction result of the first depression user output by the original total speech feature recognition model with Dp1, Dn1 and Dz1 as input values; ModelA1(Dp2, Dn2, Dz2) represents the total speech feature prediction result of the second depression user output by the original total speech feature recognition model with Dp2, Dn2 and Dz2 as input values; ModelA1(Hp1, Hn1, Hz1) represents the total speech feature prediction result of the first healthy user output by the original total speech feature recognition model with Hp1, Hn1 and Hz1 as input values; ModelA1(Hp2, Hn2, Hz2) represents the total speech feature prediction result of the second healthy user output by the original total speech feature recognition model with Hp2, Hn2 and Hz2 as input values.

[0054] It should be noted that the training sentences mentioned in the context may or may not be consistent with the specified sentences read by the target user. The training sentences are essentially sentences with specific content, which are read by depression users or healthy users, and such sentences also have corresponding emotional stimulation types.

[0055] In some other embodiments, those skilled in the art may, according to the same principle, use speech data corresponding to training sentences of two or more types of emotional stimulation for training, that is, it is not excluded that in some other embodiments, the type or quantity of ModelA1() input values ​​may be changed to train the overall speech feature recognition model.

[0056] In some embodiments, Figure 3 As shown, the depression emotional state data acquisition device 100 also includes a first feature reference data acquisition module 301. The first feature reference data acquisition module 301 is used to obtain the total voice feature reference data of depression users and the total voice feature reference data of healthy users. The first feature reference data acquisition module 301 includes:

[0057] The first total speech processing unit 3011 is used to input the speech data of multiple depression users into the total speech feature recognition model respectively, and output the total speech feature data of multiple depression users;

[0058] A first total mean value calculation unit 3012 is used to obtain total voice feature reference data of depression users by calculating the average value of total voice feature data of multiple depression users;

[0059] The second total voice processing unit 3013 is used to input the voice data of multiple healthy users into the total voice feature recognition model respectively, and output the total voice feature data of multiple healthy users;

[0060] The second total mean value calculation unit 3014 is used to obtain the total voice feature reference data of healthy users by calculating the average value of the total voice feature data of multiple healthy users.

[0061] The first feature reference data acquisition module 301 can acquire data that is representative and can reflect the overall voice features of users with depression and data that reflect the overall voice features of healthy users, and these data can be used as the overall voice feature reference data of users with depression and the overall voice feature reference data of healthy users. The following will take the case where the training sentences corresponding to the voice data of two users with depression and the voice data of two healthy users have three types of emotional stimuli as an example for further explanation.

[0062] When the training sentences corresponding to the speech data of the depression user and the speech data of the healthy user have three types of emotional stimulation, the three types of emotional stimulation may be positive emotional stimulation type, negative emotional stimulation type and neutral emotional stimulation type. At this time, PatternDA is used to represent the total speech feature reference data of the depression user, and PatternHA is used to represent the total speech feature reference data of the healthy user. PatternDA and PatternHA can be calculated according to formula (2) and formula (3) respectively.

[0063]

[0064]

[0065] n is a preset positive integer, m is a preset positive integer, and mean() represents the result of calculating the mean of the input values. For example Indicates As the input value, the result obtained by averaging the input value. Dpi, Dni and Dzi represent the three voice data formed by the depressed users reading aloud the training sentences with positive emotional stimulation type, the training sentences with negative emotional stimulation type and the training sentences with neutral emotional stimulation type respectively; Hpj, Hnj and Hzj represent the three voice data formed by the healthy users reading aloud the training sentences with positive emotional stimulation type, the training sentences with negative emotional stimulation type and the training sentences with neutral emotional stimulation type respectively. ModelA() represents the total voice feature recognition model; i and j represent the batches of voice data currently input into ModelA() respectively. ModelA(Dpi, Dni, Dzi) represents the total voice feature data of the i-th batch of depressed users obtained by inputting Dpi, Dni and Dzi as input values ​​into the total voice feature recognition model. ModelA(Hpj, Hnj, Hzj) represents the total voice feature data of the j-th batch of healthy users obtained by inputting Hpj, Hnj and Hzj as input values ​​into the total voice feature recognition model.

[0066] In some other embodiments, those skilled in the art may, according to the same principle, use speech data corresponding to training sentences of two or more types of emotional stimulation for training, that is, it is not excluded that in some other embodiments, the type or number of input values ​​of ModelA() can be determined according to the type or number of input values ​​of ModelA1().

[0067] In some embodiments, the first similarity determination module 103 determines the first similarity value according to formula (4):

[0068]

[0069] Among them, ALLS represents the first similarity value, PatternDA represents the total voice feature reference data of depression users, PatternHA represents the total voice feature reference data of healthy users, and PatternS represents the total voice feature data of the target user.

[0070] In some embodiments, PatternS=ModelA(Sp, Sn, Sz), Sp, Sn, and Sz respectively represent the speech data of the target user with a positive emotional stimulation type, the speech data of the target user with a negative emotional stimulation type, and the speech data of the target user with a neutral emotional stimulation type. ModelA(Sp, Sn, Sz) represents the total speech feature data of the target user obtained by inputting Sp, Sn, and Sz as input values ​​into the total speech feature recognition model.

[0071] In some embodiments, Figure 4 As shown, the depression state data acquisition device 100 also includes:

[0072] The subdivided speech feature recognition module 401 is used to input the speech data into the corresponding subdivided speech feature recognition model according to the emotional stimulation type of the specified sentence, and output the subdivided speech feature data corresponding to each emotional stimulation type of the target user;

[0073] A second similarity determination module 402 is used to determine a second similarity value based on the sub-speech feature data of the target user and the sub-speech feature reference data of the depression user corresponding to the same emotional stimulation type, wherein the second similarity value is a similarity value between the sub-speech feature data of the target user and the sub-speech feature reference data of the depression user;

[0074] A third similarity determination module 403 is used to determine a third similarity value according to the sub-speech feature data of the target user and the sub-speech feature reference data of the healthy user corresponding to the same emotional stimulation type, wherein the third similarity value is a similarity value between the sub-speech feature data of the target user and the sub-speech feature reference data of the healthy user;

[0075] The depressive emotional state data also includes a second similarity value and a third similarity value, or is also determined based on the second similarity value and the third similarity value.

[0076] According to the above description, in addition to obtaining the first similarity value, the depressive emotional state data acquisition device can also obtain the second similarity value and the third similarity value. The significance of the second similarity value and the third similarity value is that the tendency of the target user's emotional state can be judged from the perspective of the specific aspects of the voice features. Under this premise, the target user's depressive emotional state data can include the first similarity value, the second similarity value and the third similarity value at the same time, or it can be other values ​​obtained by calculating the first similarity value, the second similarity value and the third similarity value. When the emotional stimulus type includes three types of negative emotional stimulus type, positive emotional stimulus type and neutral emotional stimulus type, the sub-speech feature recognition model will have three corresponding types, namely ModelB(), ModelC() and ModelD() mentioned below, wherein ModelB() is used to represent the sub-speech feature recognition model with voice data of negative emotional stimulus type as input value; ModelC() is used to represent the sub-speech feature recognition model with voice data of positive emotional stimulus type as input value; ModelD() is used to represent the sub-speech feature recognition model with voice data of neutral emotional stimulus type as input value.

[0077] In some embodiments, the emotional stimulation types possessed by the plurality of designated sentences include a sad emotional stimulation type, a neutral emotional stimulation type, and a positive emotional stimulation type.

[0078] In some embodiments, the feature prediction unit 2012 is also used to input the similar voice data of two depressed users and the similar voice data of two healthy users into the original sub-speech feature recognition model, and output the corresponding prediction results of each sub-speech feature; the calculation unit 2013 is also used to substitute the prediction results of each sub-speech feature into a preset second target loss function to obtain a second loss function value; the parameter adjustment unit 2014 adjusts the model parameters of the original total speech feature recognition model or the model parameters of the original sub-speech feature recognition model when the first loss function value or the second loss function value does not meet the preset training stop condition, and instructs the second training data acquisition unit to re-acquire the similar voice data of two depressed users and the similar voice data of two healthy users; the training stop unit 2015 obtains the total speech feature recognition model and the sub-speech feature recognition model when both the first loss function value and the second loss function value meet the preset training stop condition.

[0079] The same type of voice data refers to voice data having the same emotional stimulation type. When the emotional stimulation type includes three types, namely, negative emotional stimulation type, positive emotional stimulation type and neutral emotional stimulation type, the same type of voice data also corresponds to three types.

[0080] Each sub-speech feature prediction result refers to the respective sub-speech feature prediction result.

[0081] It should be noted that the training of the speech feature recognition model and the training of the total speech feature recognition model described above can be the same in principle. The difference is that the input value of the total speech feature recognition model is speech data corresponding to two or more types of emotional stimulation, such as Dpi, Dni, Dzi, while the input value of a certain type of speech feature recognition model is speech data of one type of emotional stimulation. The preset training stop conditions include: the first loss function value is greater than the preset first loss threshold, and the second loss function value is greater than the preset second loss threshold. The specific values ​​of the first loss threshold and the second loss threshold can be determined according to actual needs.

[0082] Specifically, when the training sentences corresponding to the speech data of two users with depression and the speech data of two healthy users have an emotional stimulation type, the emotional stimulation type can be a positive emotional stimulation type, a negative emotional stimulation type, or a neutral emotional stimulation type. At this time, the second loss function value is LX, which can be calculated according to formula (5):

[0083]

[0084] α2 can take the value of -0.5, β2 can take the value of 0.5, and of course α2 and β2 can also take other values ​​as needed. cosine() represents the calculation of the cosine similarity value between the input values. DX1 represents the voice data formed by the first depressed user reading a training sentence corresponding to a certain type of emotional stimulation; DX2 represents the voice data formed by the second depressed user reading a training sentence corresponding to the above-mentioned type of emotional stimulation; HX1 represents the voice data formed by the first healthy user reading a training sentence corresponding to the above-mentioned type of emotional stimulation; HX2 represents the voice data formed by the second healthy user reading a training sentence corresponding to the above-mentioned type of emotional stimulation. It can be seen that DX1 and DX2 belong to the same type of voice data, and HX1 and HX2 belong to the same type of voice data. ModelX1() represents the original speech feature recognition model, and the output value of the model is the corresponding prediction result of each speech feature. Specifically, ModelX1(DX1) represents the prediction result of the speech features of the first depressed user after DX1 is input into the original speech feature recognition model as the input value; ModelX1(DX2) represents the prediction result of the speech features of the second depressed user after DX2 is input into the original speech feature recognition model as the input value; ModelX1(HX1) represents the prediction result of the speech features of the first healthy user after HX1 is input into the original speech feature recognition model as the input value; ModelX1(HX2) represents the prediction result of the speech features of the second healthy user after HX2 is input into the original speech feature recognition model as the input value.

[0085] At the end of the training, the original speech feature recognition model with the latest model parameters is considered as the speech feature recognition model.

[0086] More specifically, when one of the above-mentioned emotional stimulation types is a negative emotional stimulation type, the corresponding original speech feature recognition model ModelX1() is ModelB1(), and the corresponding second loss function value LX is specifically LB, and DX1, DX2, HX1 and HX2 are specifically Dn1, Dn2, Hn1 and Hn2 respectively; when one of the above-mentioned emotional stimulation types is a positive emotional stimulation type, the corresponding original speech feature recognition model ModelX1() is ModelC1(), and the corresponding second loss function value LX is specifically LC, and DX1, DX2, HX1 and HX2 are specifically Dp1, Dp2, Hp1 and Hp2 respectively. When one of the above-mentioned emotional stimulation types is a neutral emotional stimulation type, the corresponding original speech feature recognition model ModelX1() is ModelD1(), and the corresponding second loss function value LX is specifically LD, and DX1, DX2, HX1 and HX2 are specifically Dz1, Dz2, Hz1 and Hz2 respectively.

[0087] When the same type of voice data corresponds to the designated sentence of negative emotional stimulation type, the reference data of the speech characteristics of the depressed user is PatternND, the reference data of the speech characteristics of the target user is PatternNS, and the second similarity value is NDS; when the same type of voice data corresponds to the designated sentence of negative emotional stimulation type, the reference data of the speech characteristics of the depressed user is PatternPD, the reference data of the speech characteristics of the target user is PatternPS, and the second similarity value is PDS; when the same type of voice data corresponds to the designated sentence of negative emotional stimulation type, the reference data of the speech characteristics of the depressed user is PatternZD, the reference data of the speech characteristics of the target user is PatternZS, and the second similarity value is ZDS. Then, NDS, PDS and ZDS can be calculated according to formula (6), formula (7) and formula (8) respectively.

[0088] NDS=cosine(PatternND,PatternNS) (6)

[0089] PDS=cosine(PatternPD,PatternPS) (7)

[0090] ZDS=cosine(PatternZD,PatternZS) (8)

[0091] PatternND, PatternPD and PatternZD can be calculated according to formula (9), formula (10) and formula (11) respectively.

[0092]

[0093]

[0094]

[0095] ModelB() represents the speech feature recognition model when the same type of speech data corresponds to the specified sentence of negative emotional stimulation type, and also corresponds to ModelB1(); ModelC() represents the speech feature recognition model when the same type of speech data corresponds to the specified sentence of positive emotional stimulation type, and also corresponds to ModelC1(). ModelD() represents the speech feature recognition model when the same type of speech data corresponds to the specified sentence of neutral emotional stimulation type, and also corresponds to ModelD1(). ModelB(Dni) represents the speech feature data of the depressed user output by the speech feature recognition model with speech data of negative emotional stimulation type as input value input with Dni as input value; ModelC(Dpi) represents the speech feature data of the depressed user output by the speech feature recognition model with speech data of positive emotional stimulation type as input value input with Dpi as input value; ModelD(Dzi) represents the speech feature data of the depressed user output by the speech feature recognition model with speech data of neutral emotional stimulation type as input value input with Dzi as input value. For the meaning of symbols such as cosine(), mean() and Dni, please refer to the previous description. In some embodiments, PatternNS, PatternPS and PatternZD can be calculated according to formula (12), formula (13) and formula (14), respectively.

[0096] PatternNS=ModelB(Sn) (12)

[0097] PatternPS=ModelC(Sp) (13)

[0098] PatternZS=ModelD(Sz) (14)

[0099] For the meanings of Sn, Sp and Sz, please refer to the above description. ModelB(Sn) represents the target user's speech feature data output by the speech feature recognition model with Sn as the input value and the speech data with negative emotional stimulation type as the input value; ModelC(Sp) represents the target user's speech feature data output by the speech feature recognition model with Sp as the input value and the speech data with positive emotional stimulation type as the input value; ModelD(Sz) represents the target user's speech feature data output by the speech feature recognition model with Sz as the input value and the speech data with neutral emotional stimulation type as the input value.

[0100] Correspondingly, in some embodiments, when the same type of voice data corresponds to the designated sentence of the negative emotional stimulation type, the voice feature reference data of the healthy user is PatternNH, the voice feature data of the target user is PatternNS, and the third similarity value is NHS; when the same type of voice data corresponds to the designated sentence of the negative emotional stimulation type, the voice feature reference data of the healthy user is PatternPH, the voice feature data of the target user is PatternPS, and the third similarity value is PHS; when the same type of voice data corresponds to the designated sentence of the negative emotional stimulation type, the voice feature reference data of the healthy user is PatternZH, the voice feature data of the target user is PatternZS, and the third similarity value is ZHS. Then, NHS, PHS and ZHS can be calculated according to formula (15), formula (16) and formula (17) respectively.

[0101] NHS=cosine(PatternNH,PatternNS) (15)

[0102] PHS=cosine(PatternPH,PatternPS) (16)

[0103] ZHS=cosine(PatternZH,PatternZS) (17)

[0104] PatternNH, PatternPH and PatternZH can be calculated according to formula (18), formula (19) and formula (20) respectively.

[0105]

[0106]

[0107]

[0108] ModelB(Hnj) represents the divided speech feature data of healthy users output by the divided speech feature recognition model with Hnj as input value and speech data with negative emotional stimulation type as input value; ModelC(Hpj) represents the divided speech feature data of healthy users output by the divided speech feature recognition model with Hpj as input value and speech data with positive emotional stimulation type as input value; ModelD(Hzj) represents the divided speech feature data of healthy users output by the divided speech feature recognition model with Hzj as input value and speech data with neutral emotional stimulation type as input value. For the meaning of other parameters and symbols, please refer to the previous description.

[0109] In some embodiments, Figure 5 As shown, the depression state data acquisition device 100 further includes a second feature reference data acquisition module 501, and the second feature reference data acquisition module 501 includes:

[0110] The first vector set determination unit 5011 is used to input the speech data of multiple depression users into the total speech feature recognition model respectively, output the total speech feature data of the multiple depression users, and use the total speech feature data of the multiple depression users as the first vector set;

[0111] A first intermediate vector determining unit 5012 is used to determine a first standard vector located at the middle position of all vectors in the first vector set in the target coordinate system, and use the first standard vector as the total speech feature reference data of the depression user;

[0112] The second vector set determination unit 5013 is used to input the speech data of multiple healthy users into the total speech feature recognition model respectively, output the total speech feature data of the multiple healthy users, and use the total speech feature data of the multiple healthy users as the second vector set;

[0113] The second intermediate vector determination unit 5014 is used to determine a second standard vector located in the middle of all vectors in the second vector set in the target coordinate system, and use the second standard vector as the total speech feature reference data of the healthy user.

[0114] Corresponds to Figure 5 In the embodiment shown, the reference data such as PatternDA, PatternHA, PatternND, PatternPD, PatternZD, PatternNH, PatternPH and PatternZH are obtained in the following manner: when obtaining the total voice feature reference data of a depressed user, the total voice feature data of multiple depressed users can be obtained by calculating ModelA(Dpi, Dni, Dzi), and the total voice feature data of multiple depressed users is used as the first vector set, and then the first standard vector located in the middle of all vectors in the first vector set in the target coordinate system is determined, and the first standard vector is used as PatternDA. Specifically, a function named compress() is provided, and PatternDA can be calculated according to formula (21).

[0115] PatternDA=compress({x|x=ModelA(Dpi,Dni,Dzi)}) (21)

[0116] The principle of compress() is: take U as the first vector set for input, and the length of each vector element in U is equal. Assume that the dimension of U is V. In addition, provide a standard orthogonal basis set B, which is also a vector with V dimensions, and each vector has only one position of 1 and the rest are 0. Provide a set R for representing the projection of the target vector on the high-dimensional plane composed of each basis vector, assuming that x and y are the two outermost projection vectors of the projection of all vectors in U on the high-dimensional plane composed of vectors a and b, by executing For(a,b)in B×B, a! = b, and R←(x / |x|)+(y / |y|) with t=∑ a∈R (a / |a|), with t / |t| as the return value of compress(). Among them, ← means adding the vector on the right to the set on the left. It can be seen that PatternHA can also be obtained according to the above principle, that is:

[0117] When obtaining the total voice feature reference data of healthy users, the total voice feature data of multiple healthy users can be obtained by calculating ModelA(Hpj, Hnj, Hzj), and the total voice feature data of multiple healthy users can be used as the second vector set. Then, in the target coordinate system, a second standard vector located in the middle position of all vectors in the second vector set is determined, and the second standard vector is used as PatternHA. At this time, PatternHA can be calculated according to formula (22).

[0118] PatternHA=compress({x|x=ModelA(Hpj,Hnj,Hzj)}) (22)

[0119] According to this principle, other reference data can be obtained according to formula (23)-formula (28).

[0120] PatternNH=compress({x|x=ModelB(Hnj)}) (23)

[0121] PatternPH=compress({x|x=ModelC(Hpj)}) (24)

[0122] PatternZH=compress({x|x=ModelD(Hzj)}) (25)

[0123] PatternND=compress({x|x=ModelB(Dni)}) (26)

[0124] PatternPD=compress({x|x=ModelC(Dpi)}) (27)

[0125] PatternZD=compress({x|x=ModelD(Dzi)}) (28)

[0126] In some embodiments, Figure 6 As shown, the depression emotional state data acquisition device 100 also includes a third feature reference data acquisition module 601, which is used to acquire the total voice feature reference data of depression users, the total voice feature reference data of healthy users, the sub-voice feature reference data of depression users, and the sub-voice feature reference data of healthy users. The third feature reference data acquisition module 601 includes:

[0127] A total speech feature acquisition unit 6011 for a depressed user, configured to input speech data of a depressed user into a total speech feature recognition model, and output total speech feature data of the depressed user;

[0128] The depressed user speech feature acquisition unit 6012 is used to input speech data of depressed users corresponding to different emotional stimulation types into corresponding speech feature recognition models, and output speech feature data of depressed users;

[0129] A healthy user total voice feature acquisition unit 6013 is used to input the voice data of the healthy user into the total voice feature recognition model and output the total voice feature data of the healthy user;

[0130] A healthy user speech feature acquisition unit 6014 is used to input speech data of healthy users corresponding to different emotional stimulation types into a corresponding speech feature recognition model, and output speech feature data of healthy users;

[0131] A fully connected network model training unit 6015 is used to train a fully connected network model using the total speech feature data of the depression user, the total speech feature data of the healthy user, the sub-speech feature data of the depression user, and the sub-speech feature data of the healthy user as training input values ​​to obtain a trained fully connected network model;

[0132] The model parameter extraction unit 6016 is used to obtain the model parameters of the trained fully connected network model, and determine the total speech feature reference data of depressed users, the total speech feature reference data of healthy users, the sub-speech feature reference data of depressed users and the sub-speech feature reference data of healthy users based on the model parameters.

[0133] At this time, eight types of data, ModelA(Dpi,Dni,Dzi), ModelA(Hpj,Hnj,Hzj), ModelB(Hnj), ModelC(Hpj), ModelD(Hzj), ModelB(Dni), ModelC(Dpi), and ModelD(Dzi), can be used as training input values ​​and input into the fully connected network model to train the fully connected network model. Among the above eight types of data, ModelA(Dpi,Dni,Dzi) represents the total voice feature data of depression users, ModelA(Hpj,Hnj,Hzj) represents the total voice feature data of healthy users, ModelB(Hnj), ModelC(Hpj), and ModelD(Hzj) respectively represent the sub-voice feature data of different types of healthy users, and ModelB(Dni), ModelC(Dpi), and ModelD(Dzi) respectively represent the sub-voice feature data of different types of depression users.

[0134] After completing the training of the fully connected network model, the model parameters are obtained; in the model parameters, eight reference data such as PatternDA, PatternHA, PatternND, PatternPD, PatternZD, PatternNH, PatternPH and PatternZH are extracted. Specifically, the training input value can be standardized to obtain a standardized vector. Assuming that the standardized vector has V dimensions, the model parameter M of the fully connected network model is a tensor with V rows and 8 columns. Therefore, the function softmax(Q*M) can be used for classification. Q is the type corresponding to the standardized vector, and these types correspond to the aforementioned eight reference data. The classification goal is to determine which reference data the training input value corresponds to. The cross entropy function is used as the loss function to perform reverse gradient propagation calculation and update the model parameter M. Specifically, M / |M| is used to replace the model parameter M. After multiple rounds of training, the latest model parameter M is extracted, and each column of data corresponds to one of the eight reference data.

[0135] In some embodiments, the total voice feature recognition module inputs the voice data into a preset total voice feature recognition model to obtain the initial total voice feature data, adjusts the size value of the initial total voice feature data to the preset size value, obtains the total voice feature data of the target user and outputs the total voice feature data of the target user. Specifically, the size value of the initial total voice feature data can be adjusted by setting an adaptive pooling layer in the total voice feature recognition model. Similarly, the subdivided voice feature recognition module inputs the voice data into the corresponding sub-voice feature recognition model according to the emotional stimulation type of the specified sentence, obtains the original sub-voice feature data of the target user corresponding to each emotional stimulation type, adjusts the size value of each original sub-voice feature data to the preset size value, obtains the sub-voice feature data of the target user corresponding to each emotional stimulation type, and outputs the sub-voice feature data of the target user corresponding to each emotional stimulation type. In this way, it can be ensured that the size values ​​of the output total voice feature data of the target user and the sub-voice feature data of the target user remain consistent.

[0136] Each module in the above-mentioned depressive mood state data acquisition device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0137] In some embodiments, a system for acquiring data on a depressive mood state is provided, comprising: Figure 7 The server 701 and the terminal 702 are shown. The server 701 and the terminal 702 can communicate through the network, and the terminal 702 can collect the voice data of the target user and then send it to the server 701. The terminal 702 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the server 701 can be implemented as an independent server or a server cluster composed of multiple servers.

[0138] In some embodiments, terminal 702 is used to collect voice data formed by the target user reading aloud specified sentences, and send the voice data to the server, wherein the number of specified sentences is multiple, and at least two specified sentences have different emotional stimulation types; server 701 is used to obtain the voice data sent by the terminal, input the voice data into a preset total voice feature recognition model, output the total voice feature data of the target user, and determine a first similarity value based on the total voice feature data of the target user, the preset total voice feature reference data of the depressed user, and the preset total voice feature reference data of the healthy user, the first similarity value is the similarity value between the total voice feature data of the target user and the total voice feature reference data of the depressed user, and the depressive emotional state data includes the first similarity value or is determined based on the first similarity value.

[0139] In some embodiments, the step of collecting voice data formed by the target user reading a specified sentence executed by the terminal 702 includes: prompting the content of the current specified sentence; after detecting the confirmation instruction of the target user, recording the voice of the target user to generate current audio information; judging whether the current audio information meets the collection conditions; when it does not meet the collection conditions, prompting to re-read the current specified sentence, and then returning to the steps of executing the detection confirmation instruction, generating the current audio information, and judging whether it meets the collection conditions; when it meets the collection conditions, using the latest generated current audio information as the voice data.

[0140] Specifically, the method of prompting the content of the current designated sentence and prompting to re-read the current designated sentence includes but is not limited to the terminal 702 prompting through a display interface or prompting through a speaker.

[0141] In some embodiments, determining whether the current audio information meets the collection conditions includes: converting the content corresponding to the preset recording time in the current audio information into text information, and determining whether it meets the collection conditions based on whether the conversion is successful; if the conversion is successful, it is deemed to meet the collection conditions; if the conversion fails, it is deemed to not meet the collection conditions.

[0142] In some embodiments, determining whether the current audio information meets the collection conditions includes: converting the current audio information into text information, and determining whether it meets the collection conditions based on a similarity value between the text information and a preset reference text information of the currently specified sentence; if the similarity value is less than a preset text similarity value, it is deemed that the collection conditions are not met; if the similarity value is not less than a preset text similarity value, it is deemed that the collection conditions are met.

[0143] In some embodiments, the step of collecting voice data formed by the target user reading a specified sentence executed by the terminal 702 also includes: before prompting the content of the current specified sentence, prompting the content of the test sentence, the test sentence is a sentence of neutral emotional stimulation type; recording the test voice of the target user to generate test audio information; obtaining the sound feature value of the test voice according to the test audio information; when the sound feature value is greater than the preset recording threshold, prompting the target user to adjust the pronunciation. The sound feature value includes sound amplitude or speech rate. That is, before formally collecting voice data, a preliminary test phase can be set up first, by prompting the target user with the content of the test sentence, asking the target user to read the test sentence, and judging whether the sound amplitude or speech rate of the target user when reading meets the conditions. The purpose is to make the target user adapt to the required reading rhythm, so that the parameters such as reading time and reading volume corresponding to the subsequently collected voice data remain within a certain range.

[0144] In some embodiments, terminal 702 collects voice data generated by the target user reading a specified sentence and sends it to server 701. Server 701 obtains the depressive mood state data by executing the steps of the depressive mood state data acquisition method, and then sends the depressive mood state data to terminal 702 for the target user to view, or it can also be sent to other terminals for people in need to view.

[0145] For more specific limitations on the server 701 of the depressive mood state data acquisition system, please refer to the above limitations on the depressive mood state data acquisition device, which will not be repeated here.

[0146] The present application also provides a method for obtaining depressive emotional state data, which is illustrated by applying the method to server 701, and includes the following steps: obtaining voice data formed by a target user reading a specified sentence, the number of the specified sentences is multiple, and at least two of the specified sentences have different emotional stimulation types; inputting the voice data into a preset total voice feature recognition model, and outputting the total voice feature data of the target user; determining a first similarity value based on the total voice feature data of the target user, the preset total voice feature reference data of a depressed user, and the preset total voice feature reference data of a healthy user, the first similarity value being the similarity value between the total voice feature data of the target user and the total voice feature reference data of the depressed user, and the depressive emotional state data includes the first similarity value or is determined based on the first similarity value.

[0147] In some embodiments, the step of training the total speech feature recognition model includes: obtaining speech data of two users with depression and speech data of two healthy users; inputting the speech data of the two users with depression and the speech data of two healthy users into the original total speech feature recognition model respectively, and outputting the corresponding total speech feature prediction results; substituting each total speech feature prediction result into a preset first target loss function to obtain a first loss function value; when the first loss function value does not meet the preset training stop condition, adjusting the model parameters of the original total speech feature recognition model, and returning to execute the step of obtaining the speech data of two users with depression and the speech data of two healthy users; when the first loss function value meets the preset training stop condition, obtaining the total speech feature recognition model.

[0148] In some embodiments, a method for acquiring data on depressive emotional states includes: inputting voice data of multiple depressed users into a total voice feature recognition model, respectively, and outputting the total voice feature data of multiple depressed users; obtaining total voice feature reference data of depressed users by calculating the average value of the total voice feature data of multiple depressed users; inputting voice data of multiple healthy users into a total voice feature recognition model, respectively, and outputting the total voice feature data of multiple healthy users; obtaining total voice feature reference data of healthy users by calculating the average value of the total voice feature data of multiple healthy users.

[0149] In some embodiments, a method for acquiring data on depressive emotional states includes: inputting voice data of multiple users with depression into a total voice feature recognition model, outputting the total voice feature data of multiple users with depression, and using the total voice feature data of multiple users with depression as a first vector set; determining a first standard vector located in the middle of all vectors in the first vector set in a target coordinate system, and using the first standard vector as total voice feature reference data for users with depression; inputting voice data of multiple healthy users into a total voice feature recognition model, outputting the total voice feature data of multiple healthy users, and using the total voice feature data of multiple healthy users as a second vector set; determining a second standard vector located in the middle of all vectors in the second vector set in the target coordinate system, and using the second standard vector as total voice feature reference data for healthy users.

[0150] In some embodiments, the first similarity value is determined according to formula (4).

[0151] In some embodiments, a method for acquiring depressive emotional state data includes: inputting voice data into a corresponding speech feature recognition model according to the emotional stimulation type of a specified sentence, and outputting the speech feature data of the target user corresponding to each emotional stimulation type; determining a second similarity value based on the speech feature data of the target user corresponding to the same emotional stimulation type and the speech feature reference data of a depressed user, the second similarity value being the similarity value between the speech feature data of the target user and the speech feature reference data of the depressed user; determining a third similarity value based on the speech feature data of the target user corresponding to the same emotional stimulation type and the speech feature reference data of a healthy user, the third similarity value being the similarity value between the speech feature data of the target user and the speech feature reference data of a healthy user; the depressive emotional state data also includes the second similarity value and the third similarity value, or is also determined based on the second similarity value and the third similarity value.

[0152] In some embodiments, the training steps of the sub-speech feature recognition model include: obtaining similar speech data of two users with depression and similar speech data of two healthy users; inputting the similar speech data of the two users with depression and the similar speech data of two healthy users into the original sub-speech feature recognition model respectively, and outputting the corresponding prediction results of each sub-speech feature; substituting each sub-speech feature prediction result into a preset second target loss function to obtain a second loss function value; when the first loss function value or the second loss function value does not meet the preset training stop condition, adjusting the model parameters of the original total speech feature recognition model or the model parameters of the original sub-speech feature recognition model, and returning to execute the step of obtaining similar speech data of two users with depression and similar speech data of two healthy users; when the first loss function value and the second loss function value both meet the preset training stop condition, obtaining the total speech feature recognition model and the sub-speech feature recognition model.

[0153] In some embodiments, a method for acquiring data on a depressive emotional state includes: inputting voice data of a depressed user into a total voice feature recognition model, and outputting the total voice feature data of the depressed user; inputting voice data of a healthy user into a total voice feature recognition model, and outputting the total voice feature data of the healthy user; inputting voice data of a depressed user corresponding to different types of emotional stimulation into a corresponding sub-voice feature recognition model, and outputting the sub-voice feature data of the depressed user; inputting voice data of a healthy user corresponding to different types of emotional stimulation into a corresponding sub-voice feature recognition model, and outputting the sub-voice feature data of the healthy user; using the total voice feature data of the depressed user, the total voice feature data of the healthy user, the sub-voice feature data of the depressed user, and the sub-voice feature data of the healthy user as training input values, training a fully connected network model, and obtaining a fully connected network model that has been trained; obtaining model parameters of the fully connected network model that has been trained, and determining the total voice feature reference data of the depressed user, the total voice feature reference data of the healthy user, the sub-voice feature reference data of the depressed user, and the sub-voice feature reference data of the healthy user based on the model parameters.

[0154] For more definitions of the technical features in the method for acquiring depressive emotional state data, please refer to the above description of the corresponding modules or units in the device for acquiring depressive emotional state data, which will not be repeated here.

[0155] In some embodiments, a computer-readable storage medium is provided on which a computer program is stored. In different embodiments, when the computer program is executed by a processor, it respectively implements the steps performed by each functional module or functional unit of the depressive mood state data acquisition device described in the previous embodiments.

[0156] Those of ordinary skill in the art can understand that all or part of the process of implementing the method for acquiring data of a depressive mood state can be completed by instructing related hardware through a computer program. The aforementioned computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the process of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0157] In some embodiments, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. In different embodiments, when the computer program is executed by the processor, it respectively and correspondingly implements the steps described in the above embodiments and executed by each functional module or functional unit of the depressive mood state data acquisition device.

[0158] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0159] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A device for acquiring data on a depressive mood state, characterized in that: The device comprises: A total voice data acquisition module is used to acquire voice data formed by a target user reading a specified sentence, wherein the number of the specified sentences is multiple, and at least two of the specified sentences have different emotional stimulation types; the emotional stimulation types of the multiple specified sentences include a sad emotional stimulation type, a neutral emotional stimulation type, and a positive emotional stimulation type; A total voice feature recognition module, used for inputting the voice data into a preset total voice feature recognition model and outputting the total voice feature data of the target user; A first similarity determination module is used to determine a first similarity value based on the total voice feature data of the target user, the total voice feature reference data of a preset depression user, and the total voice feature reference data of a preset healthy user, wherein the first similarity value is: the ratio of the cosine similarity value between the total voice feature data of the target user and the total voice feature reference data of the depression user, the cosine similarity value between the total voice feature data of the target user and the total voice feature reference data of the depression user, and the sum of the cosine similarity values ​​between the total voice feature data of the target user and the total voice feature reference data of the depression user, and the sum of the cosine similarity values ​​between the total voice feature data of the target user and the total voice feature reference data of the healthy user; the depressive emotional state data includes the first similarity value or is determined based on the first similarity value.

2. The device according to claim 1, characterized in that The device also includes a model training module, and the model training module includes: A training data acquisition unit, used to acquire speech data of two users with depression and speech data of two healthy users; A feature prediction unit, used to input the speech data of the two depression users and the speech data of the two healthy users into the original total speech feature recognition model respectively, and output corresponding total speech feature prediction results; A calculation unit, used for substituting each of the total speech feature prediction results into a preset first target loss function to obtain a first loss function value; a parameter adjustment unit, configured to adjust the model parameters of the original total speech feature recognition model when the first loss function value does not meet a preset training stop condition, and instruct the training data acquisition unit to reacquire speech data of two depression users and speech data of two healthy users; A training stop unit is used to obtain the overall speech feature recognition model when the first loss function value meets the preset training stop condition.

3. The device according to claim 1, characterized in that The device further includes a first feature reference data acquisition module, wherein the first feature reference data acquisition module includes: A first total voice processing unit, used to input voice data of multiple depression users into the total voice feature recognition model respectively, and output the total voice feature data of multiple depression users; A first total mean value calculation unit, configured to obtain total voice feature reference data of the depression user by calculating an average value of total voice feature data of the plurality of depression users; A second total voice processing unit, used for inputting voice data of multiple healthy users into the total voice feature recognition model respectively, and outputting total voice feature data of multiple healthy users; The second total mean value calculation unit is used to obtain the total voice feature reference data of the healthy users by calculating the average value of the total voice feature data of the multiple healthy users.

4. The device according to claim 1, characterized in that The device further includes a second feature reference data acquisition module, wherein the second feature reference data acquisition module includes: A first vector set determination unit is used to input the speech data of multiple depression users into the total speech feature recognition model respectively, output the total speech feature data of multiple depression users, and use the total speech feature data of the multiple depression users as the first vector set; A first intermediate vector determination unit, configured to determine, in a target coordinate system, a first standard vector located at an intermediate position of all vectors in the first vector set, and use the first standard vector as total speech feature reference data of the depression user; A second vector set determination unit, configured to input the speech data of a plurality of healthy users into the total speech feature recognition model respectively, output the total speech feature data of the plurality of healthy users, and use the total speech feature data of the plurality of healthy users as the second vector set; The second intermediate vector determination unit is used to determine a second standard vector located at the middle position of all vectors in the second vector set in the target coordinate system, and use the second standard vector as the total speech feature reference data of the healthy user.

5. The device according to claim 2, characterized in that The device also includes: A subdivided speech feature recognition module, used for inputting the speech data into a corresponding subdivided speech feature recognition model according to the emotional stimulation type of the designated sentence, and outputting the subdivided speech feature data of the target user corresponding to each emotional stimulation type; A second similarity determination module is used to determine a second similarity value according to the sub-speech feature data of the target user and the sub-speech feature reference data of the depression user corresponding to the same emotional stimulation type, wherein the second similarity value is a similarity value between the sub-speech feature data of the target user and the sub-speech feature reference data of the depression user; A third similarity determination module is used to determine a third similarity value according to the sub-speech feature data of the target user and the sub-speech feature reference data of the healthy user corresponding to the same emotional stimulation type, wherein the third similarity value is a similarity value between the sub-speech feature data of the target user and the sub-speech feature reference data of the healthy user; The depressive emotional state data also includes the second similarity value and the third similarity value, or is also determined based on the second similarity value and the third similarity value.

6. The device according to claim 5, characterized in that include: The feature prediction unit is also used to input the similar voice data of the two depression users and the similar voice data of the two healthy users into the original sub-speech feature recognition model respectively, and output the corresponding prediction results of each sub-speech feature; The calculation unit is also used to substitute each of the speech feature prediction results into a preset second target loss function to obtain a second loss function value; When the first loss function value or the second loss function value does not satisfy the preset training stop condition, the parameter adjustment unit adjusts the model parameters of the original total speech feature recognition model or the model parameters of the original sub-speech feature recognition model, and instructs the training data acquisition unit to re-acquire similar speech data of two users with depression and similar speech data of two healthy users; The training stop unit obtains the overall speech feature recognition model and the sub-speech feature recognition model when the first loss function value and the second loss function value both meet the preset training stop condition.

7. The device according to claim 5, characterized in that The device further includes a third feature reference data acquisition module, and the third feature reference data acquisition module includes: A total voice feature acquisition unit for a depressed user, used to input voice data of a depressed user into the total voice feature recognition model and output the total voice feature data of the depressed user; A sub-speech feature acquisition unit for users with depression, used to input speech data of users with depression corresponding to different emotional stimulation types into a corresponding sub-speech feature recognition model, and output sub-speech feature data of users with depression; A healthy user total voice feature acquisition unit, used to input the healthy user's voice data into the total voice feature recognition model and output the healthy user's total voice feature data; A healthy user speech feature acquisition unit, used to input speech data of healthy users corresponding to different emotional stimulation types into a corresponding speech feature recognition model, and output the speech feature data of healthy users; A fully connected network model training unit, used to train a fully connected network model by using the total voice feature data of the depression user, the total voice feature data of the healthy user, the sub-voice feature data of the depression user, and the sub-voice feature data of the healthy user as training input values, to obtain a trained fully connected network model; A model parameter extraction unit is used to obtain the model parameters of the trained fully connected network model, and determine the total speech feature reference data of the depressed user, the total speech feature reference data of the healthy user, the sub-speech feature reference data of the depressed user and the sub-speech feature reference data of the healthy user based on the model parameters.

8. A system for acquiring data on a depressive mood state, characterized in that: Including servers and terminals; The terminal is used to collect voice data formed by a target user reading a specified sentence, and send the voice data to the server, the number of the specified sentences is multiple, at least two of the specified sentences have different emotional stimulation types; the emotional stimulation types of the multiple specified sentences include a sad emotional stimulation type, a neutral emotional stimulation type, and a positive emotional stimulation type; The server is used to obtain the voice data sent by the terminal, input the voice data into a preset total voice feature recognition model, output the total voice feature data of the target user, and determine a first similarity value according to the total voice feature data of the target user, preset total voice feature reference data of a user with depression, and preset total voice feature reference data of a healthy user, wherein the first similarity value is: a ratio of a cosine similarity value between the total voice feature data of the target user and the total voice feature reference data of the user with depression, a cosine similarity value between the total voice feature data of the target user and the total voice feature reference data of the user with depression, and a sum of cosine similarity values ​​between the total voice feature data of the target user and the total voice feature reference data of the user with depression, and a sum of cosine similarity values ​​between the total voice feature data of the target user and the total voice feature reference data of the healthy user; The depressive emotional state data includes the first similarity value or is determined according to the first similarity value.

9. A method for acquiring depressive mood state data, characterized in that: include: Acquire speech data formed by a target user reading aloud designated sentences, wherein the number of the designated sentences is multiple, and at least two of the designated sentences have different emotional stimulation types; the emotional stimulation types of the multiple designated sentences include a sad emotional stimulation type, a neutral emotional stimulation type, and a positive emotional stimulation type; Input the voice data into a preset total voice feature recognition model, and output the total voice feature data of the target user; Determine a first similarity value according to the total voice feature data of the target user, the total voice feature reference data of a preset depression user, and the total voice feature reference data of a preset healthy user, wherein the first similarity value is: a ratio of a cosine similarity value between the total voice feature data of the target user and the total voice feature reference data of the depression user, to a sum of a cosine similarity value between the total voice feature data of the target user and the total voice feature reference data of the depression user, and to a sum of a cosine similarity value between the total voice feature data of the target user and the total voice feature reference data of the depression user, and to a sum of a cosine similarity value between the total voice feature data of the target user and the total voice feature reference data of the healthy user; The depressive emotional state data includes the first similarity value or is determined according to the first similarity value.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 9 are implemented.

Images