A state detection method, device, apparatus and storage medium

Abstract

This application discloses a state detection method, apparatus, device, and storage medium. The method includes: acquiring video and audio about a target object; extracting at least a first facial feature of the target object from the video, extracting a first speech feature of the target object from the audio, and obtaining semantic completeness features of the text information corresponding to the audio based on the audio; obtaining a state detection result about the target object based at least on the first facial feature, the first speech feature, and the semantic completeness feature, wherein the state detection result is used to determine whether the target object exists in a preset state. Through the above methods, this application can improve the accuracy of state detection of the target object.

Description

Technical Field

[0001] This application relates to the field of intelligent detection technology, and in particular to a state detection method, apparatus, device, and storage medium. Background Technology

[0002] In daily life, target objects (such as people) always exhibit various states. Generally, states are used to describe the situation of target objects. However, due to various reasons, target objects may not be aware of their state. For example, the state may correspond to the state exhibited by Parkinson's patients. Many target objects have already exhibited the state corresponding to Parkinson's patients, but due to insufficient knowledge or the degree of state manifestation, they do not know or are unsure of their own state, thus delaying the opportunity to formulate effective coping measures.

[0003] Therefore, it is of great significance to know how to detect the state and obtain the corresponding state detection results. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a state detection method, apparatus, device, and storage medium that can improve the accuracy of state detection of a target object.

[0005] To address the aforementioned technical problems, this application provides a state detection method, which includes: acquiring video and audio about a target object; extracting at least a first facial feature of the target object from the video, extracting a first speech feature of the target object from the audio, and obtaining semantic integrity features of the text information corresponding to the audio based on the audio; obtaining a state detection result about the target object based at least on the first facial feature, the first speech feature, and the semantic integrity feature, wherein the state detection result is used to determine whether the target object has a preset state.

[0006] The video contains multiple frames of facial images of the target object. Extracting the first facial feature of the target object from the video includes: extracting the second facial feature from at least one frame of facial images to obtain the first facial feature; and / or, the audio contains multiple frames of speech images of the target object. Extracting the speech feature and semantic integrity feature of the target object from the audio includes: extracting the second speech feature from at least one frame of speech images to obtain the first speech feature; and / or, extracting the second speech feature corresponding to each speech frame, fusing the second speech features to obtain the speech fusion feature of the audio; performing speech recognition on the speech fusion feature to obtain the text information corresponding to the audio; and processing the text information to obtain the semantic integrity feature.

[0007] The process of extracting second facial features from at least one frame of facial image includes: extracting multiple facial key points from each frame of facial image; obtaining spatial relationship values ​​between a first line segment and each second line segment, wherein the first line segment is composed of a line connecting at least two facial key points, and each second line segment is composed of a line connecting a facial key point corresponding to one end of the first line segment and each facial key point; and sorting the spatial relationship values ​​corresponding to the facial image in a preset order to obtain the second facial features in the facial image.

[0008] Specifically, based at least on the first facial feature, the first speech feature, and the semantic integrity feature, a state detection result for the target object is obtained, including: fusing the first facial feature and the first speech feature to obtain a first fused feature; fusing the first fused feature and the semantic integrity feature to obtain a second fused feature; and performing state detection on the second fused feature to obtain a state detection result for the target object.

[0009] The video contains multiple frames of facial images of the target object, and the first facial feature includes the second facial features of each frame of facial images. The audio contains multiple frames of speech images of the target object, and the first speech feature includes the second speech features of each frame of speech images. The first fusion feature is obtained by fusing the first facial feature and the first speech feature, including: fusing each second facial feature to obtain a facial fusion feature, and fusing each second speech feature to obtain a speech fusion feature, wherein the facial fusion feature is the first facial feature and the speech fusion feature is the first speech feature; the third fusion feature is obtained by fusing the facial fusion feature and the speech fusion feature; the third fusion feature is processed using a processing model to obtain the first fusion feature; and / or, state detection is performed on the second fusion feature to obtain a state detection result for the target object, including: processing the second fusion feature using a classification model to obtain a state detection result for the target object.

[0010] Among them, there are multiple sets of videos and audios related to the target object, and the state detection results include the probability that the target object exists in a preset state. After obtaining the state detection results corresponding to each set of videos and audios, the state detection method also includes: obtaining the final state detection result of the target object based on the probability in the state detection results corresponding to each set of videos and audios.

[0011] The first speech feature is obtained using a speech recognition toolkit, the first facial feature is obtained using a feature extraction tool in an open-source face recognition library, and / or the preset state is the state caused by the neural degeneration of the target object.

[0012] To address the aforementioned technical problems, another technical solution adopted in this application is: providing a state detection device, which includes: an acquisition module for acquiring video and audio about a target object; a feature extraction module for extracting at least a first facial feature of the target object from the video and extracting a first speech feature and semantic integrity feature of the target object from the audio; and a state detection module for obtaining a state detection result about the target object based at least on the first facial feature, the first speech feature, and the semantic integrity feature, wherein the state detection result is used to determine whether the target object exists in a preset state.

[0013] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide an electronic device, including a memory and a processor coupled to each other, wherein the memory stores program instructions; and the processor is used to execute the program instructions stored in the memory to implement the above-mentioned method.

[0014] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium for storing program instructions that can be executed to implement the above-mentioned method.

[0015] The beneficial effects of this application are as follows: This application first obtains video and audio about the target object, then extracts the first facial features of the target object from the video, extracts the first speech features of the target object from the audio, and obtains the semantic completeness features of the corresponding text based on the audio. Then, based on at least the first facial features, the first speech features, and the semantic completeness features, it obtains the state detection results of the target object. The state detection results are used to determine whether the target object has a preset state, thus realizing the state detection of the target object. Moreover, the solution of this application can at least comprehensively integrate the first facial features, the first speech features, and the semantic completeness features exhibited by the target object to realize the state detection of the target object. Compared with realizing the state detection based only on facial features and speech features, it improves the accuracy of the state detection of the target object. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating an embodiment of the state detection method provided in this application;

[0017] Figure 2 yes Figure 1 The flowchart of step S12 shown is a schematic diagram of one embodiment;

[0018] Figure 3 yes Figure 1 A flowchart illustrating another embodiment of step S12 is shown.

[0019] Figure 4 This is a schematic diagram of obtaining a second facial feature through facial key points provided in this application;

[0020] Figure 5 yes Figure 1 The flowchart of step S13 shown is a schematic diagram of one embodiment.

[0021] Figure 6 yes Figure 5 The flowchart of step S51 shown is a schematic diagram of an embodiment.

[0022] Figure 7 This is a flowchart illustrating an embodiment of the state detection method provided in this application;

[0023] Figure 8 This is a schematic diagram of the framework of an embodiment of the state detection device provided in this application;

[0024] Figure 9 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application;

[0025] Figure 10 This is a schematic diagram of the structure of the computer-readable storage medium provided in this application. Detailed Implementation

[0026] To make the purpose, technical solution and effects of this application clearer and more explicit, the following describes this application in further detail with reference to the accompanying drawings and embodiments.

[0027] It should be noted that if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0028] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the state detection method provided in this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily replace it. Figure 1 The illustrated process sequence is limited. For example... Figure 1 As shown, this embodiment includes:

[0029] S11: Obtain video and audio data about the target object.

[0030] This embodiment is used to analyze and detect the acquired video and audio data about the target object to obtain the state result of the target object, so as to determine whether the target object has a preset state.

[0031] The target object described in this article is the object to be detected for a preset state. The preset state can be any state exhibited by the target object, including but not limited to its mental state or states manifested due to illness. In one embodiment, the preset state is a state caused by neurodegeneration in the target object, such as some states exhibited by Parkinson's disease patients. It should be noted that the presence of preset states exhibited by Parkinson's disease patients can be determined based on the target object's facial expressions, speech patterns, and semantic completeness. Specifically, the states exhibited by Parkinson's disease patients generally include limited facial expressions, slow facial reactions, low-pitched, hoarse, and monotone speech, as well as incomplete semantic expression that is difficult for others to understand. Therefore, this embodiment can detect whether the target object exhibits preset states exhibited by Parkinson's disease patients based on its facial features, speech features, and semantic completeness. Of course, this embodiment can also detect other preset states, such as mental states, based on the target object's facial features, speech features, and semantic completeness. Specific preset states can be determined according to the actual situation and are not specifically limited here.

[0032] In one embodiment, the audio and video of the target object can also be obtained from local storage or cloud storage.

[0033] In another embodiment, the audio and video of the target object can also be acquired using an audio and video acquisition device. Specifically, after startup, the audio and video acquisition device can randomly select pre-stored questions from a question database, guide the target object to answer the questions according to the requirements, and acquire the target object's voice and images during the answering process to obtain video and audio data about the target object. In some scenarios, the pre-stored questions include the questions to be answered and the facial expressions to be used when answering, such as a simple question like "Please answer your recent health condition with a smile," to facilitate accurate identification of the target object's state later. Of course, the pre-stored questions can also only contain questions to be answered; the content and difficulty of the specific pre-stored questions can be determined according to the actual usage scenario and the purpose of detection. Furthermore, in one embodiment, the audio and video acquisition device sets a corresponding answering time for each pre-stored question, so that when the target object's answering time reaches the preset duration, the device will automatically stop acquiring audio and images. The answering time is the time range between the start and end points of the answering process. Of course, in other embodiments, no answering time may be set; if the audio and video acquisition device does not receive the target object's voice within the predetermined time range, it will automatically stop acquiring audio and images.

[0034] S12: Extract the first facial features of the target object from the video, extract the first speech features of the target object from the audio, and obtain the semantic completeness features of the text information corresponding to the audio based on the audio.

[0035] The acquired video of the target object contains multiple frames of facial images of the target object. In one embodiment, these multiple frames of facial images are acquired by an audio / video acquisition device. To ensure the accuracy of subsequent state detection results, clear images can be selected as the multiple frames of facial images of the target object. After acquiring the multiple frames of facial images of the target object, at least one frame is selected from the multiple frames, and second facial features are extracted from each of these frames to obtain a first facial feature. Here, "at least one frame" can refer to, but is not limited to, one, two, or all frames from the multiple frames. In some implementation scenarios, one frame can be selected from the multiple frames, and the second facial features can be extracted from that frame, using it as the first facial feature. In other implementation scenarios, multiple facial images can be selected from the multiple frames, and the second facial features of each frame can be extracted. The second facial features are then concatenated to obtain the first facial feature. The number of frames of facial images selected can be determined according to the actual application scenario. For example, if the preset state to be detected requires observing the differences between multiple facial images (such as changes in expression), at least two facial images can be selected. Furthermore, in order to more accurately observe whether there are changes in expression of the target object, multiple facial images can be selected.

[0036] In one embodiment, for each frame of a facial image in at least one frame, multiple facial key points can be extracted from the corresponding facial image, and second facial features of each frame can be obtained using each key point. These second facial features are then concatenated to obtain a first facial feature. In other embodiments, a facial feature extraction model or related feature extraction algorithm can be directly used to extract each second facial feature from each frame of the facial image, and these second facial features can be concatenated to obtain a first facial feature.

[0037] The acquired audio of the target object contains multiple speech frames of the target object. In one embodiment, the audio is acquired through an audio-visual acquisition device. To improve the efficiency of state detection, speech frames from the start of the target object's answer to the end of the answer can be extracted from the audio, and these extracted speech frames during the answer time period are taken as multiple speech frames. After acquiring multiple speech frames of the target object's answering process, at least one speech frame is selected from the multiple speech frames, and a second speech feature is extracted from the at least one speech frame to obtain a first speech feature. The selected at least one frame can represent one frame, a portion of multiple frames, or all frames. If the at least one frame represents one frame, then the second speech feature is the first speech feature. If the at least one frame represents multiple frames, then the multiple second speech features extracted from the multiple speech frames are concatenated to obtain the first speech feature. The specific number of at least one frames can be determined according to the preset state used for detection. For example, if the preset state used for detection requires analysis of the target object's pitch, then at least two speech frames need to be selected to analyze the target object's pitch characteristics based on at least two second speech features.

[0038] In one embodiment, the first speech feature can be obtained using a speech recognition toolkit (e.g., HTK, a toolkit for building hidden Markov models). HTK can accurately extract filterbank features (e.g., filterbank40 features). Filterbank is a front-end processing algorithm that processes audio in a manner similar to the human ear, based on the nonlinear response of the human ear to the sound spectrum, thus improving the speech recognition effect. The steps for obtaining the filterbank features of the speech signal include pre-emphasis, framing, windowing, short-time Fourier transform, and filtering using a Mel filter. Here, 40 refers to the number of filters used, and the final feature vector is also 40-dimensional. Of course, the specific dimension of the extracted feature vector can be determined according to the actual effect, and is not specifically limited here. In other embodiments, the first speech feature can be obtained using other speech recognition algorithms or speech recognition models, and is not specifically limited here.

[0039] In one embodiment, after acquiring the audio of the target object, the semantic completeness features of the corresponding text information can be obtained based on the acquired audio.

[0040] Specifically, please refer to Figure 2 Please see Figure 2 , Figure 2 yes Figure 1 The flowchart shown is a schematic diagram of one embodiment of step S12. It should be noted that if substantially the same result is achieved, this embodiment does not necessarily follow the same pattern. Figure 2 The illustrated process sequence is limited. For example... Figure 2As shown, in this embodiment, the semantic completeness features of the text information corresponding to the audio are obtained based on the audio, specifically including:

[0041] S21: Extract the second speech features corresponding to each speech frame.

[0042] In this embodiment, after acquiring multiple audio frames of the target object, the second speech features corresponding to each audio frame are extracted. The extraction method for the second speech features can be referred to above and will not be elaborated further here. It should be noted that this embodiment is used to obtain the semantic completeness features of the target object based on its audio, in order to facilitate subsequent analysis of the target object's semantic expression state. Therefore, in one implementation, the second speech features corresponding to each audio frame containing the target object's speech can be extracted, facilitating the subsequent acquisition of features related to the semantic completeness of the target object through these second speech features.

[0043] S22: Fuse each second speech feature to obtain the speech fusion feature of the audio.

[0044] In one embodiment, the second speech features are fused by splicing to obtain the speech fusion features of the audio. The speech fusion features can be features in the audio that contain all the speech content of the target object.

[0045] S23: Perform speech recognition on the speech fusion features to obtain the text information corresponding to the audio.

[0046] In one embodiment, a trained speech recognition model can be used to identify speech fusion features to obtain the text information corresponding to the audio. The trained speech recognition model is an encoder-decoder framework model. In other embodiments, relevant speech recognition algorithms can also be used to obtain the text information corresponding to the audio; no specific limitations are made here.

[0047] S24: Process the text information to obtain the semantic completeness features of the text information.

[0048] In one embodiment, the text information can be processed using a pre-trained bidirectional long short-term memory (LSTM) recurrent neural network model or other neural network model to obtain the semantic completeness features of the text information. The obtained semantic completeness features are 1*1 feature matrices with feature values ​​between 0 and 1. It can be understood that the more complete the expression of the text, the higher its semantic completeness.

[0049] It should be noted that the semantic completeness feature of the obtained text information is the completeness of the semantic expression of the target object in response to the corresponding question. The completeness of semantic expression is used to represent the semantic expression state of the target object.

[0050] S13: Based at least on the first facial feature, the first speech feature, and the semantic integrity feature, obtain the state detection result of the target object. The state detection result is used to determine whether the target object has a preset state.

[0051] The preset state can be a comprehensive state representing the mental state of the target object, or a comprehensive state caused by neurodegeneration of the target object. The specific state can be determined according to actual needs and is not specifically limited here. The preset state can be, but is not limited to, a comprehensive state including facial state, voice state, and semantic expression state, or a comprehensive state including limb state. The specific setting can be based on actual needs. In this embodiment, if the detected state result matches the preset state, it is determined that the target object possesses the preset state. Conversely, if it does not match, it is determined that the target object does not possess the preset state.

[0052] In this embodiment, a state detection result for a target object can be obtained based at least on a first facial feature, a first speech feature, and a semantic completeness feature. The first facial feature corresponds to the facial state feature of the target object, the first speech feature corresponds to the speech state feature of the target object, and the semantic completeness feature corresponds to the semantic expression state feature of the target object. The state detection result is used to determine whether the target object exists in a preset state. In other words, the state detection result is a result that integrates at least the facial state feature, speech state feature, and semantic expression state feature of the target object, and this state detection result can be used to determine whether the target object exists in a preset state. It should be noted that in some embodiments, the state detection result can be obtained by integrating the first facial feature, the first speech feature, and the semantic completeness feature. Of course, in other embodiments, to further improve the accuracy of the state detection result, other features that affect the accuracy of the state detection result can also be considered.

[0053] For example, if the preset state is the condition caused by neurodegeneration in Parkinson's patients, which is generally characterized by limited facial expression and slow facial responses, a low and monotone voice, and incomplete semantic expression, then at least the facial, vocal, and semantic expression completeness of the target subject can be considered together to detect whether the target subject exhibits the preset state of Parkinson's patients, thus determining whether the target subject has Parkinson's disease. Of course, to further improve the accuracy of the detection results, the presence of motor states such as balance disorders, characteristic of Parkinson's patients, can also be considered to obtain the state detection result.

[0054] In this embodiment, video and audio of the target object are first acquired. Then, the first facial feature and first speech feature of the target object are extracted from the video and audio, respectively. Semantic completeness features of the corresponding text are obtained based on the audio. Finally, based at least on the first facial feature, first speech feature, and semantic completeness feature, a state detection result for the target object is obtained. The state detection result is used to determine whether the target object exists in a preset state. Therefore, state detection of the target object is achieved. Moreover, this application's solution can at least comprehensively utilize the first facial feature, first speech feature, and semantic completeness feature exhibited by the target object to achieve state detection, which improves the accuracy of state detection compared to state detection based solely on facial and speech features.

[0055] Please see Figure 3 , Figure 3 yes Figure 1 The flowchart of another embodiment of step S12 is shown. It should be noted that if substantially the same result is achieved, this embodiment does not necessarily follow the same pattern. Figure 3 The illustrated process sequence is limited. For example... Figure 3 As shown, in this embodiment, multiple facial key points are extracted from each frame of facial images, and the second facial features of each frame of facial images are obtained using each key point. Specifically, this includes:

[0056] S31: For each frame of facial image, extract multiple facial key points from the facial image.

[0057] This embodiment is used to extract the second facial features of each frame of facial images from at least one frame of facial images.

[0058] For each frame of facial image, multiple facial key points can be extracted using a facial recognition database or relevant machine learning algorithms. These multiple facial key points are key points of at least one of the following parts of the target object's facial contour, eyes, nose, and mouth. The specific number of facial key points can be determined based on the tools used and the actual application scenario. Figure 3 As shown, Figure 3 To extract 68 facial key points using the dlib open-source face recognition library, which contains a face detection module and a feature extraction module, this method can extract 68 key points from a facial image and their corresponding coordinates.

[0059] S32: Obtain the spatial relationship values ​​between the first line segment and each of the second line segments respectively. The first line segment is composed of the line connecting at least two facial key points, and each of the second line segments is composed of the facial key point corresponding to one end of the first line segment and the line connecting each facial key point.

[0060] In this embodiment, for each frame of facial image, after acquiring multiple key points, at least two facial key points are randomly selected from the multiple key points (wherein, the selected at least two facial key points are on the same straight line), and the line connecting the at least two facial key points is taken as the first line segment. Then, the key point corresponding to one end of the first line segment is taken as one endpoint of the second line segment, and any one of the multiple key points is taken as the other endpoint of the second line segment, thus forming multiple second line segments, and the first line segment and the second line segment form an angle. Figure 4 As shown, the line connecting key points 28 to 31 is the first line segment, key point 31 is one endpoint of the second line segment, and any one of the 68 key points is the other endpoint of the second line segment. The first and second line segments form an angle. It can be understood that the number of second line segments is the same as the number of extracted facial key points.

[0061] After determining the first line segment and each second line segment, the spatial relationship values ​​between the first line segment and each second line segment are obtained respectively. The spatial relationship values ​​between the first line segment and each second line segment can be the trigonometric function values ​​of the angles between the first line segment and each second line segment, such as the cosine function value, sine function value, tangent function value, etc. of each angle. In a specific embodiment, the trigonometric function value of the angle between the first line segment and each second line segment is the cosine function value of each angle, which can be calculated according to formula (1).

[0062]

[0063] Where cos∠n is the cosine of the angle formed by key points 31, 28, and n, (x n y n Let (x) be the coordinates of any keypoint n among multiple keypoints, and (x) be the coordinates of the keypoint n. 31 y 31 (x) represents the coordinates of key point number 31. 28 y 28 ) represents the coordinates of key point 28. In particular, key point 31 is the vertex formed by the first line segment and each of the second line segments. When key point n is key point 31, the corresponding cosine value is set to 1.

[0064] S33: Sort the spatial relationship values ​​corresponding to the facial images in a preset order to obtain the second facial features in the facial images.

[0065] In this embodiment, for each frame of facial image, the spatial relationship values ​​corresponding to the acquired facial image are sorted according to a preset order to obtain the second facial feature corresponding to each frame of facial image. The specific preset order can be determined according to the actual scene.

[0066] Changes in multiple facial key points can represent changes in the target object's face. However, the expression and its changes can be represented by the relative relationships between key points in different facial images. It's important to note that for each frame of facial images, using the spatial relationships (e.g., the trigonometric function values ​​of the angles between the first and second line segments formed by the facial key points) can reduce the impact of positional changes of the target object relative to the audio / video capture device. This allows for the determination of the target object's current expression based on the trigonometric function values ​​in each frame of facial images, and accurate judgment of facial expression changes based on the changes in spatial relationships. For example, when capturing a facial image of the target object, if the target object's facial expression hasn't changed, but the face has moved, the coordinates of the key points will change significantly, but the angles between the first and second line segments will remain almost unchanged. This demonstrates that using the spatial relationships (e.g., the trigonometric function values ​​of the angles between the first and second line segments formed by the facial key points) can mitigate misidentification of expression changes caused by positional changes relative to the audio / video capture device, thus facilitating accurate description of facial state changes.

[0067] Please see Figure 5 , Figure 5 yes Figure 1 The diagram shows a flowchart of one embodiment of step S13. It should be noted that if substantially the same result is achieved, this embodiment does not necessarily follow that approach. Figure 5 The illustrated process sequence is limited. For example... Figure 5 As shown, in this embodiment, the state detection result of the target object is obtained based at least on the first facial feature, the first voice feature, and the semantic completeness feature, specifically including:

[0068] S51: The first facial feature and the first speech feature are fused to obtain the first fused feature.

[0069] In one embodiment, a first facial feature of the target object with respect to a facial image and a first speech feature of the target object with respect to speech can be concatenated to obtain a first fused feature.

[0070] In another embodiment, a first fused feature can be obtained by fusing a first facial feature and a first speech feature using a multi-layer neural network.

[0071] Specifically, please refer to Figure 6 , Figure 6 yes Figure 5 The flowchart shown is a schematic diagram of one embodiment of step S51. It should be noted that if substantially the same result is achieved, this embodiment does not necessarily follow the same pattern. Figure 6 The illustrated process sequence is limited. For example... Figure 6 As shown, in this embodiment, step S51 further includes:

[0072] S61: Fuse each second facial feature to obtain a facial fusion feature, and fuse each second speech feature to obtain a speech fusion feature, wherein the facial fusion feature is the first facial feature and the speech fusion feature is the first speech feature.

[0073] In one embodiment, after obtaining the second facial features of each facial image and the second speech features of each speech frame, the second facial features are fused by concatenation to obtain facial fusion features, and the second speech features are fused by concatenation to obtain speech fusion features. The facial fusion features are the first facial features, and the speech fusion features are the first speech features.

[0074] S62: Fuse facial fusion features and speech fusion features to obtain a third fusion feature.

[0075] In one embodiment, facial fusion features related to facial images and speech fusion features related to speech are concatenated to obtain a third fusion feature, wherein the third fusion feature is a fusion feature that combines facial features and speech features.

[0076] S63: The third fusion feature is processed using the processing model to obtain the first fusion feature.

[0077] In one embodiment, the processing model is a multi-layer neural network model. The first and second layers of this model are fully connected layers, the third to tenth layers are bidirectional conformer structures with a multi-head attention mechanism, and the eleventh layer is a fully connected layer. A third fused feature, which integrates facial and speech features, is input into this processing model, allowing the extraction of a first fused feature that integrates facial and speech features. Because the processing model learns the mapping relationship between the features, the first fused feature obtained after model processing contains features that include the relationship between facial and speech, which is more conducive to the accuracy of subsequent model recognition results.

[0078] S52: The first fusion feature and the semantic completeness feature are combined to obtain the second fusion feature.

[0079] In one embodiment, the feature fusion method is concatenation. Specifically, the first fusion feature and the semantic completeness feature are concatenated to obtain the second fusion feature. The second fusion feature includes facial features, speech features, and semantic completeness features.

[0080] S53: Perform state detection on the second fusion feature to obtain the state detection result for the target object.

[0081] In one embodiment, a classification model is used to process the second fused feature to obtain a state detection result for the target object. Optionally, a state discrimination classification layer can be added to the above-mentioned multi-layer neural network model to perform state detection on the second fused feature and obtain a state detection result for the target object. This state detection result can be, for example, either the existence of a preset state or the absence of a preset state, or only the probability of the preset state existing. In one embodiment, the state discrimination classification layer performs state detection on the second fused feature to obtain two feature values, which are then activated by a softmax activation function to obtain a state detection result. The state detection result includes the probability of the preset state existing, or the probability of the preset state existing and the probability of the preset state not existing. The softmax activation function is:

[0082]

[0083] Where, p i Let z be the probability of the preset state existing. i This indicates that the preset state has a corresponding feature value, z. j This indicates that the preset state does not have a corresponding feature value.

[0084] In one embodiment, multiple sets of video and audio data about the target object are acquired. Specifically, after the audio and video acquisition device is started, multiple questions can be randomly selected from the question library. The entire video and audio data can be cropped according to the number of questions to obtain multiple sets of video and audio data, where each set corresponds to a question.

[0085] In one embodiment, the state detection result includes the probability that the target object exists in a preset state. After obtaining the state detection results corresponding to each group of videos and audio, a final state detection result for the target object is obtained based on the probabilities in the state detection results corresponding to each group of videos and audio. Specifically, the probability values ​​in the state detection results corresponding to multiple groups of videos and audio can be added together and averaged. The average value is then compared with a preset threshold to obtain the final state detection result for the target object, i.e., the result of whether the target object exists in the preset state.

[0086] It should be noted that, in one embodiment, before fusing the first facial features and the first speech features using the aforementioned multi-layer neural network to obtain the first fused feature, the multi-layer neural network needs to be trained first to obtain an optimized multi-layer neural network. This multi-layer neural network model has 12 layers: layers 1 and 2 are fully connected layers; layers 3 to 10 are bidirectional conformer structure layers with a multi-head attention mechanism; layer 11 is a fully connected layer; and layer 12 is a state discrimination and classification layer. The following describes the training process of this multi-layer neural network, using the preset state of a Parkinson's patient due to neurodegeneration as an example. Specifically, the training steps of the multi-layer neural network include:

[0087] First, training and testing sets are obtained. Specifically, audio and video recordings of multiple subjects are collected to form datasets. All subjects are clinically validated, and the ratio of Parkinson's patients to non-patients is 1:1. A number of subjects are randomly selected from both the Parkinson's patient and non-patient datasets to form the training set, with the remainder used as the testing set. Data from non-Parkinson's patients is labeled as 0, and data from Parkinson's patients is labeled as 1. To ensure training effectiveness, the acquired audio and video recordings must adhere to the following principles: the question-and-answer design should not be overly complex (e.g., a question like "Please answer your recent health condition with a smile"); each answer should not exceed a preset time, such as 10 seconds; the faces of the subjects should be clear, complete, and unobstructed; and the audio volume should be normal with no significant noise. These principles can be adjusted according to actual circumstances.

[0088] Secondly, the first sample facial features of the target object are extracted from the sample video, the first sample speech features of the target object are extracted from the sample audio, and the semantic completeness features of the sample text information corresponding to the sample audio are obtained based on the sample audio. Then, the first sample speech features and the first sample facial features are concatenated to obtain the first sample fusion features.

[0089] Then, the multi-layer neural network model (12 layers in total) is trained using the first sample fusion features and semantic completeness features to obtain the trained multi-layer neural network model. During training, the first 11 layers of the multi-layer neural network model are trained using the first sample fusion features to obtain the second sample fusion features (fusing the first sample facial features and the first sample speech features). Then, the second sample fusion features and semantic completeness features are concatenated to obtain the third sample fusion features. The 12th layer (state discrimination classification layer) is trained using the third sample fusion features. During training, the output of the 12th layer is activated and the loss is calculated by comparing it with the actual Parkinson's disease situation. The loss function used is the cross-entropy loss function.

[0090]

[0091] Where N is the number of samples, y i p represents the true label of sample i, with 0 for no disease and 1 for disease. i Let i be the probability that sample i has Parkinson's disease.

[0092] Furthermore, after training the multi-layer neural network model using the first sample fusion features and semantic completeness features, the multi-layer neural network model is then tested using a test set.

[0093] In one specific embodiment, taking a 5.8-second audio-visual recording with 580 video and audio frames as an example, please refer to [link to specific example]. Figure 7 , Figure 7 This is a flowchart illustrating an embodiment of the state detection method provided in this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily replace it. Figure 7 The illustrated process sequence is limited. For example... Figure 7 As shown, this embodiment includes:

[0094] S71: Extract the first facial feature from the video, extract the first audio feature from the audio, and obtain the semantic completeness feature of the text information corresponding to the audio based on the audio.

[0095] Specifically, the coordinates of 68 key points of each frame of facial image are obtained using the face recognition open-source library dlib, and the cosine values ​​of the first line segment and each second line segment in the 68 key points are calculated based on formula (1) to obtain each second facial feature (68*2 matrix). The second facial features are then spliced ​​together to obtain the first facial feature (facial fusion feature), where the first facial feature is a 580*68 feature matrix.

[0096] The second audio features corresponding to each speech frame were extracted using the open-source tool HTK. The second audio features are 40-dimensional feature vectors. The first audio features (speech fusion features) with a size of 580*40 were obtained by concatenating the second audio features.

[0097] The speech recognition model of the encoder_decoder framework is used to process the first audio feature to obtain the corresponding text information. Then, the trained bidirectional LSTM recurrent neural network model is used to process the text information to obtain a 1*1 semantic integrity feature, where the feature value is between 0 and 1.

[0098] S72: The first facial feature and the first speech feature are fused to obtain the first fused feature.

[0099] The first facial feature and the first speech feature are concatenated to obtain the third fusion feature. Specifically, the first facial feature of 580*68 and the first audio feature of 580*40 are concatenated according to the second dimension to achieve frame-by-frame feature fusion, resulting in a third fusion feature with a feature matrix size of 580*108. Then, the first 11 layers of a multi-layer neural network are used to process the third fusion feature to obtain the first fusion feature of 1*8.

[0100] S73: The second fusion feature is obtained by fusing the first fusion feature and the semantic completeness feature.

[0101] The first fusion feature, which integrates each facial image frame and each speech frame, and the semantic completeness feature are concatenated to obtain the second fusion feature. Specifically, the 1*8 first fusion feature and the 1*1 semantic completeness feature are concatenated to obtain the 1*9 second fusion feature, which integrates the first facial feature, the first speech feature, and the semantic completeness feature of the target object.

[0102] S74: Perform state detection on the second fusion feature to obtain the state detection result for the target object.

[0103] The second fused feature, which integrates the first facial feature, the first speech feature, and the semantic completeness feature of the target object, is input into the 12th layer (state discrimination classification layer) of the multilayer neural network to obtain a 1*2 feature vector. Then, the softmax activation function is used to obtain the probability of the existence of the preset state, or to obtain the probability of the existence of the preset state and the probability of the absence of the preset state respectively, so as to determine the state detection result of the target object. When the probability of the existence of the preset state is greater than the preset threshold, it indicates that the target object has the preset state.

[0104] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0105] Please see Figure 8 , Figure 8This is a schematic diagram of a framework of an embodiment of the state detection device provided in this application. In this embodiment, the state detection device 80 includes an acquisition module 81, a feature extraction module 82, and a state detection module 83. The acquisition module 81 is used to acquire video and audio about a target object. The feature extraction module 82 is used to extract at least a first facial feature of the target object from the video, extract a first speech feature of the target object from the audio, and obtain semantic integrity features of the text information corresponding to the audio based on the audio. The state detection module 83 is used to obtain a state detection result about the target object based at least on the first facial feature, the first speech feature, and the semantic integrity feature. The state detection result is used to determine whether the target object has a preset state.

[0106] In some embodiments, the video acquired by the acquisition module 81 contains multiple frames of facial images of the target object, and the feature extraction module 82 extracts at least a first facial feature of the target object from the video, including: extracting a second facial feature from at least one frame of facial images to obtain the first facial feature; and / or, the audio acquired by the acquisition module 81 contains multiple frames of speech images of the target object, and the feature extraction module 82 extracts speech features of the target object from the audio, including: extracting a second speech feature from at least one frame of speech images to obtain the first speech feature; and / or, obtaining semantic integrity features of text information corresponding to the audio based on the audio, including: extracting the second speech features corresponding to each speech frame; fusing the second speech features to obtain speech fusion features of the audio; performing speech recognition on the speech fusion features to obtain the text information corresponding to the audio; and processing the text information to obtain semantic integrity features of the text.

[0107] In some embodiments, the feature extraction module 82 extracts second facial features from at least one frame of facial images, including: for each frame of facial images, extracting multiple facial key points from the facial images; obtaining spatial relationship values ​​between a first line segment and each second line segment, wherein the first line segment is composed of a line connecting at least two facial key points, and each second line segment is composed of a line connecting a facial key point corresponding to one end of the first line segment and each facial key point; sorting the spatial relationship values ​​corresponding to the facial images in a preset order to obtain the second facial features in the facial images.

[0108] In some embodiments, the state detection module 83 obtains a state detection result about the target object based at least on the first facial feature, the first speech feature, and the semantic integrity feature, including: fusing the first facial feature and the first speech feature to obtain a first fused feature; fusing the first fused feature and the semantic integrity feature to obtain a second fused feature; and performing state detection on the second fused feature to obtain a state detection result about the target object.

[0109] In some embodiments, the video acquired by the acquisition module 81 includes multiple frames of facial images of the target object, the first facial feature includes the second facial features of each frame of facial images, and the audio includes multiple frames of speech images of the target object, the first speech feature includes the second speech features of each frame of speech images; fusing the first facial feature and the first speech feature to obtain a first fused feature includes: fusing each second facial feature to obtain a facial fused feature, and fusing each second speech feature to obtain a speech fused feature, wherein the facial fused feature is the first facial feature, and the speech fused feature is the first speech feature; fusing the facial fused feature and the speech fused feature to obtain a third fused feature; processing the third fused feature using a processing model to obtain the first fused feature. And / or, using the state detection module 83 to perform state detection on the second fused feature to obtain a state detection result about the target object includes: processing the second fused feature using a classification model to obtain a state detection result about the target object.

[0110] In some embodiments, the acquisition module 81 acquires multiple sets of videos and audios about the target object, and the state detection module 83 obtains a state detection result containing the probability that the target object exists in a preset state. After obtaining the state detection results corresponding to each set of videos and audios, the state detection method further includes: obtaining a final state detection result about the target object based on the probability in the state detection results corresponding to each set of videos and audios.

[0111] In some embodiments, the first speech feature extracted by the feature extraction module 82 is obtained using a speech recognition toolkit, the first facial feature is obtained using a feature extraction tool in an open-source face recognition library; and / or, the preset state is the state caused by the neural degeneration of the target object.

[0112] Please see Figure 9 , Figure 9 This is a schematic diagram of an embodiment of the electronic device provided in this application. In this embodiment, the electronic device 90 includes a processor 91 and a memory 92.

[0113] Processor 91 can also be referred to as CPU (Central Processing Unit). Processor 91 may be an integrated circuit chip with signal processing capabilities. Processor 91 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor can be a microprocessor, or processor 91 can be any conventional processor 91, etc.

[0114] The memory 92 in the electronic device 90 is used to store the program instructions required for the processor 91 to run.

[0115] The processor 91 is used to execute program instructions to implement the methods provided in any of the above embodiments and any non-conflicting combinations thereof.

[0116] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of the computer-readable storage medium provided in this application. The computer-readable storage medium 100 of this application embodiment stores program instructions 101, which, when executed, implement the methods provided in any of the above embodiments and any non-conflicting combinations. The program instructions 101 can form a program file and be stored in the computer-readable storage medium 100 in the form of a software product, so that a computer device (which may be a personal computer, server, or network device, etc.) can execute all or part of the steps of the methods of various embodiments of this application. The aforementioned computer-readable storage medium 100 includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets.

[0117] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0118] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0120] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0121] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0122] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0124] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. A state detection method, characterized in that, The method includes: Acquire video and audio about a target object; the video contains multiple frames of facial images of the target object, and the audio contains multiple frames of speech of the target object; At least a first facial feature of the target object is extracted from the video, and a first speech feature of the target object is extracted from the audio. A semantic completeness feature of the text information corresponding to the audio is obtained based on the audio. The first facial feature is obtained using second facial features extracted from at least one frame of facial images, and the first speech feature is obtained using second speech features extracted from at least one frame of the speech image. The semantic completeness feature is obtained by processing the text information, whereby the text information is obtained by performing speech recognition on the speech fusion features of the audio. The speech fusion feature is obtained by fusing the second speech features corresponding to each speech frame. The semantic completeness feature is used to quantify the completeness of the semantic expression of the text information and represents the semantic expression state of the target object. The semantic completeness feature is a feature value obtained by processing the text information using a neural network model. Based at least on the first facial feature, the first voice feature, and the semantic integrity feature, a state detection result for the target object is obtained, and the state detection result is used to determine whether the target object has a preset state caused by neural degeneration.

2. The method according to claim 1, characterized in that, Extracting second facial features from at least one frame of the facial image, including: For each frame of the facial image, multiple facial key points are extracted from the facial image; The spatial relationship values ​​between the first line segment and each second line segment are obtained respectively. The first line segment is composed of a line connecting at least two facial key points, and each second line segment is composed of a line connecting the facial key point corresponding to one end of the first line segment and each facial key point. The spatial relationship values ​​corresponding to the facial image are sorted according to a preset order to obtain the second facial feature in the facial image.

3. The method according to claim 1, characterized in that, The process of obtaining state detection results for the target object based at least on the first facial feature, the first speech feature, and the semantic integrity feature includes: The first facial feature and the first voice feature are fused to obtain the first fused feature; The first fusion feature and the semantic completeness feature are combined to obtain the second fusion feature; The second fused feature is subjected to state detection to obtain the state detection result of the target object.

4. The method according to claim 3, characterized in that, The first facial feature includes the second facial feature of each frame of the facial image, and the first speech feature includes the second speech feature of each frame of the speech image; The process of fusing the first facial feature and the first voice feature to obtain the first fused feature includes: The second facial features are fused to obtain a facial fusion feature, and the second speech features are fused to obtain a speech fusion feature, wherein the facial fusion feature is the first facial feature and the speech fusion feature is the first speech feature; The facial fusion feature and the voice fusion feature are fused to obtain a third fusion feature; The third fusion feature is processed using a processing model to obtain the first fusion feature; And / or, the state detection of the second fused feature to obtain a state detection result regarding the target object includes: The second fused feature is processed using a classification model to obtain the state detection result of the target object.

5. The method according to claim 1, characterized in that, The video and audio about the target object are in multiple sets, and the state detection result includes the probability that the target object exists in the preset state; After obtaining the state detection results corresponding to each group of videos and audio, the method further includes: Based on the probabilities in the state detection results corresponding to the videos and audios in each group, the final state detection result for the target object is obtained.

6. The method according to claim 1, characterized in that, The first speech feature was obtained using a speech recognition toolkit, and the first facial feature was obtained using a feature extraction tool from a face recognition open-source library.

7. A state detection device, characterized in that, The device includes: The acquisition module is used to acquire video and audio about the target object; the video includes multiple frames of facial images of the target object, and the audio includes multiple frames of voice frames of the target object. A feature extraction module is used to extract at least a first facial feature of the target object from the video, extract a first speech feature of the target object from the audio, and obtain a semantic completeness feature of the text information corresponding to the audio based on the audio. The first facial feature is obtained using second facial features extracted from at least one frame of facial images, the first speech feature is obtained using second speech features extracted from at least one frame of the speech image, the semantic completeness feature is obtained by processing the text information, the text information is obtained by performing speech recognition on the speech fusion features of the audio, the speech fusion feature is obtained by fusing the second speech features corresponding to each speech frame, the semantic completeness feature is used to quantify the completeness of the semantic expression of the text information, and the semantic completeness feature is used to represent the semantic expression state of the target object. The semantic completeness feature is a feature value obtained by processing the text information using a neural network model. The state detection module is used to obtain a state detection result for the target object based at least on the first facial feature, the first voice feature and the semantic integrity feature. The state detection result is used to determine whether the target object has a preset state caused by neural degeneration.

8. An electronic device, characterized in that, Including interconnected memory and processor, The memory stores program instructions; The processor is used to execute program instructions stored in the memory to implement the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program instructions that can be executed to implement the method of any one of claims 1-6.

Images