Cognitive test method, cognitive test apparatus, electronic device and storage medium

Abstract

Disclosed in the present application are a cognitive test method, a cognitive test apparatus, an electronic device and a computer-readable storage medium. The method comprises: on the basis of an answer of a user to a cognitive test question, acquiring data to be evaluated, wherein the data to be evaluated comprises: vital sign data of the user, video data containing face information of the user and audio data containing an answer audio of the user; extracting face key point data contained in the video data; performing answer discrimination according to the face key point data and the audio data to obtain a voice discrimination result of a cognitive test; and performing state analysis according to the face key point data, an intermediate output of the answer discrimination and the vital sign data to obtain a state analysis result of the cognitive test. The solution of the present application can achieve whole-process evaluation of specialized voice interaction for cognitive tests.

Description

A cognitive detection method, cognitive detection device, electronic device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on June 30, 2023, with application number 202310803172.1 and invention name “A cognitive detection method, cognitive detection device, electronic device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application belongs to the field of intelligent interaction technology, and in particular relates to a cognitive detection method, a cognitive detection device, an electronic device, and a computer-readable storage medium. Background Art

[0003] The general logic of current voice assessment and interaction systems is that an Automatic Speech Recognition (ASR) system converts speech into text, which is then processed by a Natural Language Processing (NLP) system. Currently, ASR and NLP services are often deployed independently in the cloud, and full-process systems are rare. In particular, a comprehensive assessment system for specialized voice interactions for cognitive testing is still lacking. Technical issues

[0004] The present application provides a cognitive detection method, a cognitive detection device, an electronic device, and a computer-readable storage medium, which can realize the full-process evaluation of specialized voice interaction for cognitive detection. Technical Solutions

[0005] In a first aspect, the present application provides a cognitive detection method, comprising:

[0006] Based on the user's answers to the cognitive test questions, the data to be evaluated is obtained, which includes: the user's vital signs data, video data containing the user's facial information, and audio data containing the audio of the user's answers;

[0007] Extracting facial key point data contained in the video data;

[0008] The answer is judged based on the facial key point data and audio data to obtain the speech recognition result of the cognitive test;

[0009] State analysis is performed based on facial key points, intermediate outputs of answer judgment, and vital signs data to obtain the state analysis results of cognitive detection.

[0010] In a second aspect, the present application provides a cognitive detection device, comprising:

[0011] An acquisition module is used to acquire data to be evaluated based on the user's answers to cognitive test questions. The data to be evaluated includes: user's vital sign data, video data containing user's facial information, and audio data containing the user's answers;

[0012] An extraction module is used to extract facial key point data contained in the video data;

[0013] The discrimination module is used to discriminate the answer based on the facial key point data and audio data to obtain the speech discrimination result of cognitive detection;

[0014] The analysis module is used to perform state analysis based on facial key points, intermediate output of answer judgment and vital signs data to obtain the state analysis results of cognitive detection.

[0015] In a third aspect, the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method according to the first aspect are implemented.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method of the first aspect are implemented.

[0017] In a fifth aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed by one or more processors, it implements the steps of the method of the first aspect. Beneficial effects

[0018] Compared with the existing technology, the beneficial effects of this application are as follows: first, based on the user's answers to cognitive detection questions, the data to be evaluated is obtained, and the data to be evaluated includes: the user's vital sign data, video data containing the user's facial information, and audio data containing the user's answer audio; then, the facial key point data contained in the video data is extracted; then, the answer is judged based on the facial key point data and audio data to obtain the voice discrimination result of the cognitive detection; and the state analysis is performed based on the facial key points, the intermediate output of the answer judgment, and the vital sign data to obtain the state analysis result of the cognitive detection. Through the present application scheme, the advantages of ASR and NLP are integrated, and a full-process offline and partially streaming cognitive detection method is realized, eliminating the dependence on the network.

[0019] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] FIG1 is a schematic diagram of an implementation flow of a cognitive detection method provided in an embodiment of the present application;

[0022] FIG2 is an exemplary diagram of the architecture of a speech evaluation interaction system provided in an embodiment of the present application;

[0023] FIG3 is an example diagram of a process for determining a category question based on phoneme fuzzy matching according to an embodiment of the present application;

[0024] FIG4 is an example diagram of a process for determining a matching question based on voice feature comparison according to an embodiment of the present application;

[0025] FIG5 is an example diagram of a probability matrix provided in an embodiment of the present application;

[0026] FIG6 is an example diagram of a process for determining a matching question based on hybrid fuzzy matching according to an embodiment of the present application;

[0027] FIG7 is a diagram illustrating the structures of an original RNNT and an optimized multi-task RNNT provided in an embodiment of the present application;

[0028] FIG8 is a diagram illustrating a specific architecture of an interactive analysis module according to an embodiment of the present application;

[0029] FIG9 is a structural block diagram of a cognitive detection device provided in an embodiment of the present application;

[0030] FIG10 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. Modes for Carrying Out the Invention

[0031] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0032] The following describes a cognitive detection method provided in an embodiment of the present application. The cognitive detection method can be applied to an electronic device equipped with a speech assessment interaction system, that is, an electronic device with a cognitive detection function. For example only, the electronic device can be a computer or a smartphone, etc., which is not limited here. Referring to FIG. 1 , the cognitive detection method in the embodiment of the present application includes:

[0033] Step 101: Obtain data to be evaluated based on the user's answers to cognitive testing questions.

[0034] The electronic device may be equipped with an output device and an input device. Among them, the output device includes but is not limited to a display screen (touch screen) and a speaker, and the input device includes but is not limited to a camera, a microphone and a vital sign detection device. The embodiment of the present application does not limit the types of the output device and the input device. After the cognitive detection begins, the electronic device can output cognitive detection questions to the user through an output device in the form of voice or video; when the user answers the cognitive detection questions based on the output of the electronic device, the electronic device can collect the user's answers to the cognitive detection questions through its input device. Based on the user's answer, the electronic device can obtain data to be evaluated. In some examples, the data to be evaluated includes but is not limited to: the user's vital sign data (specifically, the vital sign data when the user answers the questions), video data containing the user's facial information (specifically, the facial information when the user answers the questions), and audio data containing the user's answer audio.

[0035] Step 102: extracting facial key point data contained in the video data.

[0036] Since the video data contains the user's facial information, the electronic device can perform image analysis on the video data to extract the facial key point data contained in the video data. In some examples, the facial key point data includes but is not limited to: facial key point coordinate data and key point distortion degree data.

[0037] In some embodiments, the electronic device can perform facial key point recognition on the video data to obtain facial key point coordinate data. It can also perform head contour recognition on the video data to obtain user head contour data. The facial key point coordinate data and the user head contour data are then compared with pre-set standard facial data to obtain key point distortion data. The standard facial data includes standard facial key point coordinate data and standard head contour data. It is understood that this standard facial data can be obtained through calibration when the voice assessment interactive system is started and can be used as reference data for comparison.

[0038] Specifically, the comparison process for facial key point coordinate data and the user's head profile data with pre-set standard facial data can be as follows: The current user's head profile data and the standard head profile data are used to calculate the user's current facial zoom ratio to eliminate distance errors; the current facial key point coordinate data is then scaled and adjusted based on the current facial zoom ratio, and the adjusted data is compared with the standard facial key point coordinate data to determine the horizontal and vertical stretching and compression of each key point. It is understood that the key point distortion data can be used to describe whether the user's face is consistent with the standard state. In other words, from a facial perspective, it can be used to determine whether the user's attention is consistent with the standard state, thereby determining whether the user is distracted or distracted.

[0039] Step 103: perform answer discrimination based on the facial key point data and the audio data to obtain a speech discrimination result of cognitive detection.

[0040] The electronic device can identify the answer based on the audio data, thereby obtaining a speech recognition result for cognitive testing. This speech recognition result indicates whether the user's answer is accurate. Because the accuracy of the answer is related to the user's cognitive state, this speech recognition result also indicates whether the user's cognitive state is impaired or abnormal.

[0041] To reduce the impact of potential environmental noise and speech overlap on the accuracy of speech recognition, in this embodiment of the application, facial key point data and audio data can be combined to perform answer recognition. The facial key point data primarily plays a role in environmental detection. If the results of the environmental detection determine that there are no issues such as environmental noise and speech overlap that could affect the accuracy of speech recognition, the current speech recognition result obtained based on the analysis of the audio data can be considered relatively accurate.

[0042] Step 104 , performing state analysis based on the facial key point data, the intermediate output of the answer judgment, and the vital sign data to obtain the state analysis result of the cognitive detection.

[0043] During the cognitive test, the user may not hear the cognitive test questions clearly, or may encounter special situations such as obstacles in the interactive function due to emotional excitement, which will affect the results of the cognitive test. When necessary, the user is usually guided based on the user's status to make the interactive process of cognitive testing more natural. Considering that the user's status may be reflected in the facial expression or demeanor when answering questions, it may also be reflected in vital signs such as heart rate and blood pressure, and it may also be reflected in the answers made when answering questions, therefore, in the embodiment of the present application, the electronic device can perform state analysis based on the facial key point data, the intermediate output of the answer judgment and the vital sign data to obtain the state analysis result of the cognitive test. Among them, the intermediate output of the answer judgment includes but is not limited to one or more of the following: speech features, phonemes, text and speech discrete errors.

[0044] Please refer to Figure 2, which shows an example of the architecture of a speech assessment and interaction system that applies the cognitive detection method proposed in an embodiment of the present application. The speech assessment and interaction system 2 includes: a key point detection module 201, a speech recognition module 202, and an interaction analysis module 203. The following describes the input and output of each module based on steps 101-104:

[0045] The input of the key point detection module 201 is real-time video data, and the output is facial key point data;

[0046] The voice recognition module 202 receives inputs such as facial landmark data and real-time audio data, and outputs voice recognition results. The module also performs environmental detection, the results of which can be used to control subsequent processes and output. Specifically, the voice recognition results are available in two types: real-time voice recognition results and final voice recognition results. Furthermore, the module also provides intermediate outputs.

[0047] The input of the interactive analysis module 203 is the facial key point data, the intermediate output of the voice recognition module (ie, voice features, etc.) and real-time vital sign data, and the output is the real-time status analysis result.

[0048] In some embodiments, the cognitive testing questions involving speech types can be roughly categorized into three types: category questions, matching questions, and similarity questions. As an example, category questions can be questions like "Please name __ words that meet ___ characteristics," matching questions can be questions like "Please repeat the following words in order: ______," and similarity questions can be questions like "Please repeat the following sentence: _____." Based on these different types of cognitive testing questions, the specific process of answering questions can be:

[0049] A1. Obtain the intermediate output of answer discrimination based on facial key point data and audio data.

[0050] The electronic device can process the audio data using an acoustic-related model (i.e., an acoustic model) to extract valid information related to the user's answer from the audio data. This valid information is the intermediate output. The data types that the intermediate output may contain have been described above and will not be repeated here. It can be understood that the intermediate output is the information required for subsequent processing.

[0051] As previously described, to reduce the impact of potential environmental noise and speech overlap on speech recognition accuracy, the audio data may be combined with facial landmark data for environmental detection during the processing of the audio data to generate the intermediate output. It is understood that only when the environmental detection results determine that there are no issues such as environmental noise and speech overlap that could affect speech recognition accuracy can the electronic device obtain a valid intermediate output and perform subsequent recognition operations based on this intermediate output.

[0052] A2. Determine the information to be discriminated among phonemes, text, and speech features based on the target question type of the cognitive test question.

[0053] The three common question types mentioned above are described above. Each question type only requires a portion of the intermediate output to determine the answer. Based on this, the electronic device can determine the information to be determined based on the target question type of the current cognitive test question from the obtained intermediate output.

[0054] Specifically, when the cognitive detection question is a category question, the information to be judged may be phonemes; when the cognitive detection question is a matching question, the information to be judged may be speech features; when the cognitive detection question is a similarity question, the information to be judged may be text and speech features.

[0055] A3. Use the discrimination method corresponding to the target question type to perform question answer discrimination on the discrimination information to obtain the speech discrimination result.

[0056] Different types of cognitive test questions can use different discrimination methods. Specifically, when the cognitive test question is a category question, the corresponding discrimination method can be phoneme fuzzy matching; when the cognitive test question is a matching question, the corresponding discrimination method can be voice feature comparison; when the cognitive test question is a similarity question, the corresponding discrimination method can be mixed fuzzy matching, which includes: fuzzy matching with voice features as input, and fuzzy matching with text as input. The following explains and introduces each discrimination method:

[0057] The corresponding discrimination method for categorical questions is phoneme fuzzy matching. The reason for this is that, compared to text, it has the same streaming difficulty but is more inclusive in speech environments. Compared to speech feature comparison, while less inclusive, due to the large number of entries in each category, speech feature comparison imposes a significant computational burden, impacting the system's real-time performance. Therefore, to achieve a balance between inclusiveness and computational power, phoneme fuzzy matching is used as the discrimination method for categorical questions. Specifically, please refer to Figure 3, which illustrates an example of the process for discriminating categorical questions based on phoneme fuzzy matching. As shown in Figure 3, when discriminating categorical questions, the input consists of two parts: the cognitive test question and the phonemes in the intermediate output. The cognitive test question is typically a specific category, such as "animal" or "food." By pre-collecting and converting the category data for this specific category, we can obtain the category word list and phoneme list corresponding to the cognitive test question. It can be understood that the text corresponding to the specified category can be obtained from the category vocabulary, and the phonemes corresponding to the specified category can be obtained from the phoneme table. Subsequently, subsequence fuzzy matching is used to match the phonemes extracted from the audio data with the text in the category vocabulary and the phonemes in the phoneme table. During the subsequence fuzzy matching process, a matching state table can be constructed and updated each time a phoneme is input. Then, a dynamic programming (DP) algorithm can be used to achieve streaming.

[0058] The matching problem is identified using voice feature comparison, as this method minimizes errors introduced during the recognition process and increases the system's robustness. Specifically, please refer to Figure 4, which illustrates an example of the matching problem identification process based on voice feature comparison. As shown in Figure 4, the input for identifying a matching problem is the cognitive test problem and the voice features from the intermediate output. Generally speaking, a cognitive test problem consists of the text to be matched and sequence information. For example, the text "ABCDE" needs to be matched, where the three characters "ABC" can appear interchangeably but must all come before "DE." Based on the extracted voice features, the electronic device calculates the probability that each audio segment corresponds to a specific character. This probability calculation is similar to that used in connectionist temporal classification (CTC) recognition. After arranging the features in audio order, a probability matrix is ​​obtained for each audio segment's corresponding character. The probability of a matching problem is calculated by selecting the path in the probability matrix that generates the largest product of the probability of matching the text.

[0059] Please refer to Figure 5, which shows an example of a probability matrix. Taking the two text sequences "rhinoceros ant" and "ant rhino" as examples, in the probability matrix shown in Figure 5, the maximum probability product of the text sequence "ant rhino" is 0.5832, while the maximum probability product of the text sequence "rhinoceros ant" is 0.00144. Unlike CTC, CTC will calculate the impact of other slices being misidentified as "unk", but in the matching process in this scenario, the presence of such colloquialisms and additional text should not affect the matching process; based on this, the embodiment of the present application uses a method similar to CTC for recognition, which differs from CTC in that: in the embodiment of the present application, only the probability of the required matching text sequence is counted, and the influence of other noise is ignored. It can be understood that this calculation process can be streamed using the DP algorithm.

[0060] The discrimination method for similar questions is hybrid fuzzy matching. Specifically, please refer to Figure 6, which shows an example of the process of judging matching questions based on hybrid fuzzy matching. As shown in Figure 6, when judging similar questions, the input is the cognitive test question and the speech features and text in the intermediate output. Fuzzy matching with speech features as input is similar to audio feature matching corresponding to the matching question, and it outputs a probability matrix. Fuzzy matching with text as input aims to perform semantically similar fuzzy matches. For example, "I eat dinner at 7 o'clock" and "I eat dinner at 7 o'clock" have different pronunciations and cannot be judged as the same through audio feature matching. Therefore, in fuzzy matching with text as input, syntactic analysis is used to analyze the part of speech of each word in the sentence. A set of coefficients (stored in the sentence component weight table) is used to determine the error impact of each part of speech. For example, the error of noun errors is much higher than that of verb errors. The coefficients are usually statistically obtained based on pre-collected data and are updated when the pre-collected data is updated. The word fuzzy probability is then calculated using word similarity. Combining these two probabilities allows for words with the same meaning but different expressions during the retelling process, thereby calculating the likelihood that the retelling result is semantically similar to the original. Through this process, two probabilities are calculated for each word in the text: the probability of being recognized as the corresponding character in the answer (obtained through fuzzy matching using speech features as input), and the probability of being semantically similar to the answer multiplied by the probability of being recognized as a similar word (obtained through fuzzy matching using the text as input).

[0061] The following formula explains this: A is the standard answer character for the target character; B is the actual recognized character for the target character; C is the previous character of the target character that is actually recognized, which can form a vocabulary with A and B respectively; thus, CA is the vocabulary at the corresponding position in the standard answer, and CB is the vocabulary at the corresponding position in the recognition result. The above process can be expressed as:

[0062] p(A) = max(p_sound(A), p_char_sim(A))

[0063] p_char_sim(A) = p_sound(B) × p_similar(CA, CB)

[0064] Taking the two characters "dinner" and "supper" as an example: For the character "meal", based on the above formula, it can be known that the result of calculating the probability of being recognized as "meal" (i.e., p_sound(B)) multiplied by the semantic similarity between "dinner" and "supper" (i.e., p_similar(CA, CB)) can be used as the probability of semantic similarity (i.e., p_char_sim(A)). This probability (i.e., p_char_sim(A)) will then be compared with the probability of being recognized as "rice" at this position (i.e., p_sound(A)), and the larger value will be taken as its final probability.

[0065] It should be noted that the discrimination of category questions and the discrimination of matching questions are both streaming methods; in the discrimination of similarity questions, the fuzzy matching with speech features as input is a streaming method, and its result can be used for real-time interaction; however, the fuzzy matching with text as input is a non-streaming process, which needs to wait until the ASR recognition is completed before it can be carried out.

[0066] In some embodiments, the electronic device can process to obtain various intermediate outputs through an improved acoustic model, and step A1 may include:

[0067] Step B1: Extract the acoustic features of the audio data.

[0068] The electronic device can extract the acoustic features of the audio data through operations such as preprocessing and feature transformation. Currently, the extraction of acoustic features has been a relatively mature technology and will not be elaborated here.

[0069] Step B2: Encode the acoustic features through a preset acoustic encoder and obtain speech features based on the encoding results.

[0070] The acoustic model includes an acoustic encoder. Specifically, the acoustic encoder includes but is not limited to: a text encoder and a phoneme encoder. It should be noted that a part of the features of the text encoder comes from the extraction of the phoneme encoder, which can reduce the number of parameters of the neural network in the text encoder; that is, the input of the phoneme encoder completely depends on the acoustic features, and the input of the text encoder partly depends on the acoustic features and partly depends on the extraction of the phoneme encoder.

[0071] Step B3: Perform environment detection based on the speech features, acoustic features, and face key point data to determine the environment detection result.

[0072] After obtaining speech features through the acoustic encoder, these features can be used as input for environmental detection. Acoustic features can also serve as input for environmental detection. It can be understood that acoustic features are used to determine the level of ambient noise, while speech features are used to determine whether there is speech overlap. Furthermore, environmental detection can be combined with facial landmark data. For example, facial landmarks can be used to determine whether the user is currently speaking, helping to visually determine whether there is speech overlap.

[0073] Step B4: When the environmental detection result meets the preset environmental conditions, the encoding result is processed by a preset acoustic language model to obtain phonemes and text.

[0074] The acoustic model also includes an acoustic language model. When the environmental detection results meet the specified conditions, the current audio data is considered to accurately represent the user's response without interference. Based on this, the encoded results are processed using the acoustic language model to obtain phonemes and text. Specifically, the preset environmental conditions include the absence of speech overlap and the ambient noise level being below a preset noise threshold.

[0075] In some examples, the acoustic model adopted in the embodiments of the present application can use the Recurrent Neural Network Transducer (RNNT) as the basic model, and the model itself is a streaming model. In order to meet the requirements of multiple types of outputs, the embodiments of the present application can also optimize the RNNT model and optimize it into a multi-task RNNT. Please refer to Figure 7, which shows an example of the structure of the original RNNT (left side of Figure 7) and the optimized multi-task RNNT (right side of Figure 7). Among them, the variable fusion, language model and encoder in the multi-task RNNT are deep learning neural networks, which belong to the prior art and will not be described here. As can be seen from Figure 7, the output of the multi-task RNNT includes: text, phonemes, speech features and speech discrete errors, which are intermediate outputs in the question answering process.

[0076] The following explains and illustrates the optimizations made by the multi-task RNNT compared to the original RNNT with reference to Figure 7:

[0077] First, the multi-task RNNT consists of two RNNT structures, one for text and one for phoneme output. As you can see, each of these components is identical to the basic RNNT, but some features of the text encoder are derived from the phoneme encoder, as described previously and will not be repeated here.

[0078] Secondly, in the multi-task RNNT, a speech discretization error is obtained by weighted summation of the discretization loss during prediction. This discretization error serves as an intermediate output and is used in subsequent state analysis. Discretization refers to the difference between the maximum probability in the prediction result and the probability of 1. A simple calculation of the discretization loss is loss = -log p, where p is the probability of the category indicated by the prediction result. Weighting refers to the weighted summation of the text discretization loss and the phoneme discretization loss.

[0079] In addition, an environmental detection control is added between the encoder (including the text encoder and the phoneme encoder) and the model fusion. It can be understood that when the environmental detection result does not meet the preset environmental conditions (that is, the environment is relatively noisy), the model fusion (that is, the variable fusion) will remain waiting due to lack of input. Based on this, in a noisy environment, the confusion of the language model caused by the simultaneous recognition of audio from multiple sources by the language model (including the audio text language model and the audio phoneme language model) due to speech overlap in time can be reduced; at the same time, the situation where the output results are invalid can also be reduced. Considering that word-by-word output takes up a lot of computing power, this can reduce the computing power requirements of the multi-task RNNT when it is in use.

[0080] In addition, when constructing a multi-task RNNT, in order to realize the offline operation of the ASR function, the overall structure of the model should be smaller. To this end, the embodiment of the present application adopts distillation and quantization to achieve model compression. Specifically, distillation is a special training method that can convert the knowledge of a trained large model into a small model. Compared with directly using a small model, its training results are usually more stable; quantization is a way to reduce memory by reducing the number of bits of parameters stored in the computer. In the embodiment of the present application, different distillation methods are used for different parts of the multi-task RNNT, which are characterized by: first shielding the text part, using the large phoneme model to distill the phoneme part of the multi-task RNNT for training, and then using the large text model for distillation training on the text part.

[0081] In some embodiments, step 104 may include:

[0082] C1. Extract specific features based on the intermediate output.

[0083] Part or all of the intermediate outputs obtained by the electronic device can be used as input for state analysis. In some examples, the intermediate outputs that can be used as input for state analysis can include text, speech features, and speech discrete errors. The electronic device can use a feature extractor to extract features from these intermediate outputs, thereby obtaining specific features.

[0084] Considering that speech discrete loss and speech features are streamed information, a new batch of data is typically generated at regular intervals. Therefore, the electronic device can extract features from the speech discrete loss and speech features using a speech feature extractor, thereby obtaining a streamed specific feature. To facilitate distinction, this embodiment of the present application refers to the streamed specific feature as the first specific feature. That is, the electronic device can extract the first specific feature based on the speech discrete error and speech features.

[0085] Considering that the text is a single-round information, meaning that the text is typically obtained only once per round, the electronic device can extract features from the text using a semantic feature extractor, thereby obtaining specific semantic features related to the single round. To facilitate distinction, the embodiments of this application refer to specific semantic features related to the single round as second specific features. In other words, the electronic device can extract the second specific features from the text.

[0086] C2. Extract common features based on vital signs data and facial key point data.

[0087] The vital sign data detected by the electronic device through the vital sign detection device may include but is not limited to heart rate, blood pressure, and blinking frequency. The facial key point data has been described in the previous article and will not be repeated here. By extracting features from the vital sign data through the medical feature extractor and the Exploratory Data Analysis (EDA) feature extractor, the user's physical sign features can be obtained; by extracting features from the facial key point data through the attention feature extractor, visual features can be obtained. Among them, physical sign features and visual features are both streamed features, and the two can be collectively regarded as universal features.

[0088] In some examples, the medical feature extractor focuses on features artificially designed based on medical features, including but not limited to heart rate variability; the EDA feature extractor is a set of methods designed artificially based on data patterns. Its input is vital sign data and medical features extracted by the medical feature extractor. The purpose is to extract as many effective features as possible that have not yet been represented in the current medical features.

[0089] In some examples, the meaning of "attention" in the attention feature extractor is "determining where the target's attention is" or "determining whether the target is paying attention to a certain entity", which has no direct connection with whether the underlying neural network uses the attention mechanism.

[0090] C3. Perform feature fusion based on specific features and general features to obtain fused features;

[0091] Taking into account that the features are divided into streaming features and single-round features, the electronic device can perform feature fusion based on this. Specifically, the first specific feature and the general feature are both streaming features, so the electronic device can perform feature fusion based on the first specific feature and the general feature, thereby obtaining the first overall feature. It can be understood that the first overall feature is the streaming overall feature. On this basis, the electronic device can further perform statistics on all the first overall features obtained in a single round to obtain single-round statistical information, and then perform feature fusion based on the single-round statistical information and the second specific feature to obtain the second overall feature. The second overall feature is the overall feature of the single round.

[0092] C4. Perform analysis based on the fusion features to obtain the status analysis results.

[0093] The electronic device can perform normalization and logistic regression processing on the two obtained overall features, thereby obtaining a first state analysis result corresponding to the first overall feature and a second state analysis result corresponding to the second overall feature. It can be understood that the first state analysis result is a streamed analysis result, and the second state analysis result is a single-round analysis result. Both can be expressed in the form of multiple probability values ​​to express different dimensions of user status, such as language fluency or subject excitement.

[0094] Please refer to Figure 8, which shows a specific architecture example of the interaction analysis module. The following describes the state analysis process proposed above through a specific simple example in conjunction with Figure 8:

[0095] Assume that during a round of voice interaction in cognitive testing, the intermediate output generated by the user's answer is "This is a kettle." If the speech is divided into 10 time slices of predetermined length, each time slice will also have two intermediate outputs: speech discrete error and speech features. Based on the speech discrete error and speech features, the electronic device can extract a first specific feature (also known as a streamed specific feature). This first specific feature can be represented as a latent vector in vector form, for example, (100.2, 1.3).

[0096] Visual features primarily consist of two components: facial keypoint coordinate data and keypoint distortion data. After attention feature extraction, a new latent vector is obtained, similar in format to the latent vector corresponding to the first specific feature, for example, (20.1, -18.9). Ultimately, the electronic device outputs this latent vector and keypoint distortion data, which takes the form of {x-axis distortion: 15.3, y-axis distortion: 20.8, facial zoom factor: 1.05}. Physical features are sampled as a set of results at each time slice, for example, {heart rate: 81, blink rate: 24}. Medical knowledge indicates that heart rate variability is an effective tool for assessing a person's condition. Therefore, a medical feature extractor can be used to extract the features corresponding to these sampled results, for example, {heart rate variability: +10}. Furthermore, based on prior observations and research, blink rate correlates strongly with the degree of continuity in the analysis results. Therefore, blink rate extraction can be added to the EDA feature extractor, removing any correlation with existing features. The format is the same as above. Finally, the electronic device can fuse the obtained visual features and vital signs features (i.e., universal features) with the first specific feature. In some examples, this fusion can be performed by feature concatenation, where the visual features and vital signs features are added as additional dimensions to the latent vector of the first specific feature to form a new feature vector, such as (100.2, 1.3, 10, 24, 20.1, -18.9, 15.3, 20.8, 1.05).

[0097] After normalizing the values ​​of each dimension in the resulting feature vector, a probability value can be obtained using logistic regression. It should be noted that the analysis coefficients used in different analyses vary. These coefficients are obtained through existing training methods and are not detailed here. Ultimately, the electronic device obtains a result in the form of {language fluency: 0.82, subject agitation: 0.11}, which is the result of a single streamed analysis. Since there are 10 time slices, 10 streamed analysis results are obtained. Clearly, the calculation process for each streamed analysis result generates a feature vector, resulting in a total of 10 feature vectors. The values ​​of these 10 feature vectors are averaged to obtain the single-run streamed features (also known as single-run statistical information). After the speech ends, ASR completes text recognition, producing the intermediate output of text. After semantic feature extraction of the text, the second specific feature, also known as the single-run specific feature, is obtained. Its form is similar to the latent vector described above. The second specific feature is combined with the streaming feature of the single-round statistics by splicing for feature fusion, and then normalization and logistic regression are performed to obtain the analysis results of the single round, which can be expressed in the form of {question interaction correlation: 0.66, interactive spoken language unclearness: 0.23}.

[0098] It's understandable that streaming analysis results (also known as first-state analysis results) primarily assess the user's current interaction state to ensure smooth interaction. These results are generally real-time, for example, determining whether the user's language is fluent and whether their emotions are unusual. These characteristics often change continuously and significantly during interaction. Single-round analysis results (also known as second-state analysis results) primarily assess the interaction state of that round. This is generally a comprehensive assessment of the entire interaction, encompassing a complete process to determine whether the user clearly understands the cognitive test questions and can respond correctly.

[0099] As can be seen from the above, the embodiment of the present application realizes the automatic discrimination and interaction state analysis of speech-based cognitive test questions. Specifically, in terms of automatic discrimination, the embodiment of the present application classifies and designs discrimination methods that are closer to human logical patterns for different types of cognitive test questions, which can further reduce the difference between automatic discrimination and human discrimination results; in terms of interaction state analysis, the embodiment of the present application integrates vital sign data to form a state analysis of the interaction process, and extracts reusable general feature coefficients and feature construction methods. These coefficients and methods can analyze the interaction state based on general features. In addition, the embodiment of the present application also optimizes ASR and NLP, wherein the matching process of NLP is designed to be streamed and integrated with ASR to form a full-process voice module, which can realize the simultaneous use of voice features and text, providing a basis for a discrimination method that is more in line with human logic and reducing the waiting time from completing the answer to the end of the discrimination; ASR completes the miniaturization of the model through distillation and quantization, and NLP uses fewer deep learning models, which can reduce its overall memory usage and the amount of calculation, thereby completing the offline of the entire system and eliminating the possibility that the system is affected by network instability and cloud service concurrency limitations and affects the quality of discrimination. Furthermore, the embodiment of the present application also adds the function of environmental detection to ensure the controllability of the voice link environment, and reduces the confusion of the language model and invalid computing power caused by multiple people speaking at the same time by interrupting decoding. Finally, the embodiment of the present application forms a complete voice evaluation interaction system by integrating the above advantages. Compared with the existing technology, it can improve the accuracy of voice-based cognitive detection interaction, and can analyze the user's interaction status as a basis for interaction strategy, thereby helping to improve the experience of the cognitive detection interaction process.

[0100] Corresponding to the cognitive detection method provided above, the present embodiment further provides a cognitive detection device, which can be integrated into an electronic device. Referring to FIG9 , the cognitive detection device 9 in the present embodiment includes:

[0101] Acquisition module 901 is used to acquire data to be evaluated based on the user's answers to the cognitive test questions. The data to be evaluated includes: user's vital sign data, video data containing user's facial information, and audio data containing the user's answers;

[0102] Extraction module 902, used to extract facial key point data contained in the video data;

[0103] The discrimination module 903 is used to discriminate the answer based on the facial key point data and the audio data to obtain the speech discrimination result of the cognitive test;

[0104] The analysis module 904 is used to perform state analysis based on the facial key point data, the intermediate output of the answer judgment and the vital sign data to obtain the state analysis result of the cognitive detection.

[0105] In some embodiments, the determination module 903 includes:

[0106] An intermediate output determination unit, configured to obtain an intermediate output for answer determination based on facial key point data and audio data, the intermediate output including: speech features, phonemes, and text;

[0107] a unit for determining information to be determined, for determining information to be determined from phonemes, text, and speech features according to a target question type of the cognitive test question;

[0108] The answer discrimination unit is used to use the discrimination method corresponding to the target question type to perform answer discrimination on the discrimination information to obtain the speech discrimination result.

[0109] In some embodiments, the intermediate output determination unit includes:

[0110] An acoustic feature extraction subunit, used to extract acoustic features of audio data;

[0111] a first encoding subunit, configured to encode the acoustic features using a preset acoustic encoder and obtain speech features based on the encoding results;

[0112] The environment detection subunit is used to perform environment detection based on voice features, acoustic features and facial key point data and determine the environment detection results;

[0113] The second encoding subunit is used to process the encoding result through a preset acoustic language model to obtain phonemes and text when the environmental detection result meets the preset environmental conditions.

[0114] In some embodiments, the analysis module 904 includes:

[0115] A specific feature extraction unit, used to extract specific features based on the intermediate output;

[0116] A general feature extraction unit, used to extract general features based on vital sign data and facial key point data;

[0117] A feature fusion unit is used to fuse features based on specific features and general features to obtain fused features;

[0118] The state analysis unit is used to perform analysis based on the fusion features to obtain the state analysis results.

[0119] In some embodiments, the specific feature includes: a first specific feature and a second specific feature, wherein the first specific feature is a streaming specific feature and the second specific feature is a single-round specific feature; the intermediate output includes: speech discrete error, speech features, and text; the specific feature extraction unit includes:

[0120] A first specific feature extraction subunit is used to extract a first specific feature based on the speech discrete error and the speech feature;

[0121] The second specific feature extraction subunit is used to extract the second specific feature based on the text.

[0122] In some embodiments, the fused feature includes: a first overall feature and a second overall feature, wherein the first overall feature is a streamed overall feature and the second overall feature is a single-round overall feature; and the feature fusion unit includes:

[0123] A first feature fusion subunit is used to perform feature fusion based on the first specific feature and the general feature to obtain a first overall feature;

[0124] A statistical subunit, configured to perform statistics on the first overall feature to obtain single-round statistical information;

[0125] The second feature fusion subunit is used to perform feature fusion based on the single-round statistical information and the second specific feature to obtain the second overall feature.

[0126] In some embodiments, the facial key point data includes: facial key point coordinate data and key point distortion degree data; the extraction module 902 includes:

[0127] A key point recognition unit is used to recognize key points of a face in video data and obtain coordinate data of key points of the face;

[0128] A contour recognition unit, used to perform head contour recognition on video data to obtain user head contour data;

[0129] The comparison unit is used to compare the facial key point coordinate data and the user's head contour data with the preset facial standard data to obtain key point distortion degree data, wherein the facial standard data includes: facial key point standard coordinate data and head contour standard data.

[0130] As can be seen from the above, the embodiment of the present application realizes the automatic discrimination and interaction state analysis of speech-based cognitive test questions. Specifically, in terms of automatic discrimination, the embodiment of the present application classifies and designs discrimination methods that are closer to human logical patterns for different types of cognitive test questions, which can further reduce the difference between automatic discrimination and human discrimination results; in terms of interaction state analysis, the embodiment of the present application integrates vital sign data to form a state analysis of the interaction process, and extracts reusable general feature coefficients and feature construction methods. These coefficients and methods can analyze the interaction state based on general features. In addition, the embodiment of the present application also optimizes ASR and NLP, wherein the matching process of NLP is designed to be streamed and integrated with ASR to form a full-process voice module, which can realize the simultaneous use of voice features and text, providing a basis for a discrimination method that is more in line with human logic and reducing the waiting time from completing the answer to the end of the discrimination; ASR completes the miniaturization of the model through distillation and quantization, and NLP uses fewer deep learning models, thereby reducing its overall memory usage and the amount of calculation, thereby completing the offline operation of the entire system and eliminating the possibility that the system is affected by network instability and cloud service concurrency limitations and affecting the discrimination quality. Furthermore, the embodiment of the present application also adds the function of environmental detection to ensure the controllability of the voice link environment, and reduces the confusion of the language model and invalid computing power caused by multiple people speaking at the same time by interrupting decoding. Finally, the embodiment of the present application forms a complete voice evaluation interaction system by integrating the above advantages. Compared with the existing technology, it can improve the accuracy of voice-based cognitive detection interaction, and can analyze the user's interaction status as a basis for interaction strategy, thereby helping to improve the experience of the cognitive detection interaction process.

[0131] Corresponding to the cognitive detection method provided above, an embodiment of the present application further provides an electronic device. Referring to Figure 10, the electronic device 10 in the embodiment of the present application includes: a memory 1001, one or more processors 1002 (only one is shown in Figure 10) and a computer program stored in the memory 1001 and executable on the processor. Wherein: the memory 1001 is used to store software programs and modules, and the processor 1002 executes various functional applications and data processing by running the software programs and units stored in the memory 1001 to obtain the resources corresponding to the above-mentioned preset events. Specifically, the processor 1002 implements the following steps when running the above-mentioned computer program stored in the memory 1001:

[0132] Based on the user's answers to the cognitive test questions, the data to be evaluated is obtained, which includes: the user's vital signs data, video data containing the user's facial information, and audio data containing the audio of the user's answers;

[0133] Extracting facial key point data contained in the video data;

[0134] The answer is judged based on the facial key point data and audio data to obtain the speech recognition result of the cognitive test;

[0135] State analysis is performed based on facial key point data, intermediate output of answer judgment, and vital sign data to obtain the state analysis results of cognitive detection.

[0136] Assuming that the above is the first possible implementation, in the second possible implementation provided on the basis of the first possible implementation, answer discrimination is performed based on the facial key point data and the audio data to obtain the speech discrimination result of the cognitive test, including:

[0137] Obtaining intermediate outputs for answering questions based on facial key point data and audio data. The intermediate outputs include: speech features, phonemes, and text.

[0138] Determine the information to be discriminated among phonemes, text and speech features according to the target question type of cognitive testing questions;

[0139] The discrimination method corresponding to the target question type is used to perform answer discrimination on the discrimination information to obtain the speech discrimination result.

[0140] In a third possible implementation provided as a basis for the second possible implementation, obtaining an intermediate output of answer discrimination based on facial key point data and audio data includes:

[0141] Extracting acoustic features of audio data;

[0142] Encode the acoustic features using a preset acoustic encoder, and obtain speech features based on the encoding results;

[0143] Perform environmental detection based on voice features, acoustic features, and facial key point data to determine the environmental detection results;

[0144] When the environmental detection results meet the preset environmental conditions, the encoding results are processed by a preset acoustic language model to obtain phonemes and text.

[0145] In a fourth possible implementation provided as a basis for the first possible implementation, a state analysis is performed based on facial key point data, the intermediate output of the answer discrimination, and the vital sign data to obtain a state analysis result of the cognitive detection, including:

[0146] Based on the intermediate output, specific features are extracted;

[0147] Extract common features based on vital sign data and facial key point data;

[0148] Perform feature fusion based on specific features and general features to obtain fusion features;

[0149] The analysis is performed based on the fusion features to obtain the status analysis results.

[0150] In a fifth possible implementation provided as a basis for the fourth possible implementation, the specific features include: a first specific feature and a second specific feature, wherein the first specific feature is a streaming specific feature and the second specific feature is a single-round specific feature; the intermediate output includes: speech discrete error, speech features, and text; and the specific features extracted based on the intermediate output include:

[0151] Extracting a first specific feature based on the speech discrete error and the speech feature;

[0152] According to the text, a second specific feature is extracted.

[0153] In a sixth possible implementation provided based on the fifth possible implementation, the fused feature includes: a first overall feature and a second overall feature, wherein the first overall feature is a streaming overall feature, and the second overall feature is a single-round overall feature; and feature fusion is performed based on the specific feature and the general feature to obtain the fused feature, including:

[0154] Perform feature fusion based on the first specific feature and the general feature to obtain a first overall feature;

[0155] Perform statistics on the first overall feature to obtain single-round statistical information;

[0156] Feature fusion is performed based on the single-round statistical information and the second specific feature to obtain the second overall feature.

[0157] In a seventh possible implementation provided as a basis for the first possible implementation, the facial key point data includes facial key point coordinate data and key point distortion degree data, and the facial key point data contained in the video data is extracted, including:

[0158] Perform facial key point recognition on video data to obtain facial key point coordinate data;

[0159] Perform head contour recognition on the video data to obtain the user's head contour data;

[0160] The facial key point coordinate data and the user's head contour data are compared with the preset facial standard data to obtain key point distortion degree data, wherein the facial standard data includes: facial key point standard coordinate data and head contour standard data.

[0161] It should be understood that in the embodiments of the present application, the processor 1002 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0162] The memory 1001 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1002. A portion or all of the memory 1001 may also include a non-volatile random access memory. For example, the memory 1001 may also store device type information.

[0163] As can be seen from the above, the embodiment of the present application realizes the automatic discrimination and interaction state analysis of speech-based cognitive test questions. Specifically, in terms of automatic discrimination, the embodiment of the present application classifies and designs discrimination methods that are closer to human logical patterns for different types of cognitive test questions, which can further reduce the difference between automatic discrimination and human discrimination results; in terms of interaction state analysis, the embodiment of the present application integrates vital sign data to form a state analysis of the interaction process, and extracts reusable general feature coefficients and feature construction methods. These coefficients and methods can analyze the interaction state based on general features. In addition, the embodiment of the present application also optimizes ASR and NLP, wherein the matching process of NLP is designed to be streamed and integrated with ASR to form a full-process voice module, which can realize the simultaneous use of voice features and text, providing a basis for a discrimination method that is more in line with human logic and reducing the waiting time from completing the answer to the end of the discrimination; ASR completes the miniaturization of the model through distillation and quantization, and NLP uses fewer deep learning models, thereby reducing its overall memory usage and the amount of calculation, thereby completing the offline operation of the entire system and eliminating the possibility that the system is affected by network instability and cloud service concurrency limitations and affecting the discrimination quality. Furthermore, the embodiment of the present application also adds the function of environmental detection to ensure the controllability of the voice link environment, and reduces the confusion of the language model and invalid computing power caused by multiple people speaking at the same time by interrupting decoding. Finally, the embodiment of the present application forms a complete voice evaluation interaction system by integrating the above advantages. Compared with the existing technology, it can improve the accuracy of voice-based cognitive detection interaction, and can analyze the user's interaction status as a basis for interaction strategy, thereby helping to improve the experience of the cognitive detection interaction process.

[0164] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0165] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0166] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of external device software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0167] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the above modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0168] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the associated hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program, when executed by the processor, can implement the steps of the above-mentioned various method embodiments. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The above-mentioned computer-readable storage medium may include: any entity or device that can carry the above-mentioned computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer-readable memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media does not include electrical carrier signals and telecommunication signals.

[0170] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A cognitive detection method, characterized in that: include: Based on the user's answers to the cognitive test questions, the data to be evaluated is obtained, wherein the data to be evaluated includes: the user's vital sign data, the video data containing the user's facial information, and the audio data containing the audio of the user's answering questions; Extracting facial key point data contained in the video data; Performing answer discrimination based on the facial key point data and the audio data to obtain a speech discrimination result of cognitive detection; A state analysis is performed based on the facial key point data, the intermediate output of the answer judgment and the vital sign data to obtain a state analysis result of the cognitive detection.

2. The cognitive detection method according to claim 1, characterized in that: The answer discrimination is performed based on the facial key point data and the audio data to obtain the speech discrimination result of the cognitive detection, including: Obtaining an intermediate output of answer determination based on the facial key point data and the audio data, the intermediate output including: speech features, phonemes and text; Determining information to be identified from the phonemes, the text, and the speech features according to the target question type of the cognitive detection question; The identification method corresponding to the target question type is used to perform answer identification on the information to be identified to obtain the speech identification result.

3. The cognitive detection method according to claim 2, characterized in that: The intermediate output of answering the question based on the facial key point data and the audio data includes: Extracting acoustic features of the audio data; Encoding the acoustic feature by a preset acoustic encoder, and obtaining the speech feature based on the encoding result; Performing environmental detection according to the voice features, the acoustic features and the facial key point data, and determining an environmental detection result; When the environmental detection result meets the preset environmental conditions, the encoding result is processed by a preset acoustic language model to obtain the phonemes and the text.

4. The cognitive detection method according to claim 1, characterized in that: The state analysis is performed based on the facial key point data, the intermediate output of the answer determination and the vital sign data to obtain the state analysis result of the cognitive detection, including: Extracting specific features according to the intermediate output; Extracting common features according to the vital sign data and the facial key point data; Perform feature fusion based on the specific feature and the general feature to obtain a fused feature; An analysis is performed based on the fusion feature to obtain the status analysis result.

5. The cognitive detection method according to claim 4, characterized in that: The specific features include: a first specific feature and a second specific feature, wherein the first specific feature is a streaming specific feature, and the second specific feature is a single-round specific feature; the intermediate output includes: speech discrete error, speech features and text; the specific features extracted according to the intermediate output include: Extracting a first specific feature according to the speech discrete error and the speech feature; A second specific feature is extracted based on the text.

6. The cognitive detection method according to claim 5, characterized in that: The fused feature includes: a first overall feature and a second overall feature, wherein the first overall feature is a streaming overall feature, and the second overall feature is a single-round overall feature; the feature fusion based on the specific feature and the general feature to obtain the fused feature includes: Perform feature fusion based on the first specific feature and the general feature to obtain the first overall feature; Performing statistics on the first overall features to obtain single-round statistical information; Feature fusion is performed based on the single-round statistical information and the second specific feature to obtain the second overall feature.

7. The cognitive detection method according to claim 1, characterized in that: The facial key point data includes: facial key point coordinate data and key point distortion degree data; the facial key point data extracted from the video data includes: Performing facial key point recognition on the video data to obtain the facial key point coordinate data; Performing head contour recognition on the video data to obtain user head contour data; The facial key point coordinate data and the user head contour data are compared with preset facial standard data to obtain the key point distortion degree data, wherein the facial standard data includes: facial key point standard coordinate data and head contour standard data.

8. A cognitive detection device, characterized in that: include: An acquisition module, used to acquire data to be evaluated based on the user's answers to cognitive test questions, wherein the data to be evaluated includes: the user's vital sign data, video data containing the user's facial information, and audio data containing the user's answer audio; An extraction module, used to extract the facial key point data contained in the video data; A discrimination module, used to discriminate the answer according to the facial key point data and the audio data, and obtain a speech discrimination result of cognitive detection; The analysis module is used to perform state analysis based on the facial key point data, the intermediate output of the answer judgment and the vital sign data to obtain the state analysis result of the cognitive detection.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images