Identification method and related device

Abstract

The present invention provides an identification method and related devices. The above method includes: collecting the user's brain reaction data; processing the brain reaction data to obtain brain reaction parameter values; wherein the brain reaction data at least includes eye reaction data; the brain reaction parameter values ​​at least include eye reaction parameter values; the eye reaction parameter values ​​include parameter values ​​corresponding to each eye reaction parameter; the brain reaction parameter values ​​are input into a machine learning model, and the machine learning model outputs the recognition result of emotion cognition. In an embodiment of the present invention, after the brain reaction data (mainly eye reaction data) is collected, data processing is performed on it to obtain brain reaction parameter values ​​including eye reaction parameter values, and the brain reaction parameter values ​​are input into the machine learning model, and the artificial intelligence algorithm is used to analyze based on the brain reaction parameter values ​​to obtain the recognition result of emotion cognition, thereby realizing the intelligent recognition of human emotion cognition.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to an identification method and a related device. Background Art

[0002] We have been working hard to explore intelligent and scientific methods to detect / evaluate the emotional cognitive state of the human brain.

[0003] For the immediate interpretation and long-term monitoring of current brain cognition or brain disease status, the traditional methods are brain wave detection and nuclear magnetic resonance scanning. However, brain wave detection mainly detects signals from the cerebral cortex, and many of the detected signals cannot be accurately interpreted. As for nuclear magnetic resonance scanning, it does not provide much effective information on the emotional cognition of the human brain. Summary of the invention

[0004] In view of this, embodiments of the present invention provide identification methods and related devices to intelligently and scientifically identify human emotional cognition.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] An identification method, applied to a user side, comprising:

[0007] Collecting the user's brain response data;

[0008] Processing the brain reaction data to obtain brain reaction parameter values; wherein the brain reaction data at least includes eye reaction data; the brain reaction parameter values ​​at least include eye reaction parameter values; the eye reaction parameter values ​​include parameter values ​​corresponding to each eye reaction parameter;

[0009] The brain response parameter value is input into a machine learning model, and the machine learning model outputs an emotion recognition recognition result.

[0010] Optionally, the recognition result of emotion cognition output by the machine learning model includes: the machine learning model identifies the emotion cognition type and the corresponding score based on the eye reaction parameter value and the eye reaction parameter threshold; the machine learning model determines the state information corresponding to the emotion cognition type based on the state threshold corresponding to the emotion cognition type and the score; the machine learning model outputs the identified emotion cognition type and the corresponding state information; the recognition result includes: the emotion cognition type and the corresponding state information.

[0011] Optionally, the method further includes: using the eye reaction data of the user to perform personalized correction on the eye reaction parameter threshold.

[0012] Optionally, the method further includes: receiving correction data input by the user; and using the correction data to correct at least one of the emotion recognition type, state information, and state threshold.

[0013] Optionally, the emotion recognition type includes at least one of an emotion type and a cognitive type; wherein: the emotion type includes: a mood subtype and a fatigue subtype; the cognitive type includes an attention subtype and a stress subtype; the machine learning model is trained using labeled training samples; wherein the training samples include brain response parameters from healthy individuals or patient individuals; the labels include mood state information labels, fatigue state information labels, attention state information labels and stress state information labels.

[0014] Optionally, after the training is completed, the method also includes: using labeled test samples to test the recognition accuracy and recognition speed of the machine learning model; the test samples; wherein the test samples include brain reaction parameters from healthy individuals or patient individuals; the labels include: mood state information labels, fatigue state information labels, attention state information labels and stress state information labels; if the machine learning model does not meet the preset conditions, perform one or more of the following operations and re-train: reselect eye reaction parameters; adjust the weight value of the machine learning model; adjust the state threshold; adjust at least one of the type and content of the label; wherein the preset conditions include: the recognition accuracy of the machine learning model is not lower than the accuracy threshold and the recognition speed is not lower than the speed threshold.

[0015] Optionally, the method also includes: uploading the recognition results of emotion cognition and the corresponding brain reaction parameter values ​​to the cloud or backend; the uploaded brain reaction parameter values ​​will be used as training samples or test samples during the training process, and the recognition results of emotion cognition are used to mark the corresponding training samples or test samples; or, uploading the recognition results of emotion cognition and the corresponding brain reaction data to the cloud or backend; the uploaded brain reaction data is used to generate training samples or test samples during the training process, and the recognition results of emotion cognition are used to mark the corresponding training samples or test samples; after the cloud or backend optimizes the machine learning model, the optimized machine learning model will be synchronized to the user side.

[0016] Optionally, the eye reaction parameters include one or more of the following: contrast and brightness of the eyes; speed, direction and frequency of eye movement; amplitude and speed of pupil reaction; interpupillary distance; speed, amplitude and frequency of blinking; and muscle contraction of the eyes, including the periorbital area and eyebrows.

[0017] Optionally, the eye reaction data includes: eye video or eye image; the brain reaction data also includes at least one of a forebrain cortex signal and a skin electrical signal; the brain reaction parameter value also includes: at least one of a forebrain cortex parameter value and a skin electrical parameter value; wherein the forebrain cortex parameter value includes parameter values ​​corresponding to each forebrain cortex parameter, and the skin electrical parameter value includes parameter values ​​corresponding to each skin electrical parameter.

[0018] An identification system includes a collection device and a central control system; the central control system includes at least an identification device; wherein:

[0019] The collection device is used to: collect the user's brain response data;

[0020] The identification device is used to: process the brain reaction data to obtain brain reaction parameter values; wherein the brain reaction data at least includes eye reaction data; the brain reaction parameter values ​​at least include eye reaction parameter values; the eye reaction parameter values ​​include parameter values ​​corresponding to each eye reaction parameter;

[0021] The brain response parameter value is input into a machine learning model, and the machine learning model outputs an emotion recognition recognition result.

[0022] Optionally, the central control system also includes a cloud or backend; the recognition device is also used to: upload the recognition results of emotion cognition and the corresponding brain reaction parameter values ​​to the cloud or backend; the uploaded brain reaction parameter values ​​will be used as training samples or test samples during the training process, and the recognition results of emotion cognition are used to mark the corresponding training samples or test samples; or, upload the recognition results of emotion cognition and the corresponding brain reaction data to the cloud or backend; the uploaded brain reaction data is used to generate training samples or test samples during the training process, and the recognition results of emotion cognition are used to mark the corresponding training samples or test samples; the cloud or backend is used to: train the machine learning model using labeled training samples and test samples; the optimized machine learning model will be synchronized to the recognition device.

[0023] Optionally, the acquisition device includes a camera device on a smart terminal, and the identification device is specifically the smart terminal; or, the acquisition device includes: a wearable device with an eye camera function; and the identification device is a smart terminal.

[0024] Optionally, the wearable device includes: a camera device for collecting eye response data; a forebrain cortex signal sensor for collecting forebrain cortex signals; and a skin electrical signal sensor for collecting skin electrical signals.

[0025] Optionally, the wearable smart device is smart glasses; the camera device is a miniature electronic camera; wherein: the miniature electronic camera is arranged at the junction of the lens and the handle of the smart glasses; the skin electrical signal sensor is arranged on the inner side of the handle and the part in contact with the ear; the forebrain cortex signal sensor is arranged in the middle of the handle.

[0026] Optionally, the rear inner side of the temple of the smart glasses is a flexible bio-electrode, and the two nose pads of the smart glasses are flexible bio-electrodes.

[0027] An intelligent terminal, comprising:

[0028] An acquisition unit, used for acquiring the user's brain response data;

[0029] Identification unit for:

[0030] Processing the brain reaction data to obtain brain reaction parameter values; wherein the brain reaction data at least includes eye reaction data; the brain reaction parameter values ​​at least include eye reaction parameter values; the eye reaction parameter values ​​include parameter values ​​corresponding to each eye reaction parameter;

[0031] The brain response parameter value is input into a machine learning model, and the machine learning model outputs an emotion recognition recognition result.

[0032] A wearable smart device, comprising:

[0033] A camera device for collecting eye response data;

[0034] A forebrain cortex signal sensor for collecting forebrain cortex signals, and a skin electrical signal sensor for collecting skin electrical signals.

[0035] Optionally, it also includes: a data output device.

[0036] Optionally, also includes: health index monitor.

[0037] Optionally, the wearable smart device is smart glasses; the camera device is a miniature electronic camera; wherein: the miniature electronic camera is arranged at the junction of the lens and the handle of the smart glasses; the skin electrical signal sensor is arranged on the inner side of the handle and the part in contact with the ear; the forebrain cortex signal sensor is arranged in the middle of the handle.

[0038] Optionally, the rear inner side of the temple of the smart glasses is a flexible bio-electrode, and the two nose pads of the smart glasses are flexible bio-electrodes.

[0039] Optionally, it also includes: a mechanical sleep switch or a timing switch; the mechanical sleep switch or the timing switch is arranged at the connection between the temple and the frame of the smart glasses.

[0040] Optionally, it also includes: a touch screen; the touch screen is arranged on the outside of the mirror handle.

[0041] Optionally, also includes: rechargeable batteries.

[0042] Optionally, the data output device includes a Bluetooth chip, and the Bluetooth chip is built into any mirror handle.

[0043] Optionally, the data output device includes a WiFi chip.

[0044] A storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor to execute the steps in the above-mentioned identification method.

[0045] A chip system includes a processor for supporting the identification device or the smart terminal to execute the above-mentioned identification method.

[0046] In an embodiment of the present invention, after the brain reaction data (mainly eye reaction data) is collected, data processing is performed on it to obtain brain reaction parameter values ​​including eye reaction parameter values, and the brain reaction parameter values ​​are input into a machine learning model. The artificial intelligence algorithm performs analysis based on the brain reaction parameter values ​​to obtain the recognition result of emotion cognition, thereby realizing intelligent recognition of human emotion cognition.

[0047] At the same time, it should be noted that about 80% of the information obtained by the human brain comes from the visual system. Therefore, the emotional and cognitive status of the brain can be judged by the state and level of the visual system processing visual signal input. In particular, judging people's psychological and mental states through facial expressions mainly based on eyes and mouths is something that humans do automatically when communicating. Therefore, it is scientifically feasible to identify emotional cognitive states through artificial intelligence algorithms based on brain reaction parameter values ​​including eye reaction parameter values. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1a An example diagram of the structure of an identification system provided by an embodiment of the present invention;

[0049] Figure 1b An exemplary structural diagram of a smart terminal provided by an embodiment of the present invention;

[0050] Figure 2 , 3 , 6, 7, and 9 are exemplary flow charts of the identification method provided in the embodiments of the present invention;

[0051] Figure 4A schematic diagram of data uploading provided by an embodiment of the present invention;

[0052] Figure 5 An exemplary flow chart of the training process provided by an embodiment of the present invention;

[0053] Figure 8 An exemplary structural diagram of the smart glasses provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The present invention provides an identification method and related devices (such as an identification system, a smart terminal, a storage medium, a wearable smart device, a chip system, etc.) to intelligently and scientifically identify human emotional cognition in a variety of scenarios.

[0055] See also Figure 1a The above-mentioned identification system may include: a collection device 101 and a central control system, wherein the central control system at least includes an identification device 102.

[0056] The core idea of ​​the recognition method implemented by the above-mentioned recognition system is: after collecting brain reaction data (mainly eye reaction data), it will be processed to obtain brain reaction parameter values ​​including eye reaction parameter values, and the brain reaction parameter values ​​will be input into the machine learning model. The artificial intelligence algorithm will analyze the brain reaction parameter values ​​to obtain the recognition results of emotional cognition.

[0057] In one example, the acquisition device 101 may include a high-pixel camera of a smart terminal, and the identification device 102 may include the smart terminal.

[0058] Figure 1b An exemplary structure of the above-mentioned intelligent terminal is shown, including:

[0059] Acquisition unit 1021: used to acquire the user's brain response data;

[0060] Identification unit 1022: used to process the above-mentioned brain response data to obtain brain response parameter values, and input the brain response parameter values ​​into the machine learning model, and the machine learning model outputs the recognition results of emotion cognition.

[0061] Specifically, the eye reaction data may be collected by a high-pixel camera of a smart terminal (that is, the acquisition unit 1021 may specifically include a high-pixel camera). Smart terminals include but are not limited to smart phones, iPads, laptops, and the like.

[0062] In addition to the eye reaction data, the brain reaction data may also include at least one of the forebrain cortex signal and the skin electrical signal. Among them, the forebrain cortex signal can be collected by the forebrain cortex signal sensor, and the skin electrical signal can be collected by the skin electrical signal sensor. The forebrain cortex signal sensor and the skin electrical signal sensor can be installed on a wearable device (such as smart glasses), and the wearable device transmits it to the smart terminal wirelessly. In this scenario, the acquisition unit 1021 may also include a wireless receiving device to obtain data transmitted by the wearable device.

[0063] The intelligent terminal may include a processor and a storage medium in terms of hardware. The storage medium stores a variety of instructions suitable for loading by the processor. After the processor loads the instructions in the storage medium, the function of the recognition unit 1022 can be realized: data processing is performed on the above-mentioned brain reaction data to obtain brain reaction parameter values, and the brain reaction parameter values ​​are input into the machine learning model, and the machine learning model outputs the recognition result of emotion cognition.

[0064] From the user's perspective, the function of the identification unit 1022 can be implemented by application software (such as APP) installed in the smart terminal.

[0065] In another example, the acquisition device 101 may include a miniature camera on the wearable device, and the identification device 102 may include a smart terminal. In addition, at least one of a forebrain cortex signal sensor and a skin electrical signal sensor may be provided on the wearable device.

[0066] Specifically, the above-mentioned eye reaction data can be collected by a micro camera on the wearable device, and the forebrain cortex signal and skin electrical signal can also be collected by the wearable device. The wearable device transmits the collected data to the smart terminal via wireless methods such as Bluetooth and WiFi, and the smart terminal executes the subsequent steps.

[0067] In this scenario, the acquisition unit 1021 of the smart terminal includes a wireless receiving device to acquire data transmitted by the wearable device. The function of the aforementioned identification unit 1022 can be implemented by the processor of the smart terminal loading instructions in the storage medium.

[0068] From the user's perspective, the function of the identification unit 1022 can be implemented by application software (such as APP) installed in the smart terminal.

[0069] In other embodiments of the present invention, the above-mentioned acquisition device may also include various health index monitors, so that the range of brain reaction data can be expanded to cover various health index data, so that more comprehensive data can be used to obtain more accurate recognition results.

[0070] Since the acquisition device and recognition device included in the recognition system are easy and quick to operate and convenient to carry or wear, they can be used for long-term monitoring or immediate evaluation of the brain's emotional cognition.

[0071] For example, the above recognition system can be used to evaluate, monitor, and even predict some brain cognition-related diseases, and thus can be used for long-term monitoring and care of chronic mental illnesses. Of course, it can also be used to monitor the deterioration or onset of mental illnesses in real time. The recognition system can even output intervention adjustment measures and suggestions, such as but not limited to outputting breathing adjustment suggestions, music therapy measures / suggestions, light therapy measures / suggestions, or cognitive behavioral therapy measures / suggestions, etc.

[0072] As for real-time monitoring, in addition to the real-time monitoring of the deterioration or onset of mental illness mentioned above, the above-mentioned recognition system can also instantly evaluate and monitor the user's attention and fatigue, and output intervention adjustment measures and suggestions to remind the user to use the brain in a healthy and scientific way, thereby improving the user's work or study efficiency.

[0073] In addition, the above-mentioned recognition system can also be used to monitor the user's psychological reaction to watching commercial advertisements (i.e., the detection of product advertising effects), whether the user is driving fatigued, and conduct psychological lie detection, etc.

[0074] The user can also input the current occasion (scene) or the occasion (scene) that he is about to enter, and the recognition system can give targeted suggestions based on the occasion. For example, if the user inputs the interview occasion, if it is recognized that the user's current attention is not focused enough, the system can remind the user to focus.

[0075] Therefore, the identification method and related devices provided by the present invention have quite broad application prospects.

[0076] The embodiments of the present invention will be further described in detail below based on the common aspects of the present invention described above.

[0077] Figure 2 An exemplary process of the above identification method is shown, including:

[0078] S1: The collection device collects the user's brain response data.

[0079] In one example, the brain response data at least includes eye response data, and the eye response data may further include eye video, or picture data extracted from the eye video (which may be referred to as eye image).

[0080] In addition to the eye response data, in another example, the brain response data may also include at least one of a forebrain cortex signal and a skin electrical signal.

[0081] Among them, the forebrain cortex signal can specifically be an EEG signal obtained through EEG (Electroencephalogram, brain wave) technology; and the skin electrical signal can specifically be a PPG signal obtained through PPG (Photoplethymography, photoplethysmography) technology.

[0082] The forebrain cortex signal carries forebrain information, the eye response data carries midbrain information, and the skin electrical signal carries hindbrain information. If the comprehensive brain information from the midbrain, forebrain and hindbrain is used to participate in the subsequent recognition steps, it will be beneficial to the comprehensive, accurate and instant interpretation of the emotional and cognitive state of the human brain.

[0083] S1 may be executed by the aforementioned acquisition unit 1021 or the acquisition device 101 .

[0084] S2: The recognition device processes the above-mentioned brain response data to obtain brain response parameter values.

[0085] In one example, the brain response data at least includes eye response data, and correspondingly, the brain response parameter value at least includes eye response parameter value.

[0086] The eye reaction parameter value includes parameter values ​​corresponding to each eye reaction parameter. The eye reaction parameters exemplarily include but are not limited to one or more of the following:

[0087] (1) Eye contrast and brightness;

[0088] The contrast of the eyes can specifically refer to the contrast between the white of the eye (sclera part) and the eyeball (iris part).

[0089] The brightness of the eyes is affected by the blood status of the capillaries in the eyes. For example, if the capillaries are congested, the brightness will be relatively darker than when they are not congested.

[0090] (2) speed, direction, and frequency of eye movements;

[0091] Specifically, the eye movement frequency may include the frequency of the eye moving up and down, left and right.

[0092] (3) the amplitude and speed of pupillary response;

[0093] The pupil reactions here include: pupil contraction or dilation.

[0094] (4) interpupillary distance;

[0095] (5) blink speed, amplitude, and frequency;

[0096] (6) Contraction of eye muscles (including eyebrows).

[0097] For example, the eye muscles used when smiling and frowning are obviously different, so the eye muscle contraction can be used to analyze human emotions and cognition. More specifically, the eye muscle contraction can be represented by the dot matrix change of computer vision.

[0098] In another example, if the brain response data also includes forebrain cortex signals and skin electrical signals, then correspondingly, the brain response parameter value also includes: at least one of the forebrain cortex parameter value and the skin electrical parameter value.

[0099] The forebrain cortex parameter value includes parameter values ​​corresponding to various forebrain cortex parameters.

[0100] Similarly, the skin electrical parameter value includes parameter values ​​corresponding to each skin electrical parameter, and the skin electrical parameter may further include at least one of heart rate, blood pressure, temperature and respiratory rate.

[0101] S3: The recognition device uses a machine learning model (also called an AI model) to analyze the above-mentioned brain reaction parameter values ​​to obtain the recognition results of emotion cognition.

[0102] In one example, steps S2 - S3 may be performed by the aforementioned identification unit 1022 .

[0103] The above-mentioned machine learning model may include a deep learning model. Deep machine learning methods are also divided into supervised learning and unsupervised learning. The learning models established under different learning frameworks are very different. For example, convolutional neural networks (CNNs) are a machine learning model under deep supervised learning, while deep belief nets (DBNs) are a machine learning model under unsupervised learning.

[0104] Specifically, the brain response parameter values ​​can be input into a machine learning model, and the machine learning model outputs the recognition results of emotion cognition.

[0105] In addition to the brain response parameter values, other parameters such as binocular rivalry can also be input. The binocular rivalry parameters can be collected by other auxiliary devices.

[0106] The recognition result of emotion recognition may further include the recognized emotion recognition type and corresponding state information.

[0107] In one example, the emotion-cognition type may include at least one of an emotion type and a cognitive type.

[0108] Wherein: the emotion type includes at least: "mood" and "fatigue" subtypes.

[0109] Furthermore, exemplary “mood” subtypes include: happy, sad, fearful, excited, depressed, anxious, or sorrowful mood subtypes (or sub-moods), and each mood subtype may be represented by numbering, binary coding, or the like.

[0110] Exemplarily, the status information may include a text description, or may include a score; alternatively, the status information may include both a score and a text description, and the emotion recognition type ultimately displayed to the user may also be in text form.

[0111] Taking the recognition result of “poor attention” as an example, it includes the text form of the recognized emotion recognition type - “attention”, and the text description of the state information - “poor”.

[0112] It should be noted that the state information in the above “poor attention” is expressed explicitly. In addition, the state information can also be expressed implicitly or indirectly, such as “the eyes are full of anxiety and sadness”, which includes the identified emotional cognitive types - “anxiety” and “sadness”, but its state information is expressed as a whole through “the eyes are full of anxiety and sadness”.

[0113] It can be seen that in an embodiment of the present invention, after the brain reaction data (mainly eye reaction data) is collected, data processing will be performed on it to obtain brain reaction parameter values ​​including eye reaction parameter values, and the brain reaction parameter values ​​will be input into a machine learning model. The artificial intelligence algorithm will perform analysis based on the brain reaction parameter values ​​to obtain the recognition result of emotion cognition, thereby realizing intelligent recognition of human emotion cognition.

[0114] At the same time, it should be noted that about 80% of the information obtained by the human brain comes from the visual system. Therefore, the emotional and cognitive status of the brain can be judged by the state and level of the visual system processing visual signal input. In particular, judging people's psychological and mental states through facial expressions mainly based on eyes and mouths is something that humans do automatically when communicating. Therefore, it is scientifically feasible to identify emotional cognition through artificial intelligence algorithms based on brain reaction parameter values ​​including eye reaction parameter values.

[0115] The following uses the eye response parameter value as an example to introduce the specific process of the machine learning model outputting the recognition result of emotion cognition, which may include the following steps:

[0116] Step a: The machine learning model identifies the emotion recognition type and the corresponding score based on the eye response parameter value and the eye response parameter threshold.

[0117] The score can be understood as a rating or grade value. For example, a machine learning model can identify: excitement value 5 points; fear value 4 points, etc.

[0118] The cognitive type includes at least the "attention" and "stress" subtypes.

[0119] Step b: The machine learning model determines the state information corresponding to the emotion recognition type according to the state threshold and score corresponding to the emotion recognition type.

[0120] It is mentioned above that the state information may include a text description, or may include a score; or the state information may include a score and a text description at the same time. The text description may be determined according to the state threshold and the score calculated in step S31.

[0121] Still taking "attention" as an example, for example, assuming that the attention value obtained in step S31 is x, and assuming that the score is between the state threshold a and the state threshold b, the corresponding text description is "poor". If a≤x≤b, it can be determined that the text description of the state information corresponding to the "attention" subtype is "poor".

[0122] In other embodiments of the present invention, the state threshold may include a limit threshold to distinguish between a normal state and a pathological state (normal state and pathological state belong to state information). For example, if the calculated score is not less than the limit threshold, it can be determined to be in a normal state, and if it is lower than the limit threshold, it can be determined to be in a pathological state.

[0123] In addition, the state threshold may also include a pathological degree threshold to further determine the pathological degree (the pathological degree also belongs to the state information).

[0124] Similarly, the state threshold may also include a normal state degree threshold to divide the normal state into multiple degrees. For example, assuming that the initial threshold corresponding to "happy mood" is 5-7 points, a score of 3-4 points will be determined as depressed mood. If the recognition device recognizes the emotion type of mood state (e.g. "happy") for user A, and the corresponding score is 4, the output state information is "depressed mood".

[0125] The determination method of the status information corresponding to other subtypes is similar and will not be described in detail here.

[0126] Step c: The machine learning model outputs the identified emotion recognition type and the corresponding state information.

[0127] The output mode of the emotion recognition type and the corresponding status information can be visual output or voice broadcast output.

[0128] In addition, the machine learning model can also output descriptions that reflect the state of the eyes. For example, it can output descriptions such as "eyes are dull and gray" and "brows are furrowed" to reflect the state of the eyes.

[0129] Due to individual differences, the recognition results initially output by the recognition system may not be consistent with the actual situation of the user. Figure 3 After step S3, the above identification method may further include the following steps:

[0130] S4: the recognition device receives the correction data input by the user;

[0131] After or while displaying the recognition result, the recognition device may provide a human-computer interaction interface for correction, so that the user can manually input correction data.

[0132] In one example, the correction data may be used to correct at least one of the emotion recognition type, state information, and state threshold.

[0133] Taking the correction of emotion recognition type as an example, if the user sheds tears, the machine learning model may recognize the emotion type as "sadness". However, if the user is "crying with joy", the user can change the recognized emotion type to "happy", "delighted", etc.

[0134] Specifically, the user can manually input text into the human-computer interaction interface, and the system will convert the text into a corresponding type.

[0135] Of course, considering that different users may describe emotions differently, in order to facilitate unified processing, multiple emotion recognition type options can be provided in the human-computer interaction interface, and the user can select one or several of them.

[0136] Taking the correction of "status information" as an example, suppose the system shows the user "good mood", but the user may feel that his current mood is average and needs to be corrected. In this case, the user can manually enter the word "average" into the human-computer interaction interface.

[0137] Of course, considering that different users may have different descriptions of the same state, in order to facilitate unified processing, for a certain emotion recognition type, the human-computer interaction interface can provide multiple state information for users to choose from, and the user can input by selecting one of them.

[0138] Taking score correction as an example, if the system displays the score to the user, the user may have the need to correct it. In this case, the user can manually enter a specific score into the human-computer interaction interface, or the human-computer interaction interface can provide multiple scores for the user to choose from, and the user can select one of them to enter the score.

[0139] S5: The recognition device uses the correction data to correct at least one of the emotion recognition type, state information and state threshold.

[0140] In one example, S4 - S5 may be performed by the aforementioned identification unit 1022 .

[0141] As mentioned above, the status information may further include at least one of a text description and a score.

[0142] Taking text description as an example, since the text description is determined according to the state threshold, when correcting the text description in the state information, what is actually corrected is the correspondence between the text description and the state threshold, or it can be understood that what is ultimately corrected is the state threshold.

[0143] For example, assuming that the initial threshold corresponding to "happy mood" is 5-7 points, and a score of 3-4 points will be judged as depressed mood. If the recognition device recognizes the emotion type of "happy" for user A, and the corresponding score is 4, the output status information is "depressed mood".

[0144] If user A corrects the text description from "depressed" to "happy", the state threshold corresponding to "happy" can be modified to 4-7 points.

[0145] It should be noted that the extreme threshold and pathological degree threshold in the state threshold are generally not corrected using correction data.

[0146] It can be seen that in this embodiment, the emotion recognition type, state information and state threshold can be corrected according to the correction data input by the user, so that the recognition result is more appropriate and accurate to the individual.

[0147] As mentioned above, the emotion recognition type and the corresponding score can be identified based on the eye response parameter value and the eye response parameter threshold. In practice, the shape and size of the eyes, and the highest and lowest blinking frequencies of different individuals are different.

[0148] Therefore, please see Figure 3 After the above step S3, the above identification method may further include the following steps:

[0149] S6: The recognition device uses the user's eye reaction data to perform personalized correction on the eye reaction parameter threshold.

[0150] In one example, S6 may be executed by the aforementioned identification unit 1022 .

[0151] Specifically, by collecting eye reaction data over a period of time (for example, a few days, a week, etc.), the user's eye habits (such as the highest and lowest blinking frequencies, etc.), pupil size, etc. can be extracted to correct the parameter thresholds in the machine learning model.

[0152] Or in data processing, the eye response parameter values ​​(such as pupil distance, eye height, eye width, iris color, etc.) are resized.

[0153] It can be seen that in this embodiment, the parameter threshold can be corrected according to the user's eye response data, so that the recognition result is more relevant to the individual and more accurate.

[0154] In other embodiments of the present invention, the aforementioned central control system may include a cloud or backend in addition to the identification device.

[0155] In one example, the recognition device may upload the recognition results of emotion recognition and the corresponding brain reaction parameter values ​​to the cloud or backend.

[0156] In another example, the recognition device may upload the recognition results of emotion recognition and the corresponding brain response data to the cloud or backend.

[0157] The recognition device may periodically upload the brain reaction parameter value / brain reaction data. More specifically, the recognition device may directly and automatically periodically upload the brain reaction parameter value / brain reaction data, or periodically and automatically upload the brain reaction parameter value / brain reaction data after the user authorizes it.

[0158] The cloud or backend will integrate massive amounts of data while protecting user privacy, and use the uploaded brain reaction parameter values / brain reaction data to generate training samples or test samples to train the machine learning model and optimize parameters (such as the aforementioned parameter thresholds, state thresholds); and the uploaded emotion recognition results can be used to mark the corresponding training samples or test samples; finally, the optimized machine learning model will be synchronized to the recognition device.

[0159] After being synchronized to the recognition device, the recognition device may re-personalize the eye response parameter thresholds and use the correction data to correct the emotion recognition type, state information, state thresholds, etc.

[0160] The interaction process between the identification device and the cloud or backend can be found in Figure 4 .

[0161] The above-mentioned machine learning model can be obtained by training based on training samples. After the training of the machine learning model is completed, it will also be tested whether the trained machine learning model meets the expected performance requirements (including requirements in terms of recognition accuracy and recognition speed). If not, corresponding adjustments will be made until the expected performance requirements are met.

[0162] It should be noted that the first training can obtain a machine learning model, and subsequent training can optimize the machine learning model.

[0163] The following focuses on the training process of the machine learning model. Figure 5 , the training process of the machine learning model performed by the cloud or backend server may include at least the following steps:

[0164] S501: Obtain samples.

[0165] Any sample may include brain response parameters from healthy individuals or patients, including but not limited to autism, depression, Alzheimer's disease, Huntington's disease, schizophrenia, and post-traumatic stress disorder.

[0166] Of course, in the case where the recognition device uploads brain response data, the brain response data will also be processed to obtain training samples.

[0167] S502: Mark the above samples.

[0168] It should be noted that in the early stage of establishing the data source, the training samples can be manually labeled as prior knowledge for the machine learning model. In the later stage, especially after the machine learning model is officially put into use, it can be automatically labeled according to the recognition results of emotional cognition.

[0169] The so-called labeling may refer to adding one or more labels to the training samples, for example, a mood state information label, a fatigue state information label, an attention state information label, and a stress state information label may be added.

[0170] The contents of the above-mentioned labels include: emotional subtype or cognitive subtype, and corresponding state information.

[0171] In addition, a label may be added to indicate whether the sample is from a healthy individual or a patient individual (more specifically, "0" may represent a healthy individual and "1" may represent a patient individual). For samples from patient individuals, a disease label may be further added, and even a doctor's diagnosis report may be added as a label.

[0172] S503: Use the labeled samples to form a training sample set and a test sample set.

[0173] In one example, any labeled sample can be put into a training sample set or a test sample set. The samples in the training sample set are used to train the machine learning model and can be called training samples, while the samples in the test sample set are used to test the machine learning model and can be called test samples.

[0174] S504: Train the machine learning model using the training sample set.

[0175] Specifically, the training samples in the training sample set may be used as input for training.

[0176] Exemplarily, the above-mentioned machine learning model may be a neural network algorithm model, such as a CNN (Convolutional Neural Network) model.

[0177] S504: Use the test sample set to test the diagnostic performance of the machine learning model.

[0178] Specifically, the test samples in the test sample set are input into the machine learning model, and the diagnostic performance is statistically analyzed based on the output of the machine learning model.

[0179] Among them, the diagnostic performance of the model may include recognition accuracy and recognition speed.

[0180] CNN can be combined with GAN (Generative adversarial networks) for testing, which will not be described in detail here.

[0181] S505: If the machine learning model does not meet the preset conditions, perform one or more of the following operations and retrain (return to S501):

[0182] Reselect the type of eye response parameters;

[0183] Adjust the weights of the machine learning model;

[0184] Adjust status thresholds;

[0185] At least one of the type and content of the label is adjusted.

[0186] In one example, the above-mentioned preset conditions may include: the recognition accuracy of the machine learning model is not lower than the recognition accuracy threshold (95% or 98%, etc.), and the recognition speed is not lower than the speed threshold (for example, 10 seconds), so as to obtain a machine learning model that takes into account both test recognition accuracy and recognition speed.

[0187] The recognition accuracy threshold and the speed threshold can be set according to different needs. For example, the recognition accuracy threshold can be set to 95%, and the speed threshold can be set to process 1000 samples in 10 seconds.

[0188] After the machine learning model is put into use, it will continue to be trained to improve the machine learning model.

[0189] The following describes the identification methods performed using different physical entities.

[0190] See also Figure 6 First, an embodiment of collecting eye reaction data by a high-pixel camera of a smart terminal and processing the data and outputting the recognition result of emotion cognition by an APP software installed in the smart terminal is introduced, which specifically includes the following steps:

[0191] S601: The smart terminal (with a high-pixel camera) collects eye reaction data.

[0192] In this embodiment, the eye reaction data is specifically an eye video, or image data derived therefrom. The user can hold the smart terminal, aim the camera at both eyes, about 30-40 cm away from the eyes, and the user looks at the camera to shoot a video.

[0193] It should be noted that the eye video should be a high-definition video (more than 4 million pixels), and even objects reflected in the pupil of the eye can be seen.

[0194] It was mentioned above that by collecting eye reaction data over a period of time (for example, a few days, a week, etc.), the user's eye habits (such as the highest and lowest blinking frequencies, etc.), pupil size, etc. can be extracted to correct the parameter thresholds in the machine learning model (personalized correction).

[0195] This type of personalized correction is generally performed in the early stages of the use of the recognition system (or within the initial period after the machine learning model is optimized).

[0196] In order to better perform personalized correction, in one example, the camera of the smart terminal can be used to regularly capture eye videos, for example, twice a day, each time capturing about 1 minute of eye videos, to capture eye videos at the best and worst states of each day.

[0197] More specifically, you can take a video of your eyes when they are in the best condition of the day about one hour after getting up in the morning, and take a video of your eyes when they are in the worst condition of the day near the end of work.

[0198] After three or four days, you can basically establish the dynamic changes in your own emotions and cognition on a daily basis.

[0199] In one example, the smart terminal can switch between two working modes: a fixed time recording mode and a non-fixed time recording mode.

[0200] In the fixed time recording mode, the eye video can be captured twice a day as mentioned above; and in the non-fixed time recording mode, the eye video can be captured anytime and anywhere according to the user's operation.

[0201] S602: The voice collection device of the intelligent terminal collects voice data.

[0202] The voice collection device may specifically be a microphone.

[0203] The content of the voice data may include a user's specific description of at least one of “mood”, “fatigue”, “attention” and “pressure”.

[0204] For example, the user may say "I am so happy", "I feel so stressed", "I am so tired", "My brain has turned into a mess", etc.

[0205] Alternatively, the content of the voice data may also be a self-rating of the emotion type or cognitive type. For example, the user may input "stress 7 points" by voice.

[0206] S603: The APP of the smart terminal recognizes the voice data and obtains a voice recognition result.

[0207] It should be noted that the speech recognition results can be used to generate at least one of the mood state information labels, fatigue state information labels, attention state information labels and stress state information labels used in the training process. In addition, by comparing the horizontal data of healthy people and patients such as patients with depression, the intelligent classification function of the machine learning model can be trained.

[0208] It should be noted that S602 and S603 can be executed at the initial stage of use of the recognition system (or within the initial period after the machine learning model is optimized), and do not need to be executed in every recognition process.

[0209] After the initial use, the algorithm optimization based on artificial intelligence can calculate the user's mental cognitive state at any time, and only 30 seconds of eye video is provided in real time.

[0210] S604: The APP of the smart terminal processes the eye reaction data to obtain eye reaction parameter values.

[0211] Please refer to the above introduction for details and will not be elaborated here.

[0212] In specific implementation, data processing can be integrated with the face recognition function of the smart terminal to identify the eye reaction parameter values.

[0213] In addition, the angle sensor and distance sensor of the smart terminal can be used to determine the angle and distance between the camera and the eye, and then the actual size of the eye can be calculated based on the determined angle and distance, so as to restore the proportion of the collected images at different distances and angles, or transform the size of the eye reaction parameter value.

[0214] S605: The APP uses a machine learning model to analyze the above eye reaction parameter values, obtains and displays the recognition results of emotional cognition.

[0215] The output method can be visual output or voice broadcast output.

[0216] Regarding how to obtain the emotion recognition results, please refer to the previous description of this article and will not be elaborated here.

[0217] S606: The APP prompts whether to calibrate manually. If the user selects "yes", the process proceeds to step S607, otherwise, the process proceeds to step S609.

[0218] In one example, after or while displaying the recognition result, the recognition device may display a human-computer interaction interface to prompt whether manual correction is required.

[0219] S607: Receive calibration data input by the user.

[0220] The correction data may include at least one of emotion recognition type status information, and the status information may further include at least one of a text description and a score.

[0221] S607 is similar to the aforementioned S4 and will not be described in detail here.

[0222] S608: Use the correction data to correct at least one of the emotion recognition type, state information and state threshold, to S609.

[0223] It should be noted that if at least one of the emotion recognition type, state information and state threshold is corrected using the correction data, the recognition result obtained in step S605 will also be corrected accordingly.

[0224] S608 is similar to the aforementioned S5 and will not be described in detail here.

[0225] S609: Using the user's eye response data, perform personalized correction on the eye response parameter threshold.

[0226] Specifically, by collecting eye reaction data over a period of time (for example, a few days, a week, etc.), the user's eye habits (such as the highest and lowest blinking frequencies, etc.), pupil size, etc. can be extracted to correct the parameter thresholds in the machine learning model.

[0227] S603 - S609 may be executed by the aforementioned identification unit 1022 .

[0228] S610: Regularly upload the eye response data and corresponding recognition data to the cloud or backend.

[0229] It should be noted that the recognition data may include at least one of the recognition results of emotion recognition and the results of speech recognition.

[0230] Before uploading, the recognition results may be desensitized to filter out sensitive information. Sensitive information may include, but is not limited to: name, age, place of residence, ID number, contact information, email address, etc.

[0231] The uploaded data can be used to train the machine learning model, and the recognition data can be used to generate labels. Please refer to the previous introduction of this article for the specific training process and related introduction of labels, which will not be repeated here.

[0232] For interaction between the cloud or backend and the user side, please refer to Figure 7 .

[0233] S610 may be executed by the aforementioned identification unit 1022 .

[0234] In addition, in other embodiments of the present invention, the user side may also be used only to collect data, display recognition results and provide a human-computer interaction interface. Data processing, recognition, use of correction data to correct emotion recognition types, state information and state thresholds, personalized corrections, etc. can be achieved by the cloud or background.

[0235] In addition, the above-mentioned APP can also calculate the physical and mental state based on personal data based on the algorithm of health big data, and put forward individual targeted suggestions and beneficial intervention strategies, such as providing breathing adjustment suggestions, rest suggestions, playing music, etc.

[0236] The above embodiments are based on smart terminals, combining existing high-resolution camera technology, mature computer vision technology and advanced artificial intelligence algorithms to realize the recognition of human emotions and cognition. Since smart terminals are widely used, they are universal and can be used to instantly evaluate the user's emotions, attention, fatigue, etc., and are suitable for the public (such as office workers) to regulate stress, use their brains scientifically, and maintain physical and mental health and balance.

[0237] Next, the recognition method based on the smart wearable device is introduced. In this embodiment, the micro camera of the smart wearable device (such as smart glasses) collects eye reaction data and transmits it to the smart terminal via wireless transmission. The APP software installed in the smart terminal processes the data and outputs the recognition result of emotion cognition.

[0238] Of course, when the identification system includes the background or cloud, smart wearable devices and smart terminals both belong to the user side.

[0239] For example, see Smart Glasses. Figure 8 There is a miniature electronic camera 801 on each side of the lens of the smart glasses, that is, at the junction of the lens and the handle.

[0240] The micro electronic camera 801 can continuously capture eye video at close range. Of course, the smart glasses can switch between two working modes: a fixed time recording mode and a continuous recording mode.

[0241] When working in the fixed time recording mode, the micro electronic camera 801 can capture eye videos twice a day. For details, please refer to the above-mentioned S601. When working in the continuous recording mode, the micro electronic camera 801 will continuously capture eye videos.

[0242] The miniature electronic camera 801 may have a lens for two-way photography.

[0243] In addition to the micro camera 801, a skin electrical signal sensor (eg, an electronic component that is sensitive to skin bioelectricity) may also be provided on the inner side of the mirror handle where it contacts the ear.

[0244] The skin electrical signal sensor may be a PPG sensor, which uses PPG technology to collect PPG data related to the autonomic nervous system, including heart rate, blood pressure, and respiratory rate. PPG technology often uses green light or red light as a measurement light source. The PPG sensor further includes an LED lamp 802 and a photoelectric sensor.

[0245] The above-mentioned LED lamps may specifically include red light LEDs and infrared LED lamps. Of course, the red light LEDs and infrared LED lamps may also be replaced by green light LED lamps.

[0246] In addition, a forebrain cortex signal sensor (such as an element sensitive to electroencephalogram signals) can be set in the middle section of the mirror handle. The forebrain cortex signal sensor can specifically be an EEG sensor, which uses EEG technology to collect relevant brain wave signals.

[0247] Currently, the accuracy of EEG signals mainly depends on the number of wires. Many EEG products on the market only have two wires (referred to as 2-lead), which makes it difficult to improve the accuracy to meet medical requirements.

[0248] Optionally, both temples and nose pads of the smart glasses are designed with flexible bio-electrodes. Specifically, the inner back side of the temples is a flexible bio-electrode 803, and the two nose pads 804 are flexible bio-electrodes. This design ensures the comfort of the glasses while ensuring a three-conductor design of the electrical signal. Compared with single-conductor and dual-conductor designs, multi-conductor can effectively reduce noise interference, improve the accuracy of collected signals and the accuracy of subsequent algorithms.

[0249] In addition to combining EEG and PPG technologies, other related technologies may also be combined, such as but not limited to EOG (electrooculogram), ERG (electroretinogram), EMG (electromyogram) and other technologies.

[0250] Optionally, a mechanical sleep switch or a timing switch 805 may be provided at the connection between the temple and the frame of the smart glasses.

[0251] Setting a mechanical sleep switch can ensure that the glasses automatically enter sleep mode after they are folded and closed.

[0252] By setting a timer switch, the user can manually set the timer and enter the sleep state when the timer is reached.

[0253] A mechanical sleep switch or a timing switch may be provided on one side, or on both sides.

[0254] A touch screen 806 may be provided on the outside of the mirror handle, and the user may operate different functions through different gestures such as clicking, double-clicking, sliding forward, sliding backward, etc.

[0255] The aforementioned input of correction data can be achieved through different gestures of the user.

[0256] In addition, a switch sensor (switch sensor) can be provided at the junction of the frame and the temple to detect the opening and closing state of the temple.

[0257] The above-mentioned smart glasses contain multiple sensors, which can compensate for the defects of different sensors and achieve multiple guarantees of accuracy.

[0258] In addition, the above sensors can also detect whether the user is wearing smart glasses:

[0259] For example, the opening and closing of the temples can be detected; obviously, when the temples are detected to be in the closed state, the user is not wearing the smart glasses;

[0260] For another example, the PPG sensors on both sides can detect whether the skin is in contact by receiving different absorption rates of red light and infrared light. Because the PPG sensor is located on the inside of the handle where it touches the ear, no PPG signal is generated when the smart glasses are placed on the legs. This can avoid the situation where the glasses are mistakenly thought to be still being worn when they are placed on the legs, thereby recording false data.

[0261] When the PPG signal and the EEG signal are detected at the same time and the temples are detected to be in an open state, it can be determined that the user is wearing smart glasses.

[0262] In addition, the smart glasses may also include a physical output device. The physical output device may include a power system and a data output device. The power system includes but is not limited to a rechargeable battery 807. In one example, the rechargeable battery 807 is disposed at the end of the glasses handle, and can work continuously for at least 2 hours after a full charge.

[0263] The data output device includes but is not limited to a Bluetooth or WiFi device. Specifically, a Bluetooth chip 808 may be built into the handle of the mirror.

[0264] In addition, the smart glasses may also include a voice collection device (such as a miniature microphone).

[0265] See also Fig. 9 The recognition method based on smart glasses includes the following steps:

[0266] S901: Smart glasses collect brain response data.

[0267] In this embodiment, the data collected by the smart glasses include eye micro-video, PPG and EEG data, which can be transmitted to a smart terminal (such as a mobile phone) via Bluetooth.

[0268] S902: The smart glasses collect voice data.

[0269] S902 is similar to the aforementioned S602 and will not be described in detail here.

[0270] Voice data can also be transmitted to smart terminals via Bluetooth.

[0271] S903: The APP of the smart terminal recognizes the voice data and obtains a voice recognition result.

[0272] S903 is similar to the aforementioned S603 and will not be described in detail here.

[0273] S904: The APP of the smart terminal processes the brain response data to obtain brain response parameter values.

[0274] During the data processing process, the left and right cameras on the smart glasses can be used to calculate the angle and distance between the smart glasses and the eyes, and then the actual size of the eyes can be inferred based on the angle and distance between the smart glasses and the eyes, so as to restore the proportion of collected images in different wearing conditions, or perform size transformation on the eye response parameter values.

[0275] For the relevant contents of S904, please refer to the aforementioned S604 and S2, which will not be elaborated here.

[0276] S905: The APP of the smart terminal uses a machine learning model to analyze the above-mentioned brain reaction parameter values ​​to obtain the recognition result of emotion cognition.

[0277] The recognition results of the above-mentioned emotion recognition can be displayed by the smart terminal. The smart terminal can also transmit the recognition results to the smart glasses, which will be displayed to the user in the form of images or voice.

[0278] Alternatively, the recognition results of emotion recognition can be uploaded to a pre-set mobile phone, cloud or background.

[0279] S905 is similar to the aforementioned S3 and will not be described in detail here.

[0280] S906-S910 are similar to the aforementioned S606-S610 and are not described in detail here.

[0281] The smart glasses in this embodiment mainly use computer vision to accurately scan eye reactions, and at the same time combine EEG technology for detecting brain electrical signals and PPG technology for detecting skin electrical signals to collect data.

[0282] Of course, the steps performed by the smart terminal APP can also be performed by smart glasses or the cloud or background.

[0283] Smart glasses can be worn for a long time, so they can be used for long-term monitoring and care of chronic mental illness. Of course, they can also be used to monitor the deterioration or onset of mental illness in real time. They can be worn by patients with common mental illnesses, including depression, autism, post-traumatic stress disorder, and schizophrenia, to provide the function of timely prediction, monitoring, and intervention of disease dynamics in daily life.

[0284] The smart terminal can also output intervention adjustment measures and suggestions through smart glasses, such as but not limited to outputting breathing adjustment suggestions, music therapy measures / suggestions, light therapy measures / suggestions or cognitive behavioral therapy measures / suggestions, making medication suggestions, etc.

[0285] In addition, smart glasses-based recognition methods can also be used to instantly capture and interpret eye stress responses to observing an object or scene.

[0286] For example, a two-way camera lens can simultaneously capture the eye and the external environment, which is conducive to the instant capture and interpretation of the eye's stress response to observing an object or scene.

[0287] For example, in tests of psychological lie detection or the effects of commercial advertising, the two-way camera lens can simultaneously capture the external environment and eye video, and the recognition results can be interpreted together with the external environment to understand the wearer's emotional or cognitive changes when observing an object or scene.

[0288] For another example, the above recognition method can be used to detect people's stress reactions to specific scenes and objects.

[0289] For example, in the treatment of post-war creative sequelae, different objects (such as photos) are shown to the wearer or different events are told, and a two-way camera lens is used to take pictures simultaneously. This can establish an association between the recognition results of different objects and emotional cognition, and determine which object or events will cause an excessive reaction, thereby helping doctors to carry out targeted treatment.

[0290] For another example, smart glasses can continuously monitor whether the wearer is driving fatigued (based on the degree of fatigue) during driving. If it is detected that the wearer is close to driving fatigued or is already driving fatigued, a prompt can be given.

[0291] In summary, the recognition system and recognition method provided by the present invention combine the existing high-resolution camera technology, mature computer vision technology and advanced artificial intelligence algorithms, mainly based on accurate scanning of eye reactions, to achieve precise, intelligent, immediate and scientific detection and evaluation of the emotional state and cognitive state of the human brain, and are timely, efficient and easy to operate.

[0292] However, there is a lack of brain health products with similar functions on the market. Take smart glasses as an example. Although the smart glasses on the market currently have camera functions, they are not used to interpret and monitor the cognitive state of the brain.

[0293] For the immediate interpretation and long-term monitoring of current brain cognition or brain disease status, the traditional method is to use brain wave detection and MRI scanning.

[0294] However, in terms of electroencephalogram (EEG) detection technology, the electrodes receive weak electrical signals from the cerebral cortex by contacting the scalp, which are often very noisy and difficult to parse clearly. EEG signals can currently be divided into several bands according to different frequencies, including δ, θ, α, and β waves. When people are asleep, the signal analysis of brain waves is relatively clear; but when people are awake during the day, the EEG signals generated are extremely complex and difficult to parse. Especially in the interpretation of human emotions and cognitive states, the research on EEG signals has been difficult to make major breakthroughs.

[0295] Although functional magnetic resonance imaging (fMRI) technology has advantages in identifying the different degrees of involvement of various brain regions in various human emotional activities, it is not convenient to operate, especially because the subjects need to lie flat and keep their heads fixed; and, when it comes to detecting the cognitive state of the human brain or emotional disorders, current research is also very limited.

[0296] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the differences can be referred to the method part.

[0297] The professionals may further appreciate that the units and model steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0298] The steps of the method or model described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a WD-ROM, or any other form of storage medium known in the art.

[0299] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A recognition method, characterized in that: Applied to the user side, including: Collecting the user's brain reaction data, the brain reaction data at least including eye reaction data, the eye reaction data is collected by a camera, and the eye reaction data includes eye video or eye image; Processing the brain reaction data to obtain brain reaction parameter values, wherein the brain reaction parameter values ​​at least include eye reaction parameter values, wherein the eye reaction parameter values ​​include parameter values ​​corresponding to each eye reaction parameter, wherein the eye reaction parameter includes one or more of the following: contrast and brightness of the eye; speed, direction and frequency of eyeball movement; amplitude and speed of pupil reaction; interpupillary distance; speed, amplitude and frequency of blinking; muscle contraction of the eye, wherein the eye includes periorbital area and eyebrows; The brain reaction parameter value is input into a machine learning model, and the machine learning model outputs an emotion cognition recognition result, wherein the emotion cognition recognition result includes the recognition of the emotion cognition type and the corresponding state information, wherein the emotion cognition type includes an emotion type and a cognition type, wherein the emotion type includes a mood subtype and a fatigue subtype, wherein the cognition type includes an attention subtype and a stress subtype, and wherein the state information includes a text description and / or a score; wherein the machine learning model is trained using labeled training samples, wherein the training samples include brain reaction parameters from healthy individuals or patients; wherein the labels include a mood state information label, a fatigue state information label, an attention state information label, and a stress state information label, and wherein the labels are obtained by: using a microphone to collect voice data of a user, wherein the content of the voice data includes a specific state description of mood, fatigue, attention, and stress, or a self-rating of an emotion type or a cognition type, and generating a mood state information label, a fatigue state information label, an attention state information label, and a stress state information label of the user during the training process based on the voice recognition result obtained from the voice data; Using the user's eye response data, performing personalized modification on the eye response parameter threshold; Regularly upload brain reaction data and corresponding recognition data to the cloud or background, the recognition data includes at least one of the recognition results of emotion cognition and speech recognition results, the uploaded brain reaction data is used to generate training samples or test samples, the recognition data is used to generate labels for training samples or labels for test samples, and the training samples are used to train the machine learning model; the recognition method also includes using labeled test samples to test the recognition accuracy and recognition speed of the machine learning model, and the test samples include brain reaction parameters from healthy individuals or patient individuals; if the machine learning model does not meet the preset conditions, perform one or more of the following operations and retrain: reselect eye reaction parameters; adjust the weight value of the machine learning model; adjust the state threshold corresponding to the emotion cognition type; adjust at least one of the type and content of the label; wherein the preset conditions include: the recognition accuracy of the machine learning model is not lower than the accuracy threshold and the recognition speed is not lower than the speed threshold.

2. The identification method according to claim 1, characterized in that: The output of the recognition result of emotion cognition by the machine learning model includes: the machine learning model identifies the emotion cognition type and the corresponding score according to the eye reaction parameter value and the eye reaction parameter threshold; The machine learning model determines the state information corresponding to the emotion recognition type according to the state threshold and the corresponding score corresponding to the emotion recognition type.

3. The identification method according to claim 2, characterized in that: The recognition method further includes: receiving correction data input by the user; and using the correction data to correct at least one of the emotion recognition type, state information, and state threshold.

4. The identification method according to claim 1, characterized in that: The brain response data also includes at least one of a forebrain cortex signal and a skin electrical signal; the brain response parameter value also includes: at least one of a forebrain cortex parameter value and a skin electrical parameter value; wherein, the forebrain cortex parameter value includes parameter values ​​corresponding to each forebrain cortex parameter, and the skin electrical parameter value includes parameter values ​​corresponding to each skin electrical parameter.

5. A recognition system, characterized in that: The system comprises a collection device and a central control system, wherein the central control system at least comprises an identification device, a cloud or a backend, and the collection device comprises smart glasses, wherein a micro-electronic camera is arranged at the junction of the lens and the handle of the smart glasses; wherein: the smart glasses are used to: collect the brain reaction data of the user, wherein the brain reaction data at least comprises the eye reaction data, wherein the eye reaction data is collected by the micro-electronic camera, and the eye reaction data comprises an eye video or an eye image; The recognition device is used to: process the brain reaction data to obtain brain reaction parameter values, the brain reaction parameter values ​​at least include eye reaction parameter values, the eye reaction parameter values ​​include parameter values ​​corresponding to each eye reaction parameter, the eye reaction parameters include one or more of the following: eye contrast and brightness; speed, direction and frequency of eye movement; amplitude and speed of pupil reaction; interpupillary distance; speed, amplitude and frequency of blinking; eye muscle contraction, the eye includes the periorbital area and eyebrows; input the brain reaction parameter values ​​into a machine learning model, and the machine learning model outputs an emotion recognition result, the emotion recognition result includes identifying an emotion recognition type and corresponding state information, the emotion recognition type includes an emotion type and a cognitive type, the emotion type includes a mood subtype and a fatigue subtype, the cognitive type includes an attention subtype and a stress subtype, and the state information includes a text description and / or a score; wherein the machine learning model is trained using labeled training samples, the training samples include brain reaction parameters from healthy individuals or patients; the labels include mood state information labels, fatigue subtypes, and attention subtypes. Fatigue state information label, attention state information label and stress state information label, the label is obtained in the following ways: using a microphone to collect the user's voice data, the content of the voice data includes a specific description of the mood, fatigue, attention and stress state or a self-rating of the emotional type or cognitive type, and generating the user's mood state information label, fatigue state information label, attention state information label and stress state information label during the training process based on the voice recognition result obtained from the voice data; using the user's eye reaction data to personalize the eye reaction parameter threshold; regularly uploading brain reaction data and corresponding recognition data to the cloud or background, the recognition data includes at least one of the recognition result of emotional cognition and the voice recognition result; the cloud or background is used to: receive the uploaded brain reaction data for generating training samples or test samples, receive the uploaded recognition data for generating labels of training samples or labels of test samples, and the training samples are used to train the machine learning model; use the labeled test samples to test the recognition accuracy and recognition speed of the machine learning model, and the test samples include brain reaction parameters from healthy individuals or patients;If the machine learning model does not meet the preset conditions, perform one or more of the following operations and retrain: reselect eye reaction parameters; adjust the weight value of the machine learning model; adjust the state threshold corresponding to the emotion recognition type; adjust at least one of the type and content of the label; wherein the preset conditions include: the recognition accuracy of the machine learning model is not less than the accuracy threshold and the recognition speed is not less than the speed threshold. ; 6. The identification system according to claim 5, characterized in that: The smart glasses also include a skin electrical signal sensor and a forebrain cortex signal sensor; the skin electrical signal sensor is arranged at the inner side of the mirror handle and the part in contact with the ear, and is used to collect skin electrical signals; the forebrain cortex signal sensor is arranged in the middle of the mirror handle, and is used to collect forebrain cortex signals.

7. The identification system according to claim 6, characterized in that The rear inner side of the temple of the smart glasses is a flexible bio-electrode, and the two nose pads of the smart glasses are flexible bio-electrodes.

8. An intelligent terminal, characterized in that: include: an acquisition unit, for acquiring the user's brain reaction data, wherein the brain reaction data at least includes eye reaction data, the eye reaction data is collected by a camera, and the eye reaction data includes eye video or eye image; an identification unit, for processing the brain reaction data to obtain brain reaction parameter values, wherein the brain reaction parameter values ​​at least include eye reaction parameter values, the eye reaction parameter values ​​include parameter values ​​corresponding to each eye reaction parameter, and the eye reaction parameters include one or more of the following: eye contrast and brightness; speed, direction and frequency of eye movement; amplitude and speed of pupil reaction; interpupillary distance; speed, amplitude and frequency of blinking; eye muscle contraction, wherein the eye includes the periorbital area and eyebrows; the brain reaction parameter values ​​are input into a machine learning model, and the machine learning model outputs an emotion recognition recognition result, wherein the emotion recognition recognition result includes recognizing the emotion The cognitive type and the corresponding state information, the emotional cognitive type includes the emotional type and the cognitive type, the emotional type includes the mood subtype and the fatigue subtype, the cognitive type includes the attention subtype and the stress subtype, and the state information includes a text description and / or a score; wherein the machine learning model is trained using labeled training samples, and the training samples include brain reaction parameters from healthy individuals or patient individuals; the labels include mood state information labels, fatigue state information labels, attention state information labels and stress state information labels, and the label acquisition method includes: using a microphone to collect the user's voice data, and the content of the voice data includes The specific state description of mood, fatigue, attention and stress, or the self-rating of emotion type or cognitive type, generates the user's mood state information label, fatigue state information label, attention state information label and stress state information label during the training process based on the speech recognition result obtained from the speech data; uses the user's eye reaction data to personalize the eye reaction parameter threshold; regularly uploads brain reaction data and corresponding recognition data to the cloud or background, the recognition data includes at least one of the recognition results of emotion cognition and the speech recognition result, the uploaded brain reaction data is used to generate training samples or test samples, and the recognition data is used to generate labels or test samples of training samples. A label for a test sample, wherein the training sample is used to train the machine learning model; using the labeled test sample to test the recognition accuracy and recognition speed of the machine learning model, wherein the test sample includes brain reaction parameters from a healthy individual or a patient individual; if the machine learning model does not meet the preset conditions, performing one or more of the following operations and retraining: reselecting eye reaction parameters; adjusting the weight value of the machine learning model; adjusting the state threshold corresponding to the emotion recognition type; adjusting at least one of the type and content of the label; wherein the preset conditions include: the recognition accuracy of the machine learning model is not less than the accuracy threshold and the recognition speed is not less than the speed threshold.

9. A storage medium, characterized in that: The storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the identification method according to any one of claims 1 to 4.

Images