Human factor intelligent cognitive ability training method and device, electronic equipment and storage medium

Abstract

The application discloses a human factor intelligent cognitive ability training method and device, electronic equipment and computer storage medium, wherein the human factor intelligent cognitive ability training method comprises the following steps: acquiring multi-modal data of a testee, wherein the multi-modal data comprises at least two pieces of data in at least one category of human body data of the testee, environment data of an environment where the testee is located and interaction data of the testee and the environment; predicting current cognitive state information of the testee according to the multi-modal data; and generating state feedback information according to the current cognitive state information of the testee, wherein the state feedback information is used for guiding the testee to adjust the cognitive state to a target cognitive state or maintain the cognitive state in the target cognitive state. The technical scheme provided by the application can significantly improve the efficiency and effect of human factor intelligent cognitive ability training and improve the reliability of the training result.

Description

Technical Field

[0001] This application belongs to the fields of human factors intelligence and medical health technology, and particularly relates to a human factors intelligence cognitive ability training method, device, electronic device and computer storage medium, as well as a human factors intelligence cognitive ability training system, and a training method and device for a state classification model. Background Technology

[0002] Existing research has shown that biofeedback training based on EEG signals can improve cognitive abilities such as attention and executive function. The scientific basis for this training lies in neuroplasticity, which is the phenomenon that the strength, resilience, and stability of connections between neurons can change in response to external environmental stimuli to adapt to the external environment. Biofeedback training based on EEG signals uses operant conditioning to train specific EEG patterns associated with optimal cognitive states. This can optimize the strength of synaptic connections between brain neurons, enhance the resilience and stability of neural links, and improve the dominant rhythm and amplitude of bioelectrical signals, thereby enhancing cognitive abilities.

[0003] Existing cognitive ability training based on biofeedback technology is all based on electroencephalogram (EEG) signals. Examples include EEG-based relaxation and attention enhancement programs, as well as EEG-based activation and concentration programs. The EEG-based relaxation and attention enhancement programs include α / θ and α programs, while the EEG-based activation and concentration programs include frontal α-asymmetric, θ / β, and SMR-β programs. However, the inventors discovered during the development of this application that the above-mentioned cognitive ability training only involves a single EEG signal, thus resulting in a single signal source and poor reliability of the training results. Summary of the Invention

[0004] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a method, apparatus, electronic device, and computer storage medium for training human-caused intelligent cognitive abilities, as well as a system for training human-caused intelligent cognitive abilities, and a method and apparatus for training state classification models, which can train human-caused intelligent cognitive abilities based on rich signal sources and improve the reliability of training results.

[0005] Firstly, this application provides a method for training human cognitive abilities based on human factors, including:

[0006] Acquire multimodal data of the test subject, including at least two data items from at least one category of the test subject's human body data, the environmental data of the environment in which the test subject is located, and the interaction data between the test subject and the environment.

[0007] Predict the current cognitive state information of the test subjects based on multimodal data;

[0008] Based on the test subject's current cognitive state information, a status feedback information is generated. This status feedback information is used to guide the test subject to adjust their cognitive state to the target cognitive state or maintain the target cognitive state.

[0009] In some embodiments, predicting the current cognitive state information of a test subject based on multimodal data includes:

[0010] Obtain cognitive state category information of test subjects based on multimodal data; or,

[0011] Obtain the data threshold corresponding to the target cognitive state, and determine the threshold compliance information of the multimodal data based on the multimodal data and the data threshold.

[0012] In some embodiments, obtaining cognitive state category information of the test subject based on multimodal data includes:

[0013] Multimodal data is input into a pre-trained state classification model to obtain cognitive state category information of the test subjects.

[0014] In some embodiments, the cognitive state includes at least one sub-cognitive state among attention state, memory state, executive state and divergent thinking state, and each sub-cognitive state is respectively set with a corresponding target cognitive state.

[0015] Multimodal data is input into a pre-trained state classification model to obtain cognitive state category information of the test subjects, including:

[0016] Multimodal data is input into a pre-trained state classification model to obtain the number of sub-cognitive states in which the test subject has reached the target cognitive state, and / or the cognitive state category of at least one sub-cognitive state.

[0017] In some embodiments, the human body data of the test subject includes at least two of the following: electroencephalogram (EEG) signal data, electrocardiogram (ECG) signal data, electrodermal conductance (TEA) signal data, electromyography (EMG) signal data, body temperature data, pulse data, respiratory rate data, near-infrared spectroscopy data, heart rate data, and blood oxygen saturation data.

[0018] Obtain the data thresholds corresponding to the target cognitive state, including:

[0019] Obtain data thresholds for at least two data points corresponding to the target cognitive state;

[0020] Determining threshold compliance information for the multimodal data based on the multimodal data and the data threshold includes:

[0021] The number of data points reaching the data threshold is determined based on the multimodal data and the data threshold, and / or, the order of magnitude of at least two data points.

[0022] In some embodiments, state feedback information is generated based on the test subject's current cognitive state information, including:

[0023] Status feedback information is generated based on cognitive state category information or threshold attainment information.

[0024] In some embodiments, it also includes:

[0025] The target cognitive state is determined based on the test subject's current cognitive state information predicted at least twice in the current environment.

[0026] In some embodiments, when the number of predictions of the test subject's current cognitive state information in the current environment reaches a preset threshold, the target cognitive state is determined based on the test subject's current cognitive state information predicted at least twice in the current environment.

[0027] In some embodiments, determining the target cognitive state based on at least two predictions of the test subject's current cognitive state in the current environment includes:

[0028] Based on the current cognitive state information of the test subject predicted at least twice in the current environment, determine at least two cognitive states of the test subject.

[0029] Choose the best of at least two cognitive states as the target cognitive state.

[0030] In some embodiments, acquiring the multimodal data of the test subject includes:

[0031] Receive multimodal data of the test subjects sent by the signal acquisition equipment.

[0032] In some embodiments, the above method further includes:

[0033] A status feedback message is sent to the information feedback device so that the status feedback message is fed back on the information feedback device.

[0034] In some embodiments, the status feedback information includes at least one of visual feedback information, auditory feedback information, and tactile feedback information.

[0035] In some embodiments, the above method further includes:

[0036] Generate stimulus control information based on current cognitive state information;

[0037] The stimulation control information is sent to the stimulation device to control the stimulation device to apply stimulation signals to the test subject, so that the test subject adjusts the cognitive state to the target cognitive state or maintains the target cognitive state.

[0038] In some embodiments, the stimulation signal includes at least one of an optical signal, an electrical signal, or a magnetic signal.

[0039] In some embodiments, the above method further includes:

[0040] Generate environmental control information based on current cognitive state information;

[0041] Send environmental control information to the environmental control equipment to control the environmental control equipment to adjust the environmental data of the environment in which the test subject is located, so that the cognitive state of the test subject is adjusted to the target cognitive state or maintained at the target cognitive state.

[0042] Secondly, this application provides a training method for a state classification model, including:

[0043] Acquire sample data, which includes multimodal data of the test subjects and cognitive state category information of the test subjects corresponding to the multimodal data. The multimodal data includes at least two data items from at least one category of human body data of the test subjects, environmental data of the environment in which the test subjects are located, and interaction data between the test subjects and the environment.

[0044] Based on the sample data, the state classification model is trained to obtain a pre-trained state classification model.

[0045] In some embodiments, obtaining cognitive state category information of the test subject corresponding to the multimodal data includes:

[0046] Obtain behavioral response data of the test subjects corresponding to the multimodal data;

[0047] The cognitive state category information of the test subjects is determined based on behavioral response data.

[0048] In some embodiments, cognitive states include at least one sub-cognitive state selected from attention state, memory state, executive function state, and divergent thinking state. Determining the cognitive state category information of the test subject based on the behavioral response data includes:

[0049] The cognitive state category information for each sub-cognitive state of the test subject is determined based on the behavioral response data.

[0050] In some embodiments, sample data includes historical sample data and current environmental sample data;

[0051] The above-mentioned training of the state classification model based on sample data to obtain a pre-trained state classification model includes:

[0052] Based on historical sample data, the classification state model is initially trained to obtain the initially trained state classification model.

[0053] Based on the sample data of the current environment, the initially trained state classification model is optimized to obtain the optimized state classification model.

[0054] Thirdly, this application provides a human-centric intelligent cognitive ability training device, including:

[0055] The data acquisition module is used to acquire multimodal data of the test subjects. The multimodal data includes at least two data items from at least one category of the test subjects' human body data, environmental data of the environment in which the test subjects are located, and interaction data between the test subjects and their environment.

[0056] The prediction module is used to predict the current cognitive state information of the test subjects based on multimodal data;

[0057] The feedback generation module is used to generate status feedback information based on the test subject's current cognitive state information. The status feedback information is used to guide the test subject to adjust their cognitive state to the target cognitive state or maintain the target cognitive state.

[0058] Fourthly, this application provides a human factor intelligent cognitive ability training system, including the aforementioned human factor intelligent cognitive ability training device and a signal acquisition device. The signal acquisition device is used to collect multimodal data of the test subject and send the multimodal data of the test subject to the human factor intelligent cognitive ability training device.

[0059] In some embodiments, it also includes:

[0060] Information feedback device, used to receive status feedback information sent by human cognitive ability training device, and to feed back the status feedback information; and / or,

[0061] The stimulation device is used to apply stimulation signals to the test subject according to stimulation control information, so that the test subject adjusts the cognitive state to the target cognitive state or maintains the target cognitive state.

[0062] Fifthly, this application provides a training device for a state classification model, comprising:

[0063] The sample acquisition module is used to acquire sample data, which includes multimodal data of the test subjects and cognitive state category information of the test subjects corresponding to the multimodal data. The multimodal data includes at least two data items from at least one category of the test subjects' human body data, environmental data of the environment in which the test subjects are located, and interaction data between the test subjects and the environment.

[0064] The training module is used to train the state classification model based on sample data to obtain a pre-trained state classification model.

[0065] Sixthly, this application provides an electronic device including a processor and a memory, the processor being configured to execute a computer program in the memory, the computer program being capable of executing the aforementioned human-caused intelligent cognitive ability training method or state classification model training method.

[0066] Seventhly, this application provides a computer storage medium for storing a computer program capable of executing the above-described human cognitive ability training method or state classification model training method.

[0067] The human-centered intelligent cognitive ability training scheme provided in this application predicts the current cognitive state information based on acquired multimodal data, and then generates state feedback information based on the current cognitive state information to guide the test subject to adjust the cognitive state to or maintain the target cognitive state. By using the aforementioned state feedback information, the test subject can be motivated, thereby providing a positive feedback mechanism that guides the test subject to actively improve their physiological state, adjusting their cognitive state from a non-target cognitive state to the target cognitive state, or maintaining it at the target cognitive state. This trains the user to maintain the physiological state required for the target cognitive state in the long term. Furthermore, since multimodal data is used, the multimodal data can include at least two data from at least one category of the test subject's human body data, environmental data of the test subject's environment, and interaction data between the test subject and the environment. This allows the data to more accurately reflect the test subject's physiological state and the current environmental state, and predict more accurate cognitive state information, thereby providing more precise state feedback information. This improves the accuracy of the feedback process during cognitive ability training and enhances the reliability of the training results.

[0068] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0069] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0070] Figure 1 This is a flowchart illustrating a human-centric intelligent cognitive ability training method in an embodiment of this application.

[0071] Figure 2 This is a flowchart illustrating yet another human-centric intelligent cognitive ability training method in this application embodiment;

[0072] Figure 3 This is a flowchart illustrating another human-centric intelligent cognitive ability training method in the embodiments of this application;

[0073] Figure 4 This is a flowchart illustrating another human-centric intelligent cognitive ability training method in the embodiments of this application;

[0074] Figure 5 This is a flowchart illustrating a cognitive ability training method in an embodiment of this application;

[0075] Figure 6 This is a flowchart illustrating a training method for a state classification model in an embodiment of this application.

[0076] Figure 7 This is a schematic diagram of the structure of a human-centric intelligent cognitive ability training device according to an embodiment of this application;

[0077] Figure 8 This is a schematic diagram of the structure of a human-centric intelligent cognitive ability training system according to an embodiment of this application;

[0078] Figure 9 This is a schematic diagram of the structure of a training device for a state classification model in an embodiment of this application;

[0079] Figure 10 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0080] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0081] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0082] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0083] It should be noted that the concepts of "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0084] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0085] Existing technologies for training human cognitive abilities only allow the use of a single signal source, resulting in a limited range of signal sources. This application provides a human cognitive ability training scheme that utilizes multimodal data for training. This multimodal data can include at least two data points from at least one category: the test subject's human body data, environmental data of the test subject's environment, and interaction data between the test subject and the environment. This multimodal data can more accurately and comprehensively reflect the current physiological and environmental states of the test subject. Using this multimodal data for cognitive ability training significantly improves the efficiency and effectiveness of training, and enhances the reliability of the training results.

[0086] Specifically, the human-centered intelligent cognitive ability training scheme of this application embodiment includes a human-centered intelligent cognitive ability training method that may include: acquiring multimodal data of the test subject; then, predicting the current cognitive state information of the test subject based on the multimodal data; and finally, generating state feedback information based on the current cognitive state information of the test subject. The state feedback information is used to guide the test subject to adjust the cognitive state to a target cognitive state or maintain the target cognitive state. The target cognitive state is a type of cognitive state of the test subject, which is the cognitive state pursued in the cognitive ability training process. Test subjects in the target cognitive state have better cognitive abilities.

[0087] The embodiments of this application, through the above-described technical solution, enable test subjects to quickly adjust their state to reach the target cognitive state when they are not in it. Furthermore, once the target cognitive state has been reached, the system guides the test subjects to maintain that state. This technical solution allows for the long-term maintenance of the test subjects' cognitive state at the target state. Moreover, due to the use of multimodal data, compared to EEG signal data from a single signal source, it can more accurately reflect the test subjects' physiological state and current environmental state. The current cognitive state information predicted by this multimodal physiological perception data will also more accurately reflect the test subjects' cognitive state, allowing subsequent state feedback information to more accurately match the test subjects' needs. This, in turn, more efficiently and accurately guides the test subjects to adjust their physiological state, current environmental state, and cognitive state to their optimal state, quickly reaching or maintaining the target cognitive state.

[0088] In the execution of the above-mentioned human-centered intelligent cognitive ability training method, the step of predicting the current cognitive state information of the test subject based on multimodal data can be to obtain the cognitive state category information of the test subject, which can be implemented by a pre-trained state classification model. The human-centered intelligent cognitive ability training scheme of this application embodiment also provides a training method for the state classification model. This application embodiment also provides a human-centered intelligent cognitive ability training system, including a human-centered intelligent cognitive ability training device, a signal acquisition device, and an information feedback device or a stimulation device, etc. The human-centered intelligent cognitive ability training device can execute the above-mentioned human-centered intelligent cognitive ability training method, the signal acquisition device can acquire the multimodal data of the test subject, the information feedback device can generate and feed back the stimulation content corresponding to the state feedback information, and the stimulation device can apply stimulation signals to the test subject according to stimulation control information, so that the test subject adjusts the cognitive state to the target cognitive state.

[0089] The following detailed description of the human-centric cognitive ability training scheme provided in this application is illustrated with the accompanying drawings.

[0090] Figure 1 This is a flowchart illustrating a human-centered intelligent cognitive ability training method according to an embodiment of this application. The method can be executed on an electronic device, including but not limited to servers and local computers. Figure 1 As shown, the method includes the following steps:

[0091] Step 101: Obtain multimodal data of the test subject. The multimodal data includes at least two items from at least one category of the test subject's human body data, environmental data of the test subject's environment, and interaction data between the test subject and the environment.

[0092] The technical solution provided in this application can utilize multiple signal sources to train the cognitive abilities of test subjects. Specifically, data can be collected by setting up various types of signal acquisition devices. For example, physiological signal acquisition devices can be used to collect human body data of test subjects. In some embodiments, the aforementioned human body data includes, but is not limited to, at least one of the following: electroencephalogram (EEG) signal data, electrocardiogram (ECG) signal data, electrodermal conductance (EDA) signal data, electromyography (EMG) signal data, body temperature data, pulse data, respiratory rate data, near-infrared spectral data, heart rate data, and blood oxygen saturation data. For example, environmental data acquisition devices can be used to collect environmental data, where the environment can be determined according to the application scenario. Environmental data can refer to the environmental data of the test subject's current environment, such as temperature data, humidity data, brightness data, and noise data. For example, interactive data acquisition devices can be used to collect interaction data between the test subject and the environment. For instance, in a driving test scenario, the environment is a simulated cockpit, where the interactive data can include data from the test subject's operation of the steering wheel, brakes, etc. In this application embodiment, "multimodal" refers to at least two data items collected by the signal acquisition devices, which can include one or more categories of data. This step uses multimodal data, which, compared to EEG data from a single signal source, can obviously reflect the physiological state of the test subject more accurately.

[0093] Each type of data can be acquired by the corresponding signal acquisition device. The acquisition of the multimodal data mentioned above in this step can be the multimodal data of the test subject sent by the signal acquisition device.

[0094] Step 102: Predict the current cognitive state information of the test subject based on multimodal data;

[0095] This step involves predicting the test subject's current cognitive state based on their physiological, environmental, or interactive information. Cognitive state refers to the subjective state exhibited by the test subject in the cognitive situation, which can be reflected through the test subject's behavioral response data. This behavioral response data includes, but is not limited to, one or more parameters such as accuracy, reaction speed, false positive rate, false negative rate, and correct rejection rate. The test subject's cognitive state can be determined through one or more of these parameters.

[0096] In the specific implementation of this step, predicting the current cognitive state information of the test subject based on multimodal data can include two situations: The first situation is to obtain the cognitive state category information of the test subject based on the multimodal data obtained in step 101 above. This category information can include the specific category of the test subject's cognitive state, which can directly reflect the test subject's current cognitive state information. The second situation is to obtain the data threshold corresponding to the target cognitive state, and determine the threshold attainment information of the multimodal data based on the multimodal data and the data threshold. Since the above data threshold directly corresponds to the target cognitive state, if the multimodal data reaches the data threshold, the test subject's cognitive state will also reach the target cognitive state. The threshold attainment information of the multimodal data can include whether each data in the multimodal data reaches the data threshold, and / or the difference between each data and the data threshold, etc., which can also be used to reflect the test subject's current cognitive state information. This will be described in detail in subsequent embodiments.

[0097] Regarding the first scenario described above, this can be considered a positive prediction technical solution. It utilizes a pre-established correspondence between the multimodal data of the test subjects and cognitive state category information. After obtaining the multimodal data, the cognitive state category information of the test subjects can be predicted based on this correspondence. Specifically, in this embodiment, the cognitive state category is a division of the cognitive states of the test subjects. It can be directly divided into two categories: target cognitive state and non-target cognitive state, or it may include other categories. For the target cognitive state category, the behavioral response data of the test subjects in this target cognitive state can reflect that the test subjects are in a better physiological state and possess better cognitive states such as attention, memory, executive function, or divergent thinking. Specifically, in this embodiment, it can be used to test groups such as drivers, children, or the elderly who need cognitive ability enhancement. The target cognitive state is the cognitive state pursued during the cognitive ability training and enhancement process.

[0098] In some embodiments, cognitive states can also be classified into levels, such as high, medium and low levels, where high-level cognitive states are target cognitive states, and medium and low-level cognitive states are non-target cognitive states. This classification method can be based on rating cognitive states, such as rating cognitive states on a scale of 0 to 10. A cognitive state of 0-5 is considered low-level, a cognitive state of 5-8 is considered medium-level, and a cognitive state of 8-10 is considered high-level.

[0099] In some embodiments, a pre-trained state classification model can be used to obtain the current cognitive state information of the test subject. Therefore, the step of "obtaining the cognitive state category information of the test subject based on multimodal data" mentioned above can specifically include:

[0100] Multimodal data is input into a pre-trained state classification model to obtain cognitive state category information of the test subjects.

[0101] The state classification model can be a direct classification model that can directly classify the input multimodal data to obtain the cognitive state category of the test subject, such as whether it is a target cognitive state or a non-target cognitive state. Alternatively, in some cases, the state classification model can first obtain the cognitive state level or score based on the input multimodal data, such as a high-level, medium-level, or low-level cognitive state, or a cognitive state score of 90, 70, or 40 points. In this way, the cognitive state category of the test subject can also be obtained according to the predetermined classification principle, that is, whether it is a target cognitive state or a non-target cognitive state.

[0102] In the embodiments of this application, the state classification models used include, but are not limited to, logistic regression models, KNN models, support vector machine (SVM) models, decision tree models, random forest models, or other classification models with similar functions.

[0103] Regarding the second scenario described above, a correspondence between multimodal data and the target cognitive state has been established based on previous data sample accumulation. In this case, the value that the multimodal data should reach if the target cognitive state is achieved can be obtained based on this correspondence, and this value can be set as a data threshold. This allows us to obtain the data threshold corresponding to the target cognitive state. Then, based on the multimodal data and the data threshold, threshold attainment information can be determined. Because the data threshold and the target cognitive state directly correspond, the threshold attainment information can reflect the current cognitive state of the test subject. Those skilled in the art will understand that since the multimodal data in this embodiment includes at least two data items, the set data threshold can be a threshold set for all or some data items. The threshold attainment information can include which specific data items reach the data threshold, and if the data threshold is not reached, the magnitude of that data item can be obtained, i.e., the difference between it and the data threshold.

[0104] In this step, the current cognitive state information of the test subject has been predicted based on multimodal data. Whether cognitive state category information or threshold attainment information is used to characterize it, it is a further prediction based on the comprehensive and accurate reflection of the test subject's physiological state obtained in step 101. Therefore, the current cognitive state information obtained in this step can also more accurately reflect the cognitive situation of the test subject.

[0105] Step 103: Generate state feedback information based on the current cognitive state information mentioned above. This state feedback information is used to guide the test subject to adjust their cognitive state to the target cognitive state or maintain it at the target cognitive state.

[0106] In step 102 above, the test subject's accurate current cognitive state information has been obtained, enabling the generation of state feedback information. The specific content of the state feedback information can be determined based on the aforementioned current cognitive state information and can have a one-to-one correspondence with it. Generally speaking, if the test subject has not reached the target cognitive state, the state feedback information can guide the test subject to adjust their physiological state to reach the target cognitive state; if it is predicted that the test subject has already reached the target cognitive state, the aforementioned state feedback information can guide the test subject to maintain their physiological state and also maintain their cognitive state at the target cognitive state.

[0107] In this embodiment, the current cognitive state information can be represented by cognitive state category information or threshold achievement information. Therefore, state feedback information can be generated based on either the cognitive state category information or the threshold achievement information. Furthermore, the state feedback information included in this embodiment can include various forms, such as at least one of visual feedback information, auditory feedback information, and tactile feedback information. Using the aforementioned state feedback information can motivate the test subject, thereby providing a positive feedback mechanism that guides the test subject to actively improve their physiological state, adjusting them from a non-target cognitive state to a target cognitive state, or maintaining them in the target cognitive state. This trains the user to maintain the physiological state required for the target cognitive state in the long term. Moreover, because multimodal data is used, it can more accurately reflect the test subject's physiological state or current environmental state, thus predicting more accurate cognitive state information and providing more precise state feedback information. This improves the accuracy of the feedback process during human cognitive ability training and enhances the reliability of the training results.

[0108] In some embodiments, cognitive states may include multiple subjective states such as attention, memory, executive function, or divergent thinking. Therefore, for the cognitive states in this embodiment, they may include at least one sub-cognitive state among attention state, memory state, executive function state, and divergent thinking state. For each sub-cognitive state, a corresponding target cognitive state is set. Therefore, the current cognitive state information of the test subject in this embodiment can be understood as the state information of the above-mentioned at least one sub-cognitive state, and the cognitive state category information of the test subject includes the state category information of at least one sub-cognitive state. That is, for the above example of "inputting multimodal data into a pre-trained state classification model to obtain the cognitive state category information of the test subject", it may specifically include the following steps:

[0109] Multimodal data is input into a pre-trained state classification model to obtain the number of sub-cognitive states that the test subject has reached the target cognitive state, and / or the cognitive state category of at least one sub-cognitive state, where the target cognitive state is one type of cognitive state category.

[0110] Specifically, when a cognitive state includes at least one sub-cognitive state, the cognitive state category information of the test subject can actually include which sub-cognitive states of the test subject have reached the target cognitive state, i.e., the number of sub-cognitive states that have reached the target cognitive state, or the cognitive state category of the aforementioned at least one sub-cognitive state, i.e., the category in which each specific sub-cognitive state belongs. Either or a combination of the above two pieces of information can serve as the cognitive state category information of the test subject. The cognitive state category of a sub-cognitive state can also include non-target cognitive states in addition to the target cognitive state, or it can be divided into multiple cognitive state levels with finer granularity, such as high-level, medium-level, and low-level cognitive states. High-level cognitive states belong to the target cognitive state, while medium-level and low-level cognitive states belong to non-target cognitive states. In some embodiments, the proportions of high-level, medium-level, and low-level cognitive states can be 20%, 30%, and 50%, respectively.

[0111] In some embodiments, the multimodal data of the test subjects, as described in the above embodiments, may include at least two data points from at least one category. Therefore, obtaining the data threshold corresponding to the target cognitive state in the above embodiments may specifically involve obtaining the data threshold of at least two data points corresponding to the target cognitive state. In this case, determining the threshold compliance information of the multimodal data based on the multimodal data and the data threshold in the above embodiments may specifically include:

[0112] The number of data points reaching the data threshold is determined based on multimodal data and the data threshold, and / or, the order of magnitude of at least two data points.

[0113] Specifically, the data threshold in this embodiment is determined based on the target cognitive state. If the data reaches the perception data threshold, it means that the cognitive state of the test subject will also reach the target cognitive state. In this embodiment, the target cognitive state may correspond to multiple data items, such as four data items. If all four data items reach the corresponding data threshold, it means that the cognitive state of the test subject has reached the target cognitive state. If three data items reach the corresponding data threshold and one data item does not reach the corresponding data threshold, it can be determined that the test subject has not reached the target cognitive state based on the threshold achievement information of the number of data items that reach the data threshold. In addition, the magnitude of each data item can be set. Specifically, for the data threshold of the data item that has been acquired, the magnitude can be used to characterize the relationship between the data item and the data threshold, such as the ratio between the data item and the data threshold, or the difference between the data item and the data threshold. The ratio can be 80%, 60%, or 20% of the data threshold, etc., thereby characterizing the difference between the two. This difference can reflect the physiological state and environmental state of the test subject, and the difference between them and the ideal physiological state and ideal environmental state that can achieve the target cognitive state. In this embodiment of the application, any or all of the above-mentioned number or magnitude can be included in the threshold attainment information and used as information to determine the current cognitive state of the test subject.

[0114] Based on the above embodiments, state feedback information can be generated according to the number of sub-cognitive states that the test subject has reached the target cognitive state, and / or the cognitive state category of the at least one sub-cognitive state; or, state feedback information can be generated according to the number of data reaching the data threshold, and / or the magnitude of the at least two data points. This also allows the generated state feedback information to match the test subject's current cognitive state, thereby motivating the test subject and providing a positive feedback mechanism to guide the test subject to actively improve their physiological state. When a sub-cognitive state is in a non-target cognitive state, the subject should actively adjust to the target cognitive state; if a sub-cognitive state is already in the target cognitive state, it should be maintained at the target cognitive state.

[0115] In the above embodiments of this application, it has been mentioned that it is necessary to establish a correspondence between multimodal data and cognitive states. Multimodal data may include multiple data items, and cognitive states may include multiple sub-cognitive states. In some embodiments, for a certain sub-cognitive state, the associated data may be several items from all data items. For example, for a certain sub-cognitive state A, the associated data may include data A, data B, and data C; for a certain sub-cognitive state B, the associated data may include data B, data D, and data F, etc. For details, please refer to Table 1 below. For other sub-cognitive states, such as sub-cognitive states C and D, the relationship can also be established in Table 1.

[0116] Sub-cognitive state A Data A, Data B, and Data C Sub-cognitive state B Data B, Data D, and Data F Sub-cognitive state C …… Sub-cognitive state D ……

[0117] Table 1

[0118] Based on the table above, if a sub-cognitive state A reaches the target cognitive state, then data A, B, and C all reach their corresponding data thresholds. Similarly, if sub-cognitive state B reaches the target cognitive state, then data B, D, and F all reach their data thresholds. By accumulating data generated during historical training, the data thresholds for each data point corresponding to reaching the target cognitive state can be determined. During cognitive ability training, once the specific values ​​of each data point are known, the category of each sub-cognitive state can be predicted based on whether the data reaches the corresponding data threshold. This includes identifying which sub-cognitive states have reached the target cognitive state and which are not. Furthermore, for data points that do not reach their thresholds, their magnitude can be determined, ultimately leading to the determination of threshold attainment information.

[0119] In the technical solutions provided by the above embodiments of this application, it is already possible to predict the current cognitive state information of the test subject based on the acquired multimodal data, and then generate state feedback information based on the current cognitive state information of the test subject. The above solution can be implemented by a human factor intelligent cognitive ability training device, which can be set in a hardware device, such as a computer or a server. In addition, in the implementation solution of the embodiments of this application, one or more of a signal acquisition device, an information feedback device, or a stimulation device can be set, and the above devices assist the cognitive ability training device in realizing the technical solution of human factor intelligent cognitive ability training.

[0120] In the above embodiments of this application, the cognitive state of the test subject will vary under different testing environments. The target cognitive state can be considered as the test subject being in an optimal cognitive state. Therefore, the target cognitive state can vary depending on the environment. Thus, the target cognitive state can be redefined in each environment the test subject is in. In the above embodiments of this application, the current cognitive state information of the test subject has already been determined in step 102, which can directly reflect the test subject's cognitive state. If the current cognitive state information of the test subject has been obtained multiple times, the cognitive states of multiple test subjects can be obtained, thereby obtaining the target working state of the test subject. Specifically, Figure 2 This is a flowchart illustrating another human-centered intelligent cognitive ability training method in the embodiments of this application, which is based on the above... Figure 1 The illustrated embodiment may further include a step of determining the target's cognitive state:

[0121] Step 100: Determine the target cognitive state based on the current cognitive state information of the test subject predicted at least twice in the current environment.

[0122] This step can be executed after step 102, and to obtain at least two prediction results, it can be executed multiple times after steps 101 and 102. During the specific execution process, based on at least two predictions of the test subject's current cognitive state information in the current environment, at least two cognitive states of the test subject can be determined. Each prediction of the test subject's current cognitive state information corresponds to one cognitive state, and the best state among the at least two cognitive states is selected as the target cognitive state.

[0123] The above steps for determining the target cognitive state require accumulating a certain number of prediction results. Therefore, the above steps for determining the target cognitive state can be performed when the number of predictions reaches a preset threshold or when a preset time interval is reached. For example, the preset threshold can be set to 10 times, and the preset time interval can be set to 2 minutes.

[0124] Furthermore, even under the same testing environment, the cognitive state of the test subjects may change over time. Therefore, under the same testing environment, certain target cognitive state update conditions can be set. These update conditions can be when the number of predictions reaches a preset threshold or a preset time interval. In this case, the preset threshold can be set to 100 times, and the preset time interval can be set to 10 minutes, etc. Even if the cognitive state of the test subjects changes, the target cognitive state can be adjusted accordingly.

[0125] In the embodiments described above, where the current cognitive state information is threshold attainment information, a target cognitive state and its corresponding data threshold need to be determined before the current cognitive state information can be judged. This can be achieved by first preseting an initial target cognitive state in each test environment, then determining a new target cognitive state according to the scheme described above, and replacing the initial target cognitive state with the new target cognitive state. In the embodiments described above, by using the optimal cognitive state achieved in the current environment as the target cognitive state, compared to setting a fixed target cognitive state, it is clearly more adaptable to the impact of changes in the test scenario on the cognitive state of the personnel, and can more accurately guide the test subjects to reach or maintain a target cognitive state adapted to the current environment.

[0126] Figure 3 This is a flowchart illustrating another human-centered intelligent cognitive ability training method in an embodiment of this application, such as... Figure 3 As shown, it includes the above-mentioned Figure 1 In addition to steps 101-103 in the illustrated embodiment, the following may also be included:

[0127] Step 104: Send status feedback information to the information feedback device so that the incentive content corresponding to the status feedback information can be displayed on the information feedback device.

[0128] If, as described in the above embodiments, the use of the aforementioned status feedback information can achieve the effect of motivating the test subjects, thereby providing a positive feedback mechanism to guide the test subjects to actively improve their physiological state, this step involves setting up a specific information feedback device to display the aforementioned status feedback information on the information feedback device, so as to achieve a positive motivational effect.

[0129] In some specific embodiments, the content of the aforementioned state feedback information includes, but is not limited to, one or a combination of games, music, or animation. For example, when using feedback animation, different games or animations can be presented based on specific cognitive state category information or threshold achievement information. When the test subject reaches the target cognitive state, the game or animation provides a more targeted incentive based on certain affirmation or reward, thereby forming a positive feedback mechanism.

[0130] For example, in an animation of the tortoise and the hare race, the tortoise can be designed to move towards the target at a fixed, slow speed, while the hare is controlled by state feedback information as described in this embodiment. If the state feedback information is generated based on threshold achievement information, the number of data points reaching the threshold in the threshold achievement information can be used to add various clothing, accessories, and other decorations to the hare. The speed of the hare can be controlled based on the magnitude of the data; the higher the magnitude of the data and the closer it is to the data threshold, the faster the hare runs. This continues until all data points reach their corresponding thresholds, at which point the hare reaches its fastest speed and wins the race. This also means that the test subject's cognitive state has reached the target cognitive state. Through such a feedback mechanism, users can be motivated to actively improve their cognitive state to the target cognitive state or maintain it during training.

[0131] In this embodiment, in addition to using state feedback information to display feedback content to the test subject to encourage the test subject to adjust their physiological state on their own, a stimulation device can also be used to apply stimulation signals to the test subject. The external stimulation signals stimulate the test subject to adjust their physiological state, enabling them to adjust their cognitive state to the target cognitive state or maintain it at the target cognitive state. For details, please refer to... Figure 4 As shown.

[0132] Figure 4 This is a flowchart illustrating another cognitive ability training method in an embodiment of this application, such as... Figure 4 As shown, it includes the above-mentioned Figure 1 In addition to steps 101-103 in the illustrated embodiment, the following may also be included:

[0133] Step 105: Generate stimulation device control information based on the current cognitive state information;

[0134] This step takes into account that external stimuli can also regulate the physiological state of the test subject. After predicting the test subject's current cognitive state information in step 102 above, the subsequent stimulation needs can be determined based on the current cognitive state information, and then stimulation device control information can be generated based on the stimulation needs. Specifically, if the current cognitive state information can be the cognitive state category information or threshold attainment information in the above embodiments, stimulation device control information corresponding to the above information is generated in this step. This stimulation device control information can control the stimulation device to emit stimulation signals. If the current cognitive state information indicates that the test subject has not reached the target cognitive state, the generated stimulation device control information can control the stimulation device to emit stimulation signals that stimulate the test subject's physiological state, so as to prompt the test subject to adjust their physiological state to achieve the target cognitive state.

[0135] Step 106: Send stimulation control information to the stimulation device to control the stimulation device to apply stimulation signals to the test subject, so that the test subject adjusts the cognitive state to the target cognitive state or maintains the target cognitive state.

[0136] After generating the stimulation device control information in the previous step, the stimulation device can be controlled to apply stimulation signals to the test subject according to the control information. The stimulation signals can be directly applied to the human body so that the test subject can adjust their cognitive state to the target cognitive state or maintain the target cognitive state.

[0137] In some embodiments, the stimulation signal may include at least one of an optical signal, an electrical signal, or a magnetic signal. For example, if research reveals that a magnetic signal of a certain intensity can adjust the physiological state of a test subject and ultimately improve their cognitive state, then the stimulation device control information may be control information for controlling the magnetic signal emitting device, and may control the magnetic signal device to emit an optical signal that meets the intensity requirements, thereby adjusting the physiological state of the test subject and ultimately improving their cognitive state.

[0138] Figure 5 This is a flowchart illustrating a cognitive ability training method as described in an embodiment of this application. Figure 5 As shown, it includes the above-mentioned Figure 1 In addition to steps 101-103 in the illustrated embodiment, the following may also be included:

[0139] Step 107: Generate environmental control information based on the current cognitive state information;

[0140] This step takes into account the environmental data of the test subject's environment, which can also stimulate changes in the test subject's physiological state. After predicting the test subject's current cognitive state information in step 102 above, the subsequent environmental adjustment needs can be determined based on the current cognitive state information, and then environmental control information can be generated based on the environmental adjustment needs. Specifically, if the current cognitive state information can be the cognitive state category information or threshold attainment information in the above embodiments, environmental control information corresponding to the above information is generated in this step. This environmental control information can control the environmental control equipment to adjust the environmental data. If the current cognitive state information indicates that the test subject has not reached the target cognitive state, the generated environmental control information can control the environmental control equipment to adjust the environmental data, which can stimulate the test subject's physiological state to adjust the test subject's physiological state and achieve the target cognitive state.

[0141] Step 108: Send environmental control information to the environmental control device to control the environmental control device to adjust the environmental data of the environment in which the test subject is located, so that the cognitive state of the test subject is adjusted to the target cognitive state or maintained in the target cognitive state.

[0142] After generating environmental control information in the previous step, this environmental control information mainly affects the environment in which the test subject is located. Different environmental data have a certain impact on the physiological and cognitive states of the test subject. Therefore, in this embodiment, the environmental control device can be controlled to adjust the environmental data of the environment in which the test subject is located, so that the cognitive state of the test subject is adjusted to the target cognitive state or maintained at the target cognitive state.

[0143] In some embodiments, the aforementioned environmental adjustment data may include at least one of temperature adjustment data, humidity data, brightness data, or noise data. For example, if research reveals that a specific temperature or brightness in the environment can adjust the physiological state of the test subject and ultimately improve their cognitive state, then the aforementioned environmental control information may be control information for controlling environmental control devices, and may control the environmental control devices to adjust the temperature or brightness, thereby adjusting the physiological state of the test subject and ultimately improving their cognitive state.

[0144] This application also provides a method for training a state classification model. This state classification model can be applied to the above embodiments. It obtains the cognitive state category information of the test subject based on the multimodal data of the test subject. When training the state classification model, it can obtain sample data for the test subject, that is, the multimodal data generated by the test subject in the early stage and its corresponding cognitive state category information, and then train the state classification model based on these sample data.

[0145] Figure 6 This is a flowchart illustrating a training method for a state classification model in an embodiment of this application, as shown below. Figure 6 As shown, it includes the following steps:

[0146] Step 601: Obtain sample data, which includes multimodal data of the test subjects and cognitive state category information of the test subjects corresponding to the multimodal data. The multimodal data includes at least two data from at least one category of the test subjects' human body data, environmental data of the environment in which the test subjects are located, and interaction data between the test subjects and their environment.

[0147] The state classification model in this embodiment distinguishes between different testers, and different testers correspond to different state classification models. The step of acquiring sample data in this section refers to acquiring sample data for each tester separately. Therefore, during the sample data collection process for each tester, multimodal data of that tester during the testing process, as well as the corresponding cognitive state category information, can be collected simultaneously.

[0148] The collection of multimodal data can involve acquiring sample data from various types of signal acquisition devices. For example, physiological signal acquisition devices can collect human body data of test subjects. In some embodiments, this human body data includes, but is not limited to, at least one of the following: electroencephalogram (EEG) signal data, electrocardiogram (ECG) signal data, electrodermal conductance (EDA) signal data, electromyography (EMG) signal data, body temperature data, pulse data, respiratory rate data, near-infrared spectral data, heart rate data, and blood oxygen saturation data. Another example is environmental data collected by environmental data acquisition devices, where the environment can be determined according to the application scenario. Environmental data can refer to the environmental data of the test subject's current environment, such as temperature data, humidity data, brightness data, and noise data. Yet another example is interactive data acquisition devices that collect data on the interaction between the test subject and their environment. For instance, in a driving test scenario where the environment is a simulated cockpit, the interactive data could include data from the test subject's operation of the steering wheel, brakes, etc. In this application embodiment, "multimodal" refers to at least two types of data collected by signal acquisition devices, which can include one or more categories of data. Specifically, the collected data can be preprocessed. For example, the human body data of the test subjects sent by multiple physiological signal acquisition devices during the test can be preprocessed, such as by filtering and noise reduction. Then, the preprocessed human body data can be extracted in the time domain, frequency domain, spatial domain, nonlinearity and network features to finally obtain the human body data of the test subjects.

[0149] The collection of cognitive state category information can involve collecting behavioral response data corresponding to the aforementioned multimodal data. This behavioral response data includes, but is not limited to, one or more parameters such as accuracy, reaction speed, false positive rate, false negative rate, and correct rejection rate. The cognitive state category information of the test subject is then determined based on this behavioral response data. For example, one or more of the aforementioned parameters can be formatted according to a predetermined format to ultimately determine the cognitive state category information of the test subject. Specifically, cognitive states can be categorized based on the specific values ​​of the aforementioned parameters. For example, cognitive state categories can be divided into target cognitive states and non-target cognitive states, or further, cognitive states can be divided into different levels, such as high-level, medium-level, and low-level cognitive states. High-level cognitive states can correspond to target cognitive states, while medium-level and low-level cognitive states can correspond to non-target cognitive states.

[0150] In some embodiments, the cognitive state includes at least one sub-cognitive state among attention state, memory state, executive state, and divergent thinking state. In this case, determining the cognitive state category information of the test subject based on behavioral response data may specifically include: determining the cognitive state category information of each sub-cognitive state of the test subject based on behavioral response data, that is, distinguishing the cognitive state category information of each sub-cognitive state. In this case, the trained state classification model can also be used to obtain the cognitive state category information of each sub-cognitive state.

[0151] After acquiring the aforementioned multimodal data and cognitive state category information, the cognitive state category information corresponding to each multimodal data can be obtained based on the temporal information, thereby forming a sample dataset. This sample dataset can be divided into a training set, a validation set, and a test set as needed.

[0152] Step 602: Based on the above sample data, train the state classification model to obtain a pre-trained state classification model.

[0153] This step utilizes the sample data from the training set obtained in step 601 above to train the state classification model, thus obtaining a pre-trained state classification model. Specifically, during training, the parameters in the state classification model are updated using each multimodal data point in the training set and its corresponding cognitive state category information. This can be achieved through iterative updates of the parameters using optimization algorithms, such as gradient descent, which minimizes prediction errors or maximizes classification performance. When the number of training iterations reaches a preset threshold, the training of the state classification model using the training set is complete. Subsequently, validation and test sets can be used to validate and test the state classification model to obtain the optimal state classification model.

[0154] In the above embodiments of this application, the state classification model corresponds to each test subject and is strongly correlated with the environmental state of the test subject. Therefore, the best sample data should be the sample data collected in the current environment. However, this may result in insufficient storage of samples. Therefore, in the training scheme of the state classification model provided in the embodiments of this application, the training process can be divided into two processes: incremental training and online learning.

[0155] Specifically, the sample data obtained in step 601 above may include historical sample data and current environment sample data. The current environment sample data is the sample data collected in an environment that is strongly correlated with the environmental state of the test subject. The historical sample data is the sample data collected within a time period set before the current time period. Although it differs from the current environment, considering that it is the same test subject, the correspondence between the physiological state and cognitive state is also stable, so the historical sample data also has a certain reference value. Therefore, in this embodiment of the application, historical sample data is used for incremental training, while current environment sample data is used for online training.

[0156] In this case, training the state classification model based on the sample data in step 602 above can include:

[0157] Based on historical sample data, the classification state model is initially trained to obtain the initially trained state classification model.

[0158] Based on the sample data of the current environment, the initially trained state classification model is optimized to obtain the optimized state classification model.

[0159] In this approach, historical sample data and current environmental sample data are both incorporated into the training of the state classification model. Incremental training and online training are used to update the parameters in the state classification model, which can better adapt to the physiological state of the test subjects in the current environment. At the same time, due to the increase in the number of samples, the classification performance of the state classification model can be improved, and the risk of overfitting can be reduced.

[0160] Corresponding to the above method embodiments, this application also provides a cognitive ability training device. Figure 7 This is a schematic diagram of the structure of a cognitive ability training device according to an embodiment of this application, such as... Figure 7As shown, the system includes a data acquisition module 11, a prediction module 12, and a feedback generation module 13. The data acquisition module 11 is used to acquire multimodal data of the test subject. The multimodal data includes at least two data items from at least one category: human body data of the test subject, environmental data of the environment in which the test subject is located, and interaction data between the test subject and the environment. The prediction module 12 is used to predict the current cognitive state information of the test subject based on the multimodal data. The feedback generation module 13 is used to generate state feedback information based on the current cognitive state information of the test subject. This state feedback information is used to guide the test subject to adjust their cognitive state to a target cognitive state or maintain it at the target cognitive state.

[0161] In some embodiments, the prediction module 12 may include:

[0162] The first acquisition unit is used to acquire cognitive state category information of the test subject based on multimodal data; or,

[0163] The second acquisition unit is used to acquire the perception data threshold corresponding to the target cognitive state, and to determine the threshold compliance information of the multimodal data based on the multimodal data and the perception data threshold.

[0164] In some embodiments, the first acquisition unit described above is specifically used to input multimodal data into a pre-trained state classification model to obtain cognitive state category information of the test subject.

[0165] In some embodiments, the cognitive state includes at least one sub-cognitive state among attention state, memory state, executive state and divergent thinking state, and each sub-cognitive state is respectively set with a corresponding target cognitive state.

[0166] The first acquisition unit described above is specifically used to input multimodal data into a pre-trained state classification model to obtain the number of sub-cognitive states in which the test subject has reached the target cognitive state, and / or the cognitive state category of at least one sub-cognitive state.

[0167] In some embodiments, the human body data of the test subject includes at least one of the following: electroencephalogram (EEG) signal data, electrocardiogram (ECG) signal data, electrodermal conductance (TEA) signal data, electromyography (EMG) signal data, body temperature data, pulse data, respiratory rate data, near-infrared data, heart rate data, and blood oxygen content data; and the environmental data of the environment in which the test subject is located includes at least one of the following: temperature data, humidity data, brightness data, and noise data.

[0168] The second acquisition unit described above is specifically used for:

[0169] Obtain data thresholds for at least two data points corresponding to the target cognitive state;

[0170] The number of data points reaching the data threshold is determined based on multimodal data and the data threshold, and / or, the order of magnitude of at least two data points.

[0171] In some embodiments, the feedback generation module 13 is specifically used to generate state feedback information based on cognitive state category information or threshold attainment information.

[0172] In some embodiments, the data acquisition module 11 is specifically used to receive multimodal data of the test subject sent by the signal acquisition device.

[0173] In some embodiments, it also includes:

[0174] The target determination module is used to determine the target cognitive state based on the test subject's current cognitive state information predicted at least twice in the current environment.

[0175] In some embodiments, the target determination module described above is specifically used to determine the target cognitive state based on at least two predictions of the test subject's current cognitive state information in the current environment when the number of predictions of the test subject's current cognitive state information in the current environment reaches a preset number threshold or a preset time interval.

[0176] In some embodiments, the target determination module includes:

[0177] A state determination unit is used to determine at least two cognitive states of the test subject based on at least two predictions of the test subject's current cognitive state information in the current environment.

[0178] The goal determination unit is used to select the best state from at least two cognitive states as the target cognitive state.

[0179] In some embodiments, it also includes:

[0180] The first information sending module is used to send status feedback information to the information feedback device so that the status feedback information can be fed back on the information feedback device.

[0181] In some embodiments, the status feedback information includes at least one of visual feedback information, auditory feedback information, and tactile feedback information.

[0182] In some embodiments, it also includes:

[0183] The information generation module is used to generate stimulus control information based on the current cognitive state information;

[0184] The second information sending module is used to send the stimulation control information to the stimulation device to control the stimulation device to apply stimulation signals to the test subject, so that the test subject adjusts the cognitive state to the target cognitive state or maintains the target cognitive state.

[0185] In some embodiments, the stimulation signal includes at least one of an optical signal, an electrical signal, or a magnetic signal.

[0186] In some embodiments, it also includes:

[0187] The environmental control information generation module is used to generate environmental control information based on the current cognitive state information;

[0188] The third information sending module is used to send environmental control information to the environmental control equipment to control the environmental control equipment to adjust the environmental data of the environment in which the test subject is located, so that the cognitive state of the test subject is adjusted to the target cognitive state or maintained at the target cognitive state.

[0189] This application provides a human cognitive ability training device, which is related to the above-mentioned... Figure 1-5 Corresponding to any of the method embodiments shown, the above methods can be executed and the corresponding technical effects can be obtained. The embodiments in this application will not be described in detail, but the specific methods and the technical effects achieved can be referred to the above embodiments.

[0190] This application also provides a human-centric intelligent cognitive ability training system. Figure 8 This is a schematic diagram of the structure of a human-centered intelligent cognitive ability training system according to an embodiment of this application, such as... Figure 8 As shown, the human cognitive ability training system includes the cognitive ability training device 21 in the above embodiment, and may also include a signal acquisition device 22. The signal acquisition device 22 is used to collect the multimodal data of the test subject and send the multimodal data of the test subject to the cognitive ability training device 21.

[0191] In some embodiments, the signal acquisition device 22 described above may include multiple types of sub-devices, each of which is used to acquire one or more data of a certain category in multimodal data.

[0192] In some embodiments, the cognitive ability training device 21 described above can be implemented by hardware or software. If it is implemented by software, it can be set up on a computer or server. The computer or server can provide corresponding data transmission and reception ports for the exchange of relevant data or information.

[0193] In some embodiments, the human cognitive ability training system may further include:

[0194] Information feedback device 23 is used to receive status feedback information sent by the human factor intelligent cognitive ability training device and to provide feedback on the status feedback information.

[0195] In some embodiments, the human cognitive ability training system may further include:

[0196] Stimulation device 24 is used to apply stimulation signals to the test subject according to stimulation control information, so that the test subject adjusts the cognitive state to the target cognitive state or maintains the target cognitive state.

[0197] In some embodiments, the human cognitive ability training system may further include:

[0198] The environmental control device 25 is used to adjust the environmental data of the environment in which the test subject is located based on the environmental control information, so that the cognitive state of the test subject is adjusted to the target cognitive state or maintained at the target cognitive state.

[0199] This application provides a human-based intelligent cognitive ability training system, which corresponds to the above-described method embodiments. It can execute the above methods and achieve the corresponding technical effects. This application will not describe the embodiments in detail, but the specific methods and achieved technical effects can be referred to the above embodiments.

[0200] This application also provides a training device for a state classification model. Figure 9 This is a schematic diagram of the structure of a training device for a state classification model in an embodiment of this application, as shown below. Figure 9 As shown, the device includes:

[0201] The sample acquisition module 31 is used to acquire sample data, which includes multimodal data of the test subjects and cognitive state category information of the test subjects corresponding to the multimodal data. The multimodal data includes at least two data from at least one category of human body data of the test subjects, environmental data of the environment in which the test subjects are located, and interaction data between the test subjects and the environment.

[0202] Training module 32 is used to train the state classification model based on sample data to obtain a pre-trained state classification model.

[0203] In some embodiments, the sample acquisition module 31 is specifically used for:

[0204] Obtain behavioral response data of test subjects corresponding to multimodal data;

[0205] The cognitive state category information of the test subjects is determined based on behavioral response data.

[0206] In some embodiments, the cognitive state includes at least one sub-cognitive state selected from attention state, memory state, executive function state, and divergent thinking state. The sample acquisition module 31 is specifically used for:

[0207] The cognitive state category information for each sub-cognitive state of the test subject is determined based on the behavioral response data.

[0208] In some embodiments, sample data includes historical sample data and current environmental sample data;

[0209] Training module 32 is specifically used for:

[0210] Based on historical sample data, the classification state model is initially trained to obtain the initially trained state classification model.

[0211] Based on the sample data of the current environment, the initially trained state classification model is optimized to obtain the optimized state classification model.

[0212] This application provides a training device for a state classification model, which corresponds to the training embodiment of the state classification model described above. It can execute the above method and achieve the corresponding technical effects. This application will not describe the embodiment in detail, but the specific method and the achieved technical effects can be referred to the above embodiment.

[0213] The following is a detailed reference. Figure 10 This document illustrates a structural diagram suitable for implementing the electronic device 800 in the embodiments of this application. The electronic device 800 in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. The illustrated electronic device is merely an example and should not impose any limitations on the functionality or scope of the embodiments of this application.

[0214] like Figure 10 As shown, the electronic device 800 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0215] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although an electronic device 800 with various devices is shown, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0216] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, it performs the functions defined in the human-computer interaction intelligent cognitive training method or the state classification model training method of the embodiments of this application.

[0217] It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0218] In some implementations, the client and server can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0219] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0220] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to:

[0221] Acquire multimodal data of the test subject, which includes at least two data items from at least one category: human body data of the test subject, environmental data of the environment in which the test subject is located, and interaction data between the test subject and the environment.

[0222] Predict the current cognitive state information of the test subjects based on multimodal data;

[0223] Status feedback information is generated based on the test subject's current cognitive state information. This status feedback information is used to guide the test subject to adjust their cognitive state to the target cognitive state or maintain it at the target cognitive state.

[0224] Alternatively, sample data can be obtained, including multimodal data of the test subjects and cognitive state category information of the test subjects corresponding to the multimodal data. The multimodal data includes at least two data from at least one category of the test subjects' human body data, environmental data of the environment in which the test subjects are located, and interaction data between the test subjects and their environment.

[0225] Based on the sample data, the state classification model is trained to obtain a pre-trained state classification model.

[0226] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0227] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0228] The units described in the embodiments of this application can be implemented in software or hardware. The names of the units are not, in some cases, limiting the scope of the unit itself.

[0229] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0230] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0231] According to one or more embodiments of this application, this application provides a computer-readable storage medium storing a computer program for executing any of the human-caused intelligent cognitive training methods or state classification model training methods provided in this application.

[0232] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

[0233] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Multitasking and parallel processing may be advantageous in certain environments. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0234] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for training human cognitive abilities, characterized in that, include: Acquire multimodal data of the test subject, wherein the multimodal data includes at least two data items from at least one category of the test subject's human body data, the environmental data of the environment in which the test subject is located, and the interaction data between the test subject and the environment. Predict the current cognitive state information of the test subject based on the multimodal data; Based on the current cognitive state information of the test subject, state feedback information is generated. The state feedback information is used to guide the test subject to adjust their cognitive state to the target cognitive state or maintain the target cognitive state.

2. The method according to claim 1, characterized in that, The prediction of the test subject's current cognitive state information based on multimodal data includes: Based on the multimodal data, obtain the cognitive state category information of the test subjects; or, Obtain the data threshold corresponding to the target cognitive state, and determine the threshold compliance information of the multimodal data based on the multimodal data and the data threshold.

3. The method according to claim 2, characterized in that, The step of obtaining cognitive state category information of the test subjects based on multimodal data includes: The multimodal data is input into a pre-trained state classification model to obtain the cognitive state category information of the test subjects.

4. The method according to claim 3, characterized in that, The cognitive state includes at least one of the following sub-cognitive states: attention state, memory state, executive state, and divergent thinking state, and each sub-cognitive state is set with a corresponding target cognitive state. The multimodal data is input into a pre-trained state classification model to obtain the cognitive state category information of the test subject, including: The multimodal data is input into a pre-trained state classification model to obtain the number of sub-cognitive states in which the test subject has reached the target cognitive state, and / or the cognitive state category of the at least one sub-cognitive state.

5. The method according to claim 2, characterized in that, The human body data of the test subject includes at least one of the following: electroencephalogram (EEG) signal data, electrocardiogram (ECG) signal data, electrodermal conductance (TEA) signal data, electromyography (EMG) signal data, body temperature data, pulse data, respiratory rate data, near-infrared data, heart rate data, and blood oxygen content data. The environmental data of the environment in which the test subject is located includes at least one of the following: temperature data, humidity data, brightness data, and noise data. The step of acquiring the data threshold corresponding to the target cognitive state includes: Obtain data thresholds for at least two data points corresponding to the target cognitive state; Determining threshold compliance information for the multimodal data based on the multimodal data and the data threshold includes: The number of data reaching the data threshold is determined based on the multimodal data and the data threshold of the at least two data points, and / or the magnitude of the at least two data points.

6. The method according to claim 2, characterized in that, The step of generating state feedback information based on the test subject's current cognitive state information includes: State feedback information is generated based on the cognitive state category information or the threshold attainment information.

7. The method according to claim 1, characterized in that, Also includes: The target cognitive state is determined based on the test subject's current cognitive state information predicted at least twice in the current environment.

8. The method according to claim 7, characterized in that, When the number of predictions of the test subject's current cognitive state information in the current environment reaches a preset threshold or a preset time interval, the target cognitive state is determined based on at least two predictions of the test subject's current cognitive state information in the current environment.

9. The method according to claim 7, characterized in that, The determination of the target cognitive state based on at least two predictions of the test subject's current cognitive state information in the current environment includes: Based on the current cognitive state information of the test subject predicted at least twice in the current environment, at least two cognitive states of the test subject are determined. Select the best state from the at least two cognitive states as the target cognitive state.

10. The method according to claim 1, characterized in that, The method further includes at least one of the following: The status feedback information is sent to the information feedback device to be fed back on the information feedback device; or, Stimulus control information is generated based on the current cognitive state information; The stimulation control information is sent to the stimulation device to control the stimulation device to apply stimulation signals to the test subject, so that the test subject adjusts his / her cognitive state to the target cognitive state or maintains the target cognitive state; or... Generate environmental control information based on current cognitive state information; Send environmental control information to the environmental control equipment to control the environmental control equipment to adjust the environmental data of the environment in which the test subject is located, so that the cognitive state of the test subject is adjusted to the target cognitive state or maintained at the target cognitive state.

11. A training method for a state classification model, characterized in that, include: Acquire sample data, which includes multimodal data of the test subjects and cognitive state category information of the test subjects corresponding to the multimodal data. The multimodal data includes at least two data items from at least one category of the test subjects' human body data, environmental data of the environment in which the test subjects are located, and interaction data between the test subjects and the environment. The state classification model is trained based on the sample data to obtain a pre-trained state classification model.

12. The method according to claim 11, characterized in that, The step of obtaining the cognitive state category information of the test subject corresponding to the multimodal data includes: Obtain behavioral response data of the test subjects corresponding to the multimodal data; The cognitive state category information of the test subject is determined based on the behavioral response data.

13. The method according to claim 12, characterized in that, The cognitive state includes at least one sub-cognitive state among attention state, memory state, executive function state, and divergent thinking state. Determining the cognitive state category information of the test subject based on the behavioral response data includes: The cognitive state category information for each sub-cognitive state of the test subject is determined based on the behavioral response data.

14. The method according to claim 11, characterized in that, The sample data includes historical sample data and current environmental sample data; The step of training the state classification model based on sample data to obtain a pre-trained state classification model includes: Based on historical sample data, the classification state model is initially trained to obtain the initially trained state classification model. Based on the sample data of the current environment, the initially trained state classification model is optimized to obtain an optimized state classification model.

15. A human cognitive ability training device, characterized in that, include: The data acquisition module is used to acquire multimodal data of the test subject. The multimodal data includes at least two data items from at least one category of the test subject's human body data, environmental data of the environment in which the test subject is located, and interaction data between the test subject and the environment. The prediction module is used to predict the current cognitive state information of the test subject based on the multimodal data; The feedback generation module is used to generate state feedback information based on the current cognitive state information of the test subject. The state feedback information is used to guide the test subject to adjust their cognitive state to the target cognitive state or maintain the target cognitive state.

16. A human-based intelligent cognitive ability training system, characterized in that, The device includes the human-caused intelligent cognitive ability training device as described in claim 15, and a data acquisition device, wherein the data acquisition device is used to collect multimodal data of the test subject and send the multimodal data of the test subject to the cognitive ability training device.

17. The system according to claim 16, characterized in that, It also includes at least one of the following: An information feedback device is used to receive status feedback information sent by the human-cause intelligent cognitive ability training device and to feed back the status feedback information. Stimulation device, used to apply stimulation signals to the test subject according to stimulation control information, so that the test subject adjusts the cognitive state to the target cognitive state or maintains the target cognitive state; An environmental control device is used to adjust the environmental data of the environment in which the test subject is located based on environmental control information, so that the cognitive state of the test subject is adjusted to the target cognitive state or maintained at the target cognitive state.

18. A training device for a state classification model, characterized in that, include: The sample acquisition module is used to acquire sample data, which includes multimodal data of the test subject and cognitive state category information of the test subject corresponding to the multimodal data. The multimodal data includes at least two data items from at least one category of the test subject's human body data, environmental data of the environment in which the test subject is located, and interaction data between the test subject and the environment. The training module is used to train the state classification model based on the sample data to obtain a pre-trained state classification model.

19. An electronic device, characterized in that, It includes a processor and a memory, the processor being used to execute a computer program in the memory, the computer program being capable of executing the human cognitive ability training method according to any one of claims 1-10 or the state classification model training method according to any one of claims 11-14.

20. A computer storage medium, characterized in that, Used to store a computer program capable of executing the human cognitive ability training method according to any one of claims 1-10 or the state classification model training method according to any one of claims 11-14.

Images