Cross-modal fusion intelligent joint training method and system

Abstract

The invention discloses a cross-modal fusion intelligent joint training method and a cross-modal fusion intelligent joint training system. The method comprises the steps of obtaining multi-modal perception data during user training based on a preset initial cognitive training scheme; performing cross-modal coding on the multi-modal perception data to capture the association among cognition, emotion and motion, and obtaining a corresponding association matrix; based on the incidence matrix, biological evolution calculation is carried out through a memory enhancement algorithm and multi-objective evolution optimization; jointly inputting the evolution calculation data and the multi-modal perception data into a preset three-dimensional decision model to output probability distribution of evolution training schemes, and pushing the evolution training scheme with the highest probability; carrying out dynamic optimization on the evolution training scheme, and carrying out neural biological feedback on the user; on the basis of a dynamically optimized evolution training scheme, performing personalized scheme evolution according to a cognitive impairment feature layering result of the user; and repeating the process until the training is finished.

Description

TECHNICAL FIELD

[0001] The present application relates to a cross-modal fusion intelligent joint training method, and also relates to a corresponding intelligent joint training system, and belongs to the technical field of cognitive training. BACKGROUND

[0002] Cognitive training is an effective means to delay individual cognitive decline. However, to achieve good cognitive improvement effect, individuals need to continue training for at least three months or even a long time. Therefore, how to stimulate and maintain the interest of individuals in cognitive training so that they can continue to make efforts and actions has become a key problem to be solved in the field of cognitive training task design.

[0003] The traditional cognitive training method is often based on the classic paradigm and uses a unified difficulty cognitive task for training. However, due to the huge cognitive level difference between individuals, this training method has obvious defects. For some individuals, too low task difficulty may lead them to gradually lose interest in training, reduce the degree of attention to training, and thus reduce the frequency of training; and for other individuals, too high task difficulty may make them feel great pressure, damage their training confidence, and ultimately reduce their willingness to persist in cognitive training. Therefore, in order to better meet the needs of different individuals, it is necessary to develop a personalized cognitive training system that can target individual differences. Such a system can dynamically adjust the task difficulty in training through analysis of the cognitive task difficulty gradient, and realize precise adjustment of the task difficulty according to the real-time state of the user himself.

[0004] At present, most existing cognitive training systems only focus on the physiological and cognitive aspects, ignoring the emotional stability and motor coordination, which are also important factors. In particular, for elderly patients who have both cognitive and motor dysfunction, such training systems that ignore emotions and movement have obvious limitations in rehabilitation training effect. SUMMARY

[0005] The primary technical problem to be solved by the present application is to provide a cross-modal fusion intelligent joint training method.

[0006] Another technical problem to be solved by the present application is to provide a cross-modal fusion intelligent joint training system.

[0007] To achieve the above technical purposes, the present application adopts the following technical solutions:

[0008] According to a first aspect of an embodiment of the present application, a cross-modal fusion intelligent joint training method is provided, comprising the following steps:

[0009] acquire multi-modal perception data of the user during the cognitive training process based on the preset initial cognitive training scheme; wherein the multi-modal perception data at least includes cognitive level perception data, emotion level perception data, motion level perception data and physiological level perception data;

[0010] cross-modal encode the multi-modal perception data to capture the correlation among cognition-emotion-motion and acquire a corresponding correlation matrix;

[0011] based on the correlation matrix, biological evolution algorithm is performed through memory reinforcement algorithm and multi-objective evolutionary optimization to acquire evolution algorithm data;

[0012] input the evolution algorithm data and the multi-modal perception data into a preset three-dimensional decision model to output a probability distribution of the evolution training scheme; wherein the three-dimensional decision model is constructed based on multi-modal perception data of a plurality of historical users, and is preset with a plurality of cognitive load-emotion awareness-motion coordination three-dimensional coordinated evolution evolution training schemes;

[0013] based on the probability distribution of the cognitive training scheme, the user is pushed to the evolution training scheme with the highest probability;

[0014] acquire task execution data and multi-modal perception data of the user completing the evolution training scheme to dynamically optimize the evolution training scheme and provide neurobiological feedback to the user; wherein the dynamic optimization at least includes cognitive, emotional, motion adjustment task difficulty, task type and task training time, and the neurobiological feedback at least includes displaying current training data and cognitive training results to the user;

[0015] based on the dynamically optimized evolution training scheme, feature layering is performed according to the cognitive impairment characteristics of the user to adjust the training weights of the three dimensions of cognitive load-emotion awareness-motion coordination, thereby forming a personalized cognitive training scheme for the user;

[0016] acquire multi-modal perception data of the user based on the personalized cognitive training scheme, and repeat the above process until the training is completed.

[0017] wherein preferably, the multi-modal perception data is acquired by the following method:

[0018] at the cognitive level, task response time and accuracy are used to calculate cognitive efficiency index Ccog; α+β=1, and the weight α is dynamically adjusted by the task type;

[0019] On the emotion level, based on the facial action unit coding system, the micro-expression time sequence features are extracted through a 3D convolution network, and emotion recognition is performed on the micro-expression time sequence features to output a six-dimensional emotion vector E, E=[e joy , e sad , e anger , e fear , e surprise , e neutral ];

[0020] On the motion level, the joint acceleration and Euler angle are captured by using an inertial measurement unit, after denoising by low-pass filtering, the coordinate sequence of multiple key points is extracted by using a three-dimensional skeletal pose reconstruction algorithm;

[0021] On the physiological level, the heart rate variability HRV is calculated by using the time domain standard deviation method, RR represents the heartbeat interval; N represents the total number of valid RR intervals in the analysis time period; i represents the i th RR interval; represents the average value of all RR intervals.

[0022] Among them, the multi-modal perception data is cross-modally encoded to capture the correlation between cognition-emotion-motion and obtain the corresponding correlation matrix, including:

[0023] The multi-modal perception data is cross-modally semantically aligned to obtain the corresponding encoded data of each level;

[0024] The corresponding encoded data of each level is mapped to the same representation space;

[0025] The correlation matrix corresponding to the cognition-emotion-motion interaction is calculated using a multi-head attention mechanism:

[0026]

[0027] Among them, Q is the cognitive behavior encoding data in the representation space, K is the emotion encoding data in the representation space, and V is the motion encoding data in the representation space; T represents the time sequence.

[0028] Among them, a memory reinforcement algorithm is used to combine reinforcement learning and an external memory mechanism, a readable and writable external memory module is introduced for structured storage of historical experience, to realize fast retrieval and reasoning; Specifically, the following sub-steps are included:

[0029] A double-delay deep deterministic policy gradient network is used, and the target network parameter update formula is: θ'←τθ+(1-τ)θ', τ=0.005;

[0030] The sample priority p i in the priority experience replay = |δi |+∈,;where δi is the TD error, ε=1e-6.

[0031] Preferably, the multi-objective evolutionary optimization specifically includes:

[0032] Define a fitness function F to output a combination that maximizes the efficiency of the user's physical training by adjusting the training weights of each dimension.

[0033] F=w1*(1-CognitiveLoad)+w2*EmoStability+w3*

[0034] Motor Smoothness;

[0035] Among them, the larger the value of F, the greater the physical training efficiency; w1, w2, and w3 represent the training weights of the three dimensions of cognitive load, emotional awareness, and motor coordination, respectively; CognitiveLoad represents the cognitive load parameter; EmoStability represents the emotional stability parameter; and MotorSmoothness represents the motor fluency parameter.

[0036] Preferably, the feature stratification includes at least: mild / moderate / severe cognitive impairment, mild / moderate / severe motor impairment, and mild / moderate / severe mood disorder;

[0037] For each feature stratification, personalized adjustments are made based on the principle of "training all three aspects and prioritizing training those with higher levels of difficulty".

[0038] Preferably, based on the user's heart rate variability, biological feedback is provided to the user's current training state, with the aim of enabling the user to further regulate their autonomic nervous system based on the biological feedback and achieve effective improvement in their cognitive, motor, and emotional states.

[0039] According to a second aspect of the present invention, an intelligent joint training system applying the above-described intelligent joint training method is provided, comprising:

[0040] A multimodal perception unit is used to perform multimodal perception on the user during cognitive training to obtain multimodal perception data.

[0041] A cross-modal coding unit, connected to the multimodal sensing unit, is used to perform cross-modal coding on the multimodal sensing data, thereby capturing the correlation between cognition, emotion, and movement, and obtaining the corresponding correlation matrix.

[0042] The biological evolution calculation unit is connected to the cross-modal coding unit to perform biological evolution calculations based on the correlation matrix using a memory reinforcement algorithm and multi-objective evolutionary optimization, thereby obtaining evolution calculation data.

[0043] The decision-making unit is pre-configured with a three-dimensional decision-making model and is connected to the biological evolution calculation unit and the multimodal perception unit to output the probability distribution of the evolutionary training scheme based on the evolutionary calculation data and the multimodal perception data.

[0044] A neurobiofeedback unit, connected to the decision unit, is used to collect task execution data and multimodal perception data of the user completing the evolutionary training program, so as to dynamically optimize the evolutionary training program and provide neurobiofeedback to the user.

[0045] The scheme evolution unit is connected to the neurobiofeedback unit to perform feature stratification based on the user's cognitive impairment characteristics according to the dynamically optimized evolution training scheme, so as to adjust the training weights of the three dimensions of cognitive load, emotional awareness and motor coordination, thereby forming a personalized cognitive training scheme for the user.

[0046] Furthermore, the scheme evolution unit is also connected to the multimodal perception unit, so that the multimodal perception unit can perform multimodal perception of the user based on the updated personalized cognitive training scheme.

[0047] Preferably, the multimodal sensing unit is a wearable device.

[0048] According to a third aspect of the present invention, a cross-modal fusion intelligent joint training system is provided, including a processor and a memory, wherein the processor reads a computer program in the memory for executing the above-described intelligent joint training method.

[0049] Compared with the prior art, the present invention has the following technical effects:

[0050] (1) This invention has made a breakthrough in constructing a cross-modal physiological-behavioral-cognitive dynamic perception system, achieving for the first time real-time synchronous acquisition of four levels of data: cognitive response (RT & ACC behavior), emotional micro-expressions, motion capture, and physiological indicators (respiratory rate, blood oxygen, and blood pressure). By using a wearable multi-source signal synchronization device, the problem of joint data acquisition and evaluation in multimodal assessment systems has been solved.

[0051] (2) This invention presents a pioneering biological evolution-driven three-dimensional adaptive training engine. By integrating memory reinforcement networks and real-time feedback from multimodal perception, a three-dimensional co-evolutionary model of "cognitive load - emotional awareness - motor coordination" is constructed. An innovative Pareto frontier optimization strategy is designed to achieve a balance between training intensity and user tolerance, forming an evolutionary training program.

[0052] (3) The embodiments of the present invention construct a closed-loop neurobiofeedback enhancement system, breaking through the traditional one-way stimulus response mode. By synchronously regulating modalities such as heart rate variability across scales, feedback fusion is performed to accurately match the neural plasticity time window and promote the speed of autonomic nervous system regulation response.

[0053] (4) This invention proposes a barrier feature stratification mechanism to solve the problem of intervention adaptation for people with multiple comorbidities. By dynamically adjusting the plan, a more targeted closed-loop implementation path is made for cross-population plan migration, forming a personalized plan evolution for users, and finally building a full-chain autonomous evolution intelligent intervention ecosystem. For the first time, a clinical-level application idea and system platform of the "perception-analysis-intervention-optimization" closed-loop system has been realized. Attached Figure Description

[0054] Figure 1 The flowchart illustrates the overall process of a cross-modal fusion intelligent joint training method provided in the first embodiment of the present invention.

[0055] Figure 2 A detailed flowchart of a cross-modal fusion intelligent joint training method provided in the first embodiment of the present invention;

[0056] Figure 3 This is a structural diagram of a cross-modal fusion intelligent joint training system provided in the second embodiment of the present invention;

[0057] Figure 4 This is a structural diagram of a cross-modal fusion intelligent joint training system provided in the third embodiment of the present invention. Detailed Implementation

[0058] The technical content of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0059] This invention integrates multimodal perception, cross-modal learning, and dynamic evolutionary algorithms to construct the first cross-modal "cognition-emotion-motor" collaborative intervention platform, achieving independent innovation across the entire "data collection-analysis-intervention" chain. Specifically, this invention integrates the following five core technologies: multimodal perception, cross-modal coding, biological evolutionary algorithms, neurobiofeedback, and personalized solution evolution, including:

[0060] First, multimodal perception. This embodiment of the invention integrates four levels of data to achieve multimodal perception, including cognitive level RT&ACC behavior perception, emotional level facial expression recognition perception, motion level motion capture perception, and physiological change perception.

[0061] Second, cross-modal coding. Cross-modal semantic alignment technology maps cognitive behavior, facial features, action sequences, and physiological data to a unified representation space, capturing the correlation between cognition, emotion, and movement.

[0062] Third, biological evolution algorithms. Integrating memory enhancement algorithms with multimodal feedback devices, biological evolution calculations are achieved to construct a three-dimensional adaptive training program delivery model based on "cognitive load - emotional awareness - motor coordination".

[0063] Fourth, neurobiofeedback. Based on the tasks pushed by the model, neurobiofeedback data is collected during the process to form a data pool for scheme optimization.

[0064] Fifth, the evolution of personalized treatment plans. Based on the patient's cognitive, emotional, and motor assessment characteristics, the impairment characteristics are stratified. On this basis, the weight of the intervention plan is adjusted according to the user's real-time feedback (such as training completion rate and changes in physiological indicators), thereby realizing the evolution of personalized treatment plans.

[0065] First Embodiment

[0066] like Figure 1 and Figure 2 As shown, the first embodiment of the present invention provides a cross-modal fusion intelligent joint training method, which specifically includes the following steps:

[0067] S10: Multimodal perception.

[0068] After entering the system, users need to wear wearable devices. Once worn, cognitive training begins based on a preset initial cognitive training scheme. It's important to understand that this initial cognitive training scheme is a built-in standard training scheme designed to provide users with a cognitive training scenario. During the cognitive training process, the wearable device performs multimodal perception on the user to acquire multimodal perception data.

[0069] In one embodiment of the present invention, the multimodal data includes at least: cognitive-level perceptual data, emotional-level perceptual data, motor-level perceptual data, and physiological-level perceptual data. Specifically, the perceptual data at each level is obtained in the following ways:

[0070] (1) Cognitive levels

[0071] Using task response time The Cognitive Efficacy Index (Ccog) is calculated using the Accuracy-to-Accuracy (ACC) ratio. α+β=1, and the weight α is dynamically adjusted according to the task type.

[0072] (2) Emotional levels

[0073] Based on a surface-level action unit encoding system, a 3D convolutional network is used to extract micro-expression temporal features, and then expression recognition is performed on these features to output a six-dimensional emotion vector E, where E = [e^(i-jωt)]. joy e sade anger e fear e suprise e neutral ].

[0074] (3) Movement Level

[0075] An inertial measurement unit is used to capture joint acceleration and Euler angles. After noise reduction by low-pass filtering, a three-dimensional skeleton pose reconstruction algorithm is used to extract the coordinate sequence of multiple key points.

[0076] (4) Physiological hierarchy

[0077] Heart rate variability (HRV) was calculated using the time-domain standard deviation method.

[0078]

[0079] Where RR represents the heartbeat interval; N represents the total number of valid RR intervals within the analysis period; and i represents the i-th RR interval. This represents the average of all RR intervals.

[0080] Understandably, this embodiment innovatively constructs a cross-modal physiological-behavioral-cognitive dynamic perception system, achieving for the first time real-time synchronous acquisition of four levels of data: cognitive response (RT & ACC behavior), emotional micro-expressions, motion capture, and physiological indicators (respiratory rate, blood oxygen, and blood pressure). Through a wearable multi-source signal synchronization device, joint data acquisition from the multimodal assessment system is achieved, improving data acquisition efficiency. Furthermore, based on multi-level data at the same time, the correlation between perception data at each level can be captured more accurately, providing support for the subsequent delivery of cognitive training programs.

[0081] S20: Cross-modal coding.

[0082] After obtaining the multimodal perception data for user cognitive training based on step S10 above, it is necessary to achieve heterogeneous data mapping and correlation modeling based on cross-modal semantic alignment technology. Specifically, this includes the following steps:

[0083] S21: Perform cross-modal semantic alignment on the multimodal sensing data to obtain the corresponding encoded data for each level.

[0084] (1) Targeting the cognitive efficacy index Ccog

[0085] Processing task sequences using a Transformer encoder: S cog ∈R T×d Output context vector

[0086] The position encoding uses a sine function:

[0087] (2) Regarding the six-dimensional emotion vector E

[0088] Principal component analysis (PCA) was performed on E, and the variance of E was retained by 95%. pca ∈R 3

[0089] (3) For motion coordinate sequences

[0090] Using the Transformer Encoder, a spatiotemporal attention encoder is used to extract inter-joint coordination patterns and encode the skeletal coordinate sequence into a hierarchical motion semantic vector:

[0091]

[0092] Where, ΔJ t =J t -J t-1 Joint displacement vector (capturing motion dynamics); a t Derivative features such as joint angles / angular velocities; Representative feature splicing operation.

[0093] (4) Targeting Heart Rate Variability (HRV)

[0094] HRV time-frequency features are mapped to physiological state vectors.

[0095] p τ =W p ·MLP(h τ )+b p

[0096] Among them, W p b p Trainable weights and biases; MLP: Multilayer Perceptron, learning HRV nonlinear representations; p τ This represents the physiological state vector with window τ.

[0097] S22: Map the coded data corresponding to each level to the same representation space.

[0098] In this embodiment, by designing a cross-modal contrastive loss function, the encoded data corresponding to each level is adjusted accordingly when mapping the encoded data of each level to the same representation space.

[0099] The contrastive loss function is as follows:

[0100]

[0101] Z is the one. i Z j Z kis the embedding vector of the i, j, and kth physiological modalities, i.e., different modalities of the same sample. τ is a temperature parameter used to adjust the degree of attention given to difficult negative samples (usually between 0.01 and 0.5).

[0102] S23: Calculate the correlation matrix corresponding to cognitive-emotion-motor interactions using a multi-head attention mechanism:

[0103]

[0104] Wherein, the cross-modal attention weight matrix A∈R T×T Q represents the cognitive behavior encoding data within the space, K represents the emotion encoding data within the space, V represents the motion encoding data within the space; T represents the time series.

[0105] S30: Biological evolution calculation.

[0106] After the cross-modal attention weight matrix is ​​output in step S20, biological evolutionary calculations are performed using memory reinforcement algorithms and multi-objective evolutionary optimization to obtain evolutionary calculation data. Furthermore, based on this evolutionary calculation data and multimodal perception data, a pre-defined three-dimensional decision model is used to provide the user with the optimal evolutionary training scheme.

[0107] Specifically, step S30 includes three parts: the memory reinforcement algorithm (corresponding to step S31), multi-objective evolutionary optimization (corresponding to step S32), and the acquisition of the evolutionary training scheme (corresponding to step S33). The following is a detailed explanation of each of these three parts:

[0108] S31: Memory Reinforcement Algorithm

[0109] In this embodiment, a memory reinforcement algorithm is used, combining reinforcement learning with an external memory mechanism. By introducing a readable and writable external memory module, the problem of low efficiency and poor policy generalization in traditional reinforcement learning due to long-term reliance on samples is addressed. Its core is to achieve rapid retrieval and reasoning through structured storage of historical experience.

[0110] Specifically, the invention employs a dual-delay deep deterministic strategy gradient network, with the target network parameter update formula being: θ′←τθ+(1-τ)θ′, where τ=0.005;

[0111] Priority of sample p in priority experience replay i =|δ i |+∈;where δi is the TD error,∈=1e-6.

[0112] S32: Multi-objective evolutionary optimization

[0113] In this embodiment, a fitness function F is predefined to output a combination that maximizes the effectiveness of the user's physical training by adjusting the training weights of each dimension. The fitness function F is as follows:

[0114] F=w1*(1-CognitiveLoad)+w2*EmoStability+w3*MotorSmoothness;

[0115] Among them, the larger the value of F, the greater the physical training efficiency; w1, w2, and w3 represent the training weights of the three dimensions of cognitive load, emotional awareness, and motor coordination, respectively; CognitiveLoad represents the cognitive load parameter; EmoStability represents the emotional stability parameter; and MotorSmoothness represents the motor fluency parameter.

[0116] Understandably, after step S32, the individual's comprehensive score under the current cognitive, emotional, and motor training conditions can be output, thereby providing supporting training data for the selection of subsequent comprehensive program combinations.

[0117] S33: Construction of a three-dimensional decision-making model.

[0118] Specifically, in this embodiment, a three-dimensional decision-making model is pre-constructed based on multimodal perception data from multiple historical users. This three-dimensional decision-making model has multiple evolutionary training schemes for the co-evolution of cognitive load, emotion awareness, and motor coordination.

[0119] Therefore, after obtaining the evolutionary calculation data based on step S32, this evolutionary calculation data, together with the multimodal perception data collected in step S10, is input into the three-dimensional decision model to output a comprehensive training scheme for evolutionary training (including, but not limited to, training tasks that significantly improve user cognition, emotion, and motor ability, task difficulty, training duration, and training weights under each dimension). Then, based on the probability distribution of cognitive training schemes, the evolutionary training scheme with the highest probability is pushed to the user.

[0120] S40: Neurobiofeedback.

[0121] Once the evolutionary training scheme is obtained based on step S30 above, the user will switch from the initial cognitive training scheme to this evolutionary training scheme during cognitive training. Accordingly, when the user performs cognitive training based on this evolutionary training scheme, it is necessary to collect the user's task execution data and multimodal perception data to dynamically optimize the evolutionary training scheme and provide neurobiological feedback to the user.

[0122] The dynamic optimization includes at least adjusting the task difficulty, task type, and task training duration, while the neurobiological feedback includes at least showing the user the current training data and cognitive training results.

[0123] Understandably, step S40 includes two parts: one is the optimization of parameters for the training scheme, and the other is the training feedback for the user, which allows the user to more clearly understand the current training results and the effectiveness of the training.

[0124] S50: The evolution of personalized solutions.

[0125] Specifically, based on a dynamically optimized evolutionary training scheme, users are stratified according to their characteristics, for example: mild / moderate / severe cognitive impairment + mild / moderate / severe motor impairment + mild / moderate / severe emotional disorder. Then, based on the stratification results, the training weights of the three dimensions—cognitive load, emotional awareness, and motor coordination—are adjusted in the following ways to form a personalized cognitive training scheme for each user. The specific adjustment methods are as follows:

[0126] The feature stratification includes at least: mild / moderate / severe cognitive impairment + mild / moderate / severe motor impairment + mild / moderate / severe mood disorder.

[0127] For each feature stratification, personalized adjustments are made based on the principle of "training all three aspects simultaneously, with priority given to those with higher levels of difficulty".

[0128] After the personalized solution evolution is completed, the solution is updated based on the user's personalized cognitive training solution, and the process returns to step S10 above to perform multimodal perception based on the user's personalized cognitive training solution. Steps S10-S50 above are repeated until the training ends.

[0129] Furthermore, in this embodiment, biological feedback is provided to the user's current training state based on neurobiological indicators such as heart rate variability, in order to enable the user to further regulate their autonomic nervous system based on the biological feedback and achieve effective improvement in their cognitive, motor, and emotional states.

[0130] Second Embodiment

[0131] like Figure 3 As shown, based on the first embodiment described above, the second embodiment of the present invention provides an intelligent joint training system for cross-modal fusion, including a multimodal perception unit 1, a cross-modal coding unit 2, a biological evolution calculation unit 3, a decision-making unit 4, a neurobiological feedback unit 5, and a scheme evolution unit 6.

[0132] Among them, the multimodal perception unit 1 is a wearable device used to perform multimodal perception on the user during the cognitive training process in order to obtain multimodal perception data.

[0133] The cross-modal coding unit 2 is connected to the multimodal sensing unit 1 to perform cross-modal coding on the multimodal sensing data, thereby capturing the correlation between cognition, emotion, and movement, and obtaining the corresponding correlation matrix.

[0134] Biological evolution calculation unit 3 is connected to cross-modal coding unit 2 to perform biological evolution calculation based on the correlation matrix, through memory reinforcement algorithm and multi-objective evolution optimization, thereby obtaining evolution calculation data.

[0135] The decision-making unit 4 is pre-set with a three-dimensional decision-making model and is connected to the biological evolution calculation unit 3 and the multimodal perception unit 1 to output the probability distribution of the evolutionary training scheme based on the evolution calculation data and the multimodal perception data.

[0136] The neurobiofeedback unit 5 is connected to the decision-making unit 4 to collect task execution data and multimodal perception data of the user to complete the evolutionary training program, so as to dynamically optimize the evolutionary training program and provide neurobiofeedback to the user.

[0137] The scheme evolution unit 6 is connected to the neurobiofeedback unit 5 to perform feature stratification based on the user's cognitive impairment characteristics according to the dynamically optimized evolutionary training scheme. This adjusts the training weights of the three dimensions—cognitive load, emotional awareness, and motor coordination—to form a personalized cognitive training scheme for the user. Furthermore, the scheme evolution unit 6 is also connected to the multimodal perception unit 1, enabling the multimodal perception unit to perform multimodal perception on the user based on the updated personalized cognitive training scheme.

[0138] It is understood that the functions and connections of the above-mentioned modules are only one specific implementation of the intelligent joint training method in the first embodiment. In other embodiments, the functions and connections of the modules can be adapted as needed, and no specific limitations are made here.

[0139] Third Embodiment

[0140] like Figure 4 As shown, based on the aforementioned cross-modal fusion intelligent joint training method, this invention further provides a cross-modal fusion intelligent joint training system. This intelligent joint training system includes one or more processors and a memory. The memory is coupled to the processors and is used to store one or more programs. When the one or more programs are executed by the one or more processors, the processors implement the cross-modal fusion intelligent joint training method described in the above embodiments.

[0141] The processor controls the overall operation of the intelligent joint training system to complete all or part of the steps of the aforementioned cross-modal fusion intelligent joint training method. The processor can be a central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), digital signal processing (DSP) chip, etc. The memory stores various types of data to support the operation of the intelligent joint training system. This data may include, for example, instructions for any application or method operating on the intelligent joint training system, as well as application-related data. The memory can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, etc.

[0142] In one exemplary embodiment, the intelligent joint training system can be implemented by a computer chip or physical entity, or by a product with certain functions, to perform the cross-modal fusion intelligent joint training method described above and achieve the same technical effect as the method described above. A typical embodiment is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0143] In another exemplary embodiment, the present invention also provides a computer-readable storage medium including program instructions, which, when executed by a processor, implement the steps of the cross-modal fusion intelligent joint training method in any of the above embodiments. For example, the computer-readable storage medium may be the memory including the program instructions described above, which may be executed by the processor of the intelligent joint training system to complete the cross-modal fusion intelligent joint training method described above and achieve the same technical effects as the method described above.

[0144] It should be noted that the above embodiments are merely illustrative examples. The technical solutions of each embodiment can be combined, and all are within the protection scope of this invention.

[0145] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0146] The cross-modal fusion intelligent joint training method and system provided by this invention have been described in detail above. Any obvious modifications made by those skilled in the art without departing from the essence of this invention will constitute an infringement of the patent rights of this invention and will incur corresponding legal liability.

Claims

1. A method for intelligent joint training across modalities, characterized in that comprising the following steps: based on the preset initial cognitive training scheme, the user is multi-modal perception during the cognitive training process to obtain multi-modal perception data; wherein the multi-modal perception data at least includes: cognitive level perception data, emotion level perception data, motion level perception data and physiological level perception data; The multi-modal perception data is cross-modally coded to capture the correlation among cognition-emotion-motion and obtain the corresponding correlation matrix; Based on the correlation matrix, the biological evolution algorithm is carried out by memory reinforcement algorithm and multi-objective evolutionary optimization to obtain evolution algorithm data; The evolution algorithm data and the multi-modal perception data are input into the preset three-dimensional decision model to output the probability distribution of the evolution training scheme; wherein the three-dimensional decision model is constructed based on the multi-modal perception data of a plurality of historical users, and a plurality of cognitive load-emotion awareness-motion coordination three-dimensional coordinated evolution evolution training schemes are preset; Based on the probability distribution of the cognitive training scheme, the user is pushed to the evolution training scheme with the highest probability; Collect the task execution data and multi-modal perception data of the user completing the evolution training scheme to dynamically optimize the evolution training scheme and provide neurobiological feedback to the user; wherein the dynamic optimization at least includes cognitive, emotional, motion adjustment task difficulty, task type and task training time, and the neurobiological feedback at least includes displaying the current training data and cognitive training results to the user; Based on the dynamically optimized evolution training scheme, the cognitive impairment characteristics of the user are layered to adjust the training weight of the three dimensions of cognitive load-emotion awareness-motion coordination, thereby forming a personalized cognitive training scheme for the user; Obtain the multi-modal perception data of the user based on the personalized cognitive training scheme, and repeat the above process until the training is completed. 2.The intelligent joint training method of claim 1, wherein The multi-modal perception data is obtained by the following method: At the cognitive level, the task response time and accuracy are used to calculate the cognitive efficiency index Ccog; a + b = 1, and the weight a is dynamically adjusted by the task type; On the emotion level, based on the facial action unit coding system, the micro-expression time sequence features are extracted through a 3D convolution network, and emotion recognition is performed on the micro-expression time sequence features to output a six-dimensional emotion vector E, E=[e joy ,e sad ,e anger ,e fear ,e surprise ,e neutral ] At the motion level, the inertial measurement unit is used to capture joint acceleration and Euler angle, and after low-pass filtering and noise reduction, the three-dimensional skeletal pose reconstruction algorithm is used to extract the coordinate sequence of multiple key points; At the physiological level, heart rate variability HRV is calculated using the time-domain standard deviation method, RR represents the heartbeat interval; N represents the total number of valid RR intervals within the analysis time period; i represents the ith RR interval; represents the average of all RR intervals. 3.The intelligent joint training method of claim 2, wherein The multi-modal perception data is cross-modally coded to capture the correlation among cognition-emotion-motion and obtain the corresponding correlation matrix, specifically including: Cross-modal semantic alignment is performed on the multi-modal perception data to obtain the corresponding encoding data of each level; Map the corresponding encoding data of each level to the same representation space; Use the multi-head attention mechanism to calculate the correlation matrix corresponding to the cognition-emotion-motion interaction: Wherein, Q is the cognitive behavior encoding data in the representation space, K is the emotion encoding data in the representation space, and V is the motion encoding data in the representation space; T represents the time sequence.

4. The intelligent joint training method of claim 1, characterized in that: The memory reinforcement algorithm combines reinforcement learning and external memory mechanism, introduces a readable and writable external memory module for structured storage of historical experience, to realize fast retrieval and reasoning; specifically including: A double-delay deep deterministic policy gradient network is adopted, and the target network parameter update formula is: θ'←τθ+(1-τ)θ', τ=0.005; Sample priority p in prioritized experience replay i = |δ i | + ∈, where δi is the TD error and ∈ = 1e-6. 5.The intelligent joint training method of claim 1, wherein The multi-objective evolutionary optimization includes: Defining a fitness function F to output a combination of maximum effectiveness of the user's physical training by adjusting the training weights of each dimension; F=w1*(1-CognitiveLoad)+w2*EmoStability+w3*MotorSmoothness; Wherein, F is the fitness function, w1, w2, w3 represent the training weights of cognitive load-emotion awareness-motor coordination respectively, CognitiveLoad represents the cognitive load parameter, EmoStability represents the emotion stability parameter, and MotorSmoothness represents the motor fluency parameter.

6. The intelligent joint training method of claim 1, wherein: The feature hierarchy at least includes: mild / moderate / severe cognitive impairment, mild / moderate / severe motor impairment, and mild / moderate / severe emotional impairment; For each feature hierarchy, individualized adjustment is made based on the principle of "three training and high degree of impairment priority training".

7. The intelligent joint training method of claim 2, wherein: Based on the heart rate variability of the user, biological feedback is provided for the user's current training state, so that the user can further self-regulate based on the biological feedback, and effectively improve the cognitive, motor and emotional state.

8. An intelligent joint training system for implementing the intelligent joint training method of any one of claims 1-7. Including: A multi-modal perception unit for multi-modal perception of the user during cognitive training to obtain multi-modal perception data; A cross-modal encoding unit connected to the multi-modal perception unit for cross-modal encoding of the multi-modal perception data to capture the association between cognition-emotion-motor and obtain the corresponding association matrix; A biological evolution algorithm unit connected to the cross-modal encoding unit for biological evolution algorithm based on the association matrix through memory reinforcement algorithm and multi-objective evolutionary optimization to obtain evolution algorithm data; A decision unit preset with a three-dimensional decision model and connected to the biological evolution algorithm unit and multi-modal perception unit to output the probability distribution of the evolution training scheme based on the evolution algorithm data and the multi-modal perception data; A neurobiological feedback unit connected to the decision unit for collecting task execution data and multi-modal perception data of the user completing the evolution training scheme, dynamically optimizing the evolution training scheme, and providing neurobiological feedback to the user. a scheme evolution unit, connected with the neuro-biofeedback unit, to perform feature stratification according to the cognitive impairment features of the user based on the dynamically optimized evolution training scheme, to adjust the training weights of the three dimensions of cognitive load-emotion awareness-motor coordination, so as to form a personalized cognitive training scheme for the user; Furthermore, the scheme evolution unit is also connected with the multi-modal perception unit, so that the multi-modal perception unit performs multi-modal perception on the user based on the updated personalized cognitive training scheme. 9.The intelligent joint training system of claim 8, characterized in that: the multi-modal perception unit is a wearable device.

10. A cross-modal fusion intelligent joint training system, characterized in that a processor and a memory, wherein the processor reads a computer program in the memory and is configured to execute the intelligent joint training method of any one of claims 1-7.

Images