Intelligent joint training method and system for cross-modal fusion

Abstract

The application discloses a kind of cross-modal fusion's intelligent joint training method and system.The method comprises: based on the initial cognitive training scheme of preestablished, obtain the multimodal perception data when user training;Multi-modal perception data are encoded across modalities, to capture the association between cognition-emotion-movement, and obtain the corresponding association matrix;Based on the association matrix, biological evolution algorithm is carried out by memory reinforcement algorithm and multi-objective evolutionary optimization;Evolution algorithm data and multimodal perception data are jointly input into the preset three-dimensional decision model, to output the probability distribution of evolution training scheme, and push the evolution training scheme with highest probability;Evolution training scheme is dynamically optimized, and user is carried out neurobiological feedback;Based on the evolution training scheme of dynamic optimization, according to the cognitive impairment characteristic stratification result of user, individualized scheme evolution is carried out;Repeat the above process until training is finished.

Description

Technical Field

[0001] This invention relates to a cross-modal fusion intelligent joint training method, and also to a corresponding intelligent joint training system, belonging to the field of cognitive training technology. Background Technology

[0002] Cognitive training is an effective means of delaying cognitive decline. However, to achieve good cognitive improvement, individuals need to engage in training continuously for at least three months or even longer. Therefore, how to stimulate and maintain individuals' interest in cognitive training, enabling them to continuously exert effort and take action, has become a key issue that urgently needs to be addressed in the field of cognitive training task design.

[0003] Traditional cognitive training methods are often based on classical paradigms, employing cognitive tasks of uniform difficulty. However, due to significant differences in cognitive abilities among individuals, this approach has obvious drawbacks. For some individuals, tasks that are too easy may cause them to gradually lose interest in training, reduce their focus on it, and consequently decrease the frequency of training; while for others, tasks that are too difficult may cause them immense pressure, damage their confidence, and ultimately reduce their willingness to persist in cognitive training. Therefore, to better meet the needs of different individuals, it is necessary to develop personalized cognitive training systems that can address individual differences. Such systems can dynamically adjust the difficulty of tasks during training by analyzing the gradient of cognitive task difficulty and achieve precise adjustment of task difficulty based on the user's real-time status.

[0004] Currently, most existing cognitive training systems focus only on physiological and cognitive aspects, neglecting the equally important factors of emotional stability and motor coordination. This is particularly true for elderly patients with both cognitive and motor dysfunctions, where such systems, which ignore emotional and motor aspects, have significant limitations in their rehabilitation effectiveness. Summary of the Invention

[0005] The primary technical problem to be solved by this invention is to provide an intelligent joint training method for cross-modal fusion.

[0006] Another technical problem to be solved by the present invention is to provide an intelligent joint training system with cross-modal fusion.

[0007] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solution:

[0008] According to a first aspect of the present invention, a cross-modal fusion intelligent joint training method is provided, comprising the following steps:

[0009] Based on a preset initial cognitive training scheme, the user is subjected to multimodal perception during the cognitive training process to obtain multimodal perception data; wherein, the multimodal perception data includes at least: cognitive level perception data, emotional level perception data, motor level perception data, and physiological level perception data.

[0010] The multimodal perception data is cross-modal encoded to capture the correlation between cognition, emotion, and movement, and the corresponding correlation matrix is ​​obtained.

[0011] Based on the aforementioned correlation matrix, biological evolution calculations are performed using a memory reinforcement algorithm and multi-objective evolutionary optimization to obtain evolutionary calculation data.

[0012] The evolutionary calculation data and the multimodal perception data are input into a preset three-dimensional decision model to output the probability distribution of the evolutionary training scheme; wherein, the three-dimensional decision model is constructed based on the multimodal perception data of multiple historical users and has preset multiple evolutionary training schemes for the co-evolution of cognitive load, emotion awareness and motor coordination in three dimensions.

[0013] Based on the probability distribution of the cognitive training scheme, the evolutionary training scheme with the highest probability is pushed to the user;

[0014] The system collects task execution data and multimodal perception data of the user to complete the evolutionary training program, so as to dynamically optimize the evolutionary training program and provide neurobiological feedback to the user; wherein, the dynamic optimization includes at least cognitive, emotional, and motor adjustment of task difficulty, task type and task training duration, and the neurobiological feedback includes at least showing the user the current training data and cognitive training results;

[0015] Based on the dynamically optimized evolutionary training scheme, feature stratification is performed according to the cognitive impairment characteristics of the user to adjust the training weights of the three dimensions of cognitive load, emotional awareness, and motor coordination, thereby forming a personalized cognitive training scheme for the user.

[0016] Obtain the user's multimodal perception data based on the personalized cognitive training scheme, and repeat the above process until the training ends.

[0017] Preferably, the multimodal sensing data is acquired through the following method:

[0018] At the cognitive level, the cognitive efficacy index Ccog is calculated using task response time and accuracy. α+β=1, and the weight α is dynamically adjusted according to the task type;

[0019] At the emotion level, based on a surface-level action unit encoding system, micro-expression temporal features are extracted through a 3D convolutional network, and expression recognition is performed on these micro-expression temporal features to output a six-dimensional emotion vector E, E = [e joy e sad e anger e fear e surprise e neutral ];

[0020] At the motion level, inertial measurement units are used to capture joint acceleration and Euler angles. After noise reduction through low-pass filtering, a three-dimensional skeleton pose reconstruction algorithm is used to extract the coordinate sequence of multiple key points.

[0021] At the physiological level, heart rate variability (HRV) is calculated using the time-domain standard deviation method. RR represents the heart rate interval; N represents the total number of valid RR intervals within the analysis period; i represents the i-th RR interval; This represents the average of all RR intervals.

[0022] Preferably, the multimodal perception data is cross-modal encoded to capture the correlation between cognition, emotion, and movement, and the corresponding correlation matrix is ​​obtained, including:

[0023] Cross-modal semantic alignment is performed on the multimodal sensing data to obtain the corresponding encoded data for each level;

[0024] Map the encoded data corresponding to each level to the same representation space;

[0025] Calculate the correlation matrix corresponding to cognitive-emotion-motor interactions using a multi-head attention mechanism:

[0026]

[0027] Where Q represents the cognitive behavior encoding data in the representation space, K represents the emotion encoding data in the representation space, V represents the motion encoding data in the representation space, and T represents the time series.

[0028] A preferred approach is to employ a memory reinforcement algorithm combined with reinforcement learning and external memory mechanisms. This involves introducing a readable and writable external memory module to structurally store historical experience, thereby enabling rapid retrieval and reasoning. Specifically, this includes the following sub-steps:

[0029] A dual-delay deep deterministic gradient network is adopted, and the target network parameter update formula is: θ′←τθ+(1-τ)θ′, τ=0.005;

[0030] Sample priority p in priority experience replay i =|δ i|+∈,;where δi is the TD error, ε=1e-6.

[0031] Preferably, the multi-objective evolutionary optimization specifically includes:

[0032] Define a fitness function F to output a combination that maximizes the efficiency of the user's physical training by adjusting the training weights of each dimension.

[0033] F=w1*(1-CognitiveLoad)+w2*EmoStability+w3*

[0034] Motor Smoothness;

[0035] Among them, the larger the value of F, the greater the physical training efficiency; w1, w2, and w3 represent the training weights of the three dimensions of cognitive load, emotional awareness, and motor coordination, respectively; CognitiveLoad represents the cognitive load parameter; EmoStability represents the emotional stability parameter; and MotorSmoothness represents the motor fluency parameter.

[0036] Preferably, the feature stratification includes at least: mild / moderate / severe cognitive impairment, mild / moderate / severe motor impairment, and mild / moderate / severe mood disorder;

[0037] For each feature stratification, personalized adjustments are made based on the principle of "training all three aspects and prioritizing training those with higher levels of difficulty".

[0038] Preferably, based on the user's heart rate variability, biological feedback is provided to the user's current training state, with the aim of enabling the user to further regulate their autonomic nervous system based on the biological feedback and achieve effective improvement in their cognitive, motor, and emotional states.

[0039] According to a second aspect of the present invention, an intelligent joint training system applying the above-described intelligent joint training method is provided, comprising:

[0040] A multimodal perception unit is used to perform multimodal perception on the user during cognitive training to obtain multimodal perception data.

[0041] A cross-modal coding unit, connected to the multimodal sensing unit, is used to perform cross-modal coding on the multimodal sensing data, thereby capturing the correlation between cognition, emotion, and movement, and obtaining the corresponding correlation matrix.

[0042] The biological evolution calculation unit is connected to the cross-modal coding unit to perform biological evolution calculations based on the correlation matrix using a memory reinforcement algorithm and multi-objective evolutionary optimization, thereby obtaining evolution calculation data.

[0043] The decision-making unit is pre-configured with a three-dimensional decision-making model and is connected to the biological evolution calculation unit and the multimodal perception unit to output the probability distribution of the evolutionary training scheme based on the evolutionary calculation data and the multimodal perception data.

[0044] A neurobiofeedback unit, connected to the decision unit, is used to collect task execution data and multimodal perception data of the user completing the evolutionary training program, so as to dynamically optimize the evolutionary training program and provide neurobiofeedback to the user.

[0045] The scheme evolution unit is connected to the neurobiofeedback unit to perform feature stratification based on the user's cognitive impairment characteristics according to the dynamically optimized evolution training scheme, so as to adjust the training weights of the three dimensions of cognitive load, emotional awareness and motor coordination, thereby forming a personalized cognitive training scheme for the user.

[0046] Furthermore, the scheme evolution unit is also connected to the multimodal perception unit, so that the multimodal perception unit can perform multimodal perception of the user based on the updated personalized cognitive training scheme.

[0047] Preferably, the multimodal sensing unit is a wearable device.

[0048] According to a third aspect of the present invention, a cross-modal fusion intelligent joint training system is provided, including a processor and a memory, wherein the processor reads a computer program in the memory for executing the above-described intelligent joint training method.

[0049] Compared with the prior art, the present invention has the following technical effects:

[0050] (1) This invention has made a breakthrough in constructing a cross-modal physiological-behavioral-cognitive dynamic perception system, achieving for the first time real-time synchronous acquisition of four levels of data: cognitive response (RT & ACC behavior), emotional micro-expressions, motion capture, and physiological indicators (respiratory rate, blood oxygen, and blood pressure). By using a wearable multi-source signal synchronization device, the problem of joint data acquisition and evaluation in multimodal assessment systems has been solved.

[0051] (2) This invention presents a pioneering biological evolution-driven three-dimensional adaptive training engine. By integrating memory reinforcement networks and real-time feedback from multimodal perception, a three-dimensional co-evolutionary model of "cognitive load - emotional awareness - motor coordination" is constructed. An innovative Pareto frontier optimization strategy is designed to achieve a balance between training intensity and user tolerance, forming an evolutionary training program.

[0052] (3) The embodiments of the present invention construct a closed-loop neurobiofeedback enhancement system, breaking through the traditional one-way stimulus response mode. By synchronously regulating modalities such as heart rate variability across scales, feedback fusion is performed to accurately match the neural plasticity time window and promote the speed of autonomic nervous system regulation response.

[0053] (4) This invention proposes a barrier feature stratification mechanism to solve the problem of intervention adaptation for people with multiple comorbidities. By dynamically adjusting the plan, a more targeted closed-loop implementation path is made for cross-population plan migration, forming a personalized plan evolution for users, and finally building a full-chain autonomous evolution intelligent intervention ecosystem. For the first time, a clinical-level application idea and system platform of the "perception-analysis-intervention-optimization" closed-loop system has been realized. Attached Figure Description

[0054] Figure 1 A flowchart illustrating the overall process of a cross-modal fusion intelligent joint training method provided in the first embodiment of the present invention;

[0055] Figure 2 A detailed flowchart of a cross-modal fusion intelligent joint training method provided in the first embodiment of the present invention;

[0056] Figure 3 This is a structural diagram of a cross-modal fusion intelligent joint training system provided in the second embodiment of the present invention;

[0057] Figure 4 This is a structural diagram of a cross-modal fusion intelligent joint training system provided in the third embodiment of the present invention. Detailed Implementation

[0058] The technical content of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0059] This invention integrates multimodal perception, cross-modal learning, and dynamic evolutionary algorithms to construct the first cross-modal "cognition-emotion-motor" collaborative intervention platform, achieving independent innovation across the entire "data collection-analysis-intervention" chain. Specifically, this invention integrates the following five core technologies: multimodal perception, cross-modal coding, biological evolutionary algorithms, neurobiofeedback, and personalized solution evolution, including:

[0060] First, multimodal perception. This embodiment of the invention integrates four levels of data to achieve multimodal perception, including cognitive level RT&ACC behavior perception, emotional level facial expression recognition perception, motion level motion capture perception, and physiological change perception.

[0061] Second, cross-modal coding. Cross-modal semantic alignment technology maps cognitive behavior, facial features, action sequences, and physiological data to a unified representation space, capturing the correlation between cognition, emotion, and movement.

[0062] Third, biological evolution algorithms. Integrating memory enhancement algorithms with multimodal feedback devices, biological evolution calculations are achieved to construct a three-dimensional adaptive training program delivery model based on "cognitive load - emotional awareness - motor coordination".

[0063] Fourth, neurobiofeedback. Based on the tasks pushed by the model, neurobiofeedback data is collected during the process to form a data pool for scheme optimization.

[0064] Fifth, the evolution of personalized treatment plans. Based on the patient's cognitive, emotional, and motor assessment characteristics, the impairment characteristics are stratified. On this basis, the weight of the intervention plan is adjusted according to the user's real-time feedback (such as training completion rate and changes in physiological indicators), thereby realizing the evolution of personalized treatment plans.

[0065] First Embodiment

[0066] like Figure 1 and Figure 2 As shown, the first embodiment of the present invention provides a cross-modal fusion intelligent joint training method, which specifically includes the following steps:

[0067] S10: Multimodal perception.

[0068] After entering the system, users need to wear wearable devices. Once worn, cognitive training begins based on a preset initial cognitive training scheme. It's important to understand that this initial cognitive training scheme is a built-in standard training scheme designed to provide users with a cognitive training scenario. During the cognitive training process, the wearable device performs multimodal perception on the user to acquire multimodal perception data.

[0069] In one embodiment of the present invention, the multimodal data includes at least: cognitive-level perceptual data, emotional-level perceptual data, motor-level perceptual data, and physiological-level perceptual data. Specifically, the perceptual data at each level is obtained in the following ways:

[0070] (1) Cognitive levels

[0071] Using task response time The Cognitive Efficacy Index (Ccog) is calculated using the Accuracy-to-Accuracy (ACC) ratio α+β=1, and the weight α is dynamically adjusted according to the task type.

[0072] (2) Emotional levels

[0073] Based on a surface-level action unit encoding system, a 3D convolutional network is used to extract micro-expression temporal features, and then expression recognition is performed on these features to output a six-dimensional emotion vector E, where E = [e^(i-jωt)]. joy e sad e angere fear e suprise e neutral ].

[0074] (3) Movement Level

[0075] An inertial measurement unit is used to capture joint acceleration and Euler angles. After noise reduction by low-pass filtering, a three-dimensional skeleton pose reconstruction algorithm is used to extract the coordinate sequence of multiple key points.

[0076] (4) Physiological hierarchy

[0077] Heart rate variability (HRV) was calculated using the time-domain standard deviation method.

[0078]

[0079] Where RR represents the heartbeat interval; N represents the total number of valid RR intervals within the analysis period; and i represents the i-th RR interval. This represents the average of all RR intervals.

[0080] Understandably, this embodiment innovatively constructs a cross-modal physiological-behavioral-cognitive dynamic perception system, achieving for the first time real-time synchronous acquisition of four levels of data: cognitive response (RT & ACC behavior), emotional micro-expressions, motion capture, and physiological indicators (respiratory rate, blood oxygen, and blood pressure). Through a wearable multi-source signal synchronization device, joint data acquisition from the multimodal assessment system is achieved, improving data acquisition efficiency. Furthermore, based on multi-level data at the same time, the correlation between perception data at each level can be captured more accurately, providing support for the subsequent delivery of cognitive training programs.

[0081] S20: Cross-modal coding.

[0082] After obtaining the multimodal perception data for user cognitive training based on step S10 above, it is necessary to achieve heterogeneous data mapping and correlation modeling based on cross-modal semantic alignment technology. Specifically, this includes the following steps:

[0083] S21: Perform cross-modal semantic alignment on the multimodal sensing data to obtain the corresponding encoded data for each level.

[0084] (1) Targeting the cognitive efficacy index Ccog

[0085] Processing task sequences using a Transformer encoder: S cog ∈R T×d Output context vector

[0086] The position encoding uses a sine function:

[0087] (2) Regarding the six-dimensional emotion vector E

[0088] Principal component analysis (PCA) was performed on E, and the variance of E was retained by 95%. pca ∈R 3

[0089] (3) For motion coordinate sequences

[0090] Using the Transformer Encoder, a spatiotemporal attention encoder is used to extract inter-joint coordination patterns and encode the skeletal coordinate sequence into a hierarchical motion semantic vector:

[0091]

[0092] Where, ΔJ t =J t -J t-1 Joint displacement vector (capturing motion dynamics); a t Derivative features such as joint angles / angular velocities; Representative feature splicing operation.

[0093] (4) Targeting Heart Rate Variability (HRV)

[0094] HRV time-frequency features are mapped to physiological state vectors.

[0095] p τ =W p ·MLP(h τ )+b p

[0096] Among them, W p b p Trainable weights and biases; MLP: Multilayer Perceptron, learning HRV nonlinear representations; p τ This represents the physiological state vector with window τ.

[0097] S22: Map the coded data corresponding to each level to the same representation space.

[0098] In this embodiment, by designing a cross-modal contrastive loss function, the encoded data corresponding to each level is adjusted accordingly when mapping the encoded data of each level to the same representation space.

[0099] The contrastive loss function is as follows:

[0100]

[0101] Z is the one. i Z j Z kis the embedding vector of the i, j, and kth physiological modalities, i.e., different modalities of the same sample. τ is a temperature parameter used to adjust the degree of attention given to difficult negative samples (usually between 0.01 and 0.5).

[0102] S23: Calculate the correlation matrix corresponding to cognitive-emotion-motor interactions using a multi-head attention mechanism:

[0103]

[0104] Wherein, the cross-modal attention weight matrix A∈R T×T Q represents the cognitive behavior encoding data within the space, K represents the emotion encoding data within the space, V represents the motion encoding data within the space; T represents the time series.

[0105] S30: Biological evolution calculation.

[0106] After the cross-modal attention weight matrix is ​​output in step S20, biological evolutionary calculations are performed using memory reinforcement algorithms and multi-objective evolutionary optimization to obtain evolutionary calculation data. Furthermore, based on this evolutionary calculation data and multimodal perception data, a pre-defined three-dimensional decision model is used to provide the user with the optimal evolutionary training scheme.

[0107] Specifically, step S30 includes three parts: the memory reinforcement algorithm (corresponding to step S31), multi-objective evolutionary optimization (corresponding to step S32), and the acquisition of the evolutionary training scheme (corresponding to step S33). The following is a detailed explanation of each of these three parts:

[0108] S31: Memory Reinforcement Algorithm

[0109] In this embodiment, a memory reinforcement algorithm is used, combining reinforcement learning with an external memory mechanism. By introducing a readable and writable external memory module, the problem of low efficiency and poor policy generalization in traditional reinforcement learning due to long-term reliance on samples is addressed. Its core is to achieve rapid retrieval and reasoning through structured storage of historical experience.

[0110] Specifically, the invention employs a dual-delay deep deterministic strategy gradient network, with the target network parameter update formula being: θ′←τθ+(1-τ)θ′, where τ=0.005;

[0111] Priority of sample p in priority experience replay i =|δ i |+∈;where δi is the TD error,∈=1e-6.

[0112] S32: Multi-objective evolutionary optimization

[0113] In this embodiment, a fitness function F is predefined to output a combination that maximizes the effectiveness of the user's physical training by adjusting the training weights of each dimension. The fitness function F is as follows:

[0114] F=w1*(1-CognitiveLoad)+w2*EmoStability+w3*MotorSmoothness;

[0115] Among them, the larger the value of F, the greater the physical training efficiency; w1, w2, and w3 represent the training weights of the three dimensions of cognitive load, emotional awareness, and motor coordination, respectively; CognitiveLoad represents the cognitive load parameter; EmoStability represents the emotional stability parameter; and MotorSmoothness represents the motor fluency parameter.

[0116] Understandably, after step S32, the individual's comprehensive score under the current cognitive, emotional, and motor training conditions can be output, thereby providing supporting training data for the selection of subsequent comprehensive program combinations.

[0117] S33: Construction of a three-dimensional decision-making model.

[0118] Specifically, in this embodiment, a three-dimensional decision-making model is pre-constructed based on multimodal perception data from multiple historical users. This three-dimensional decision-making model has multiple evolutionary training schemes for the co-evolution of cognitive load, emotion awareness, and motor coordination.

[0119] Therefore, after obtaining the evolutionary calculation data based on step S32, this evolutionary calculation data, together with the multimodal perception data collected in step S10, is input into the three-dimensional decision model to output a comprehensive training scheme for evolutionary training (including, but not limited to, training tasks that significantly improve user cognition, emotion, and motor ability, task difficulty, training duration, and training weights under each dimension). Then, based on the probability distribution of cognitive training schemes, the evolutionary training scheme with the highest probability is pushed to the user.

[0120] S40: Neurobiofeedback.

[0121] Once the evolutionary training scheme is obtained based on step S30 above, the user will switch from the initial cognitive training scheme to this evolutionary training scheme during cognitive training. Accordingly, when the user performs cognitive training based on this evolutionary training scheme, it is necessary to collect the user's task execution data and multimodal perception data to dynamically optimize the evolutionary training scheme and provide neurobiological feedback to the user.

[0122] The dynamic optimization includes at least adjusting the task difficulty, task type, and task training duration, while the neurobiological feedback includes at least showing the user the current training data and cognitive training results.

[0123] Understandably, step S40 includes two parts: one is the optimization of parameters for the training scheme, and the other is the training feedback for the user, which allows the user to more clearly understand the current training results and the effectiveness of the training.

[0124] S50: The evolution of personalized solutions.

[0125] Specifically, based on a dynamically optimized evolutionary training scheme, users are stratified according to their characteristics, for example: mild / moderate / severe cognitive impairment + mild / moderate / severe motor impairment + mild / moderate / severe emotional disorder. Then, based on the stratification results, the training weights of the three dimensions—cognitive load, emotional awareness, and motor coordination—are adjusted in the following ways to form a personalized cognitive training scheme for each user. The specific adjustment methods are as follows:

[0126] The feature stratification includes at least: mild / moderate / severe cognitive impairment + mild / moderate / severe motor impairment + mild / moderate / severe mood disorder.

[0127] For each feature stratification, personalized adjustments are made based on the principle of "training all three aspects simultaneously, with priority given to those with higher levels of difficulty".

[0128] After the personalized solution evolution is completed, the solution is updated based on the user's personalized cognitive training solution, and the process returns to step S10 above to perform multimodal perception based on the user's personalized cognitive training solution. Steps S10-S50 above are repeated until the training ends.

[0129] Furthermore, in this embodiment, biological feedback is provided to the user's current training state based on neurobiological indicators such as heart rate variability, in order to enable the user to further regulate their autonomic nervous system based on the biological feedback and achieve effective improvement in their cognitive, motor, and emotional states.

[0130] Second Embodiment

[0131] like Figure 3 As shown, based on the first embodiment described above, the second embodiment of the present invention provides an intelligent joint training system for cross-modal fusion, including a multimodal perception unit 1, a cross-modal coding unit 2, a biological evolution calculation unit 3, a decision-making unit 4, a neurobiological feedback unit 5, and a scheme evolution unit 6.

[0132] Among them, the multimodal perception unit 1 is a wearable device used to perform multimodal perception on the user during the cognitive training process in order to obtain multimodal perception data.

[0133] The cross-modal coding unit 2 is connected to the multimodal sensing unit 1 to perform cross-modal coding on the multimodal sensing data, thereby capturing the correlation between cognition, emotion, and movement, and obtaining the corresponding correlation matrix.

[0134] Biological evolution calculation unit 3 is connected to cross-modal coding unit 2 to perform biological evolution calculation based on the correlation matrix, through memory reinforcement algorithm and multi-objective evolution optimization, thereby obtaining evolution calculation data.

[0135] The decision-making unit 4 is pre-set with a three-dimensional decision-making model and is connected to the biological evolution calculation unit 3 and the multimodal perception unit 1 to output the probability distribution of the evolutionary training scheme based on the evolution calculation data and the multimodal perception data.

[0136] The neurobiofeedback unit 5 is connected to the decision-making unit 4 to collect task execution data and multimodal perception data of the user to complete the evolutionary training program, so as to dynamically optimize the evolutionary training program and provide neurobiofeedback to the user.

[0137] The scheme evolution unit 6 is connected to the neurobiofeedback unit 5 to perform feature stratification based on the user's cognitive impairment characteristics according to the dynamically optimized evolutionary training scheme. This adjusts the training weights of the three dimensions—cognitive load, emotional awareness, and motor coordination—to form a personalized cognitive training scheme for the user. Furthermore, the scheme evolution unit 6 is also connected to the multimodal perception unit 1, enabling the multimodal perception unit to perform multimodal perception on the user based on the updated personalized cognitive training scheme.

[0138] It is understood that the functions and connections of the above-mentioned modules are only one specific implementation of the intelligent joint training method in the first embodiment. In other embodiments, the functions and connections of the modules can be adapted as needed, and no specific limitations are made here.

[0139] Third Embodiment

[0140] like Figure 4 As shown, based on the aforementioned cross-modal fusion intelligent joint training method, this invention further provides a cross-modal fusion intelligent joint training system. This intelligent joint training system includes one or more processors and a memory. The memory is coupled to the processors and is used to store one or more programs. When the one or more programs are executed by the one or more processors, the processors implement the cross-modal fusion intelligent joint training method described in the above embodiments.

[0141] The processor controls the overall operation of the intelligent joint training system to complete all or part of the steps of the aforementioned cross-modal fusion intelligent joint training method. The processor can be a central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), digital signal processing (DSP) chip, etc. The memory stores various types of data to support the operation of the intelligent joint training system. This data may include, for example, instructions for any application or method operating on the intelligent joint training system, as well as application-related data. The memory can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, etc.

[0142] In one exemplary embodiment, the intelligent joint training system can be implemented by a computer chip or physical entity, or by a product with certain functions, to perform the cross-modal fusion intelligent joint training method described above and achieve the same technical effect as the method described above. A typical embodiment is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0143] In another exemplary embodiment, the present invention also provides a computer-readable storage medium including program instructions, which, when executed by a processor, implement the steps of the cross-modal fusion intelligent joint training method in any of the above embodiments. For example, the computer-readable storage medium may be the memory including the program instructions described above, which may be executed by the processor of the intelligent joint training system to complete the cross-modal fusion intelligent joint training method described above and achieve the same technical effects as the method described above.

[0144] It should be noted that the above embodiments are merely illustrative examples. The technical solutions of each embodiment can be combined, and all are within the protection scope of this invention.

[0145] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0146] The cross-modal fusion intelligent joint training method and system provided by this invention have been described in detail above. Any obvious modifications made by those skilled in the art without departing from the essence of this invention will constitute an infringement of the patent rights of this invention and will incur corresponding legal liability.

Claims

1. A cross-modal fusion intelligent joint training system, characterized in that... The system includes a processor and a memory, wherein the processor reads a computer program from the memory to execute a cross-modal fusion intelligent joint training method; wherein the intelligent joint training method includes the following steps: Based on a preset initial cognitive training scheme, the user is subjected to multimodal perception during the cognitive training process to obtain multimodal perception data; wherein, the multimodal perception data includes at least: cognitive level perception data, emotional level perception data, motor level perception data, and physiological level perception data. The multimodal perception data is cross-modal encoded to capture the correlation between cognition, emotion, and movement, and the corresponding correlation matrix is ​​obtained. Based on the aforementioned correlation matrix, biological evolution calculations are performed using a memory reinforcement algorithm and multi-objective evolutionary optimization to obtain evolutionary calculation data. The evolutionary calculation data and the multimodal perception data are input into a preset three-dimensional decision model to output the probability distribution of the evolutionary training scheme; wherein, the three-dimensional decision model is constructed based on the multimodal perception data of multiple historical users and has preset multiple evolutionary training schemes for the co-evolution of cognitive load, emotion awareness and motor coordination in three dimensions. Based on the probability distribution of the cognitive training scheme, the evolutionary training scheme with the highest probability is pushed to the user; The system collects task execution data and multimodal perception data of the user to complete the evolutionary training program, so as to dynamically optimize the evolutionary training program and provide neurobiological feedback to the user; wherein, the dynamic optimization includes at least cognitive, emotional, and motor adjustment of task difficulty, task type and task training duration, and the neurobiological feedback includes at least showing the user the current training data and cognitive training results; Based on the dynamically optimized evolutionary training scheme, feature stratification is performed according to the cognitive impairment characteristics of the user to adjust the training weights of the three dimensions of cognitive load, emotional awareness, and motor coordination, thereby forming a personalized cognitive training scheme for the user. Obtain the user's multimodal perception data based on the personalized cognitive training scheme, and repeat the above process until the training ends.

2. The intelligent joint training system as described in claim 1, characterized in that... The multimodal sensing data is acquired through the following methods: At the cognitive level, the cognitive efficacy index Ccog is calculated using task response time and accuracy. α + β = 1, and the weight α is dynamically adjusted according to the task type; At the emotion level, based on a surface-level action unit encoding system, a 3D convolutional network is used to extract micro-expression temporal features, and these features are then used for expression recognition to output a six-dimensional emotion vector E. ; At the motion level, inertial measurement units are used to capture joint acceleration and Euler angles. After noise reduction through low-pass filtering, a three-dimensional skeleton pose reconstruction algorithm is used to extract the coordinate sequence of multiple key points. At the physiological level, heart rate variability (HRV) is calculated using the time-domain standard deviation method. RR represents the heart rate interval; N represents the total number of valid RR intervals within the analysis period; i represents the i-th RR interval; This represents the average of all RR intervals.

3. The intelligent joint training system as described in claim 2, characterized in that... The multimodal perception data is cross-modal encoded to capture the correlation between cognition, emotion, and movement, and the corresponding correlation matrix is ​​obtained, specifically including: Cross-modal semantic alignment is performed on the multimodal sensing data to obtain the corresponding encoded data for each level; Map the encoded data corresponding to each level to the same representation space; Calculate the correlation matrix corresponding to cognitive-emotion-motor interactions using a multi-head attention mechanism: Where Q represents the cognitive behavior encoding data in the representation space, K represents the emotion encoding data in the representation space, V represents the motion encoding data in the representation space, and T represents the time series.

4. The intelligent joint training system as described in claim 1, characterized in that: This approach employs a memory reinforcement algorithm combined with reinforcement learning and external memory mechanisms. It introduces a readable and writable external memory module to structurally store historical experience, enabling rapid retrieval and reasoning. Specifically, it includes: Employing a dual-delay deep deterministic gradient network, the target network parameter update formula is as follows: τ = 0.005; Sample priority in priority experience replay Where δi is the TD error, .

5. The intelligent joint training system as described in claim 1, characterized in that... The multi-objective evolutionary optimization includes: Define a fitness function F to output a combination that maximizes the efficiency of the user's physical training by adjusting the training weights of each dimension. ； Where F is the fitness function, w1, w2, and w3 represent the training weights of the three dimensions of cognitive load, emotional awareness, and motor coordination, respectively. CognitiveLoad represents the cognitive load parameter, EmoStability represents the emotional stability parameter, and MotorSmoothness represents the motor smoothness parameter.

6. The intelligent joint training system as described in claim 1, characterized in that: The feature stratification includes at least: mild / moderate / severe cognitive impairment, mild / moderate / severe motor impairment, and mild / moderate / severe mood disorder; For each feature stratification, personalized adjustments are made based on the principle of "training all three aspects and prioritizing training those with higher levels of difficulty".

7. The intelligent joint training system as described in claim 2, characterized in that: Based on the user's heart rate variability, biological feedback is provided to the user's current training status, with the aim of enabling the user to further regulate their autonomic nervous system and achieve effective improvement in their cognitive, motor, and emotional states.

8. A cross-modal fusion intelligent joint training device, used to implement the intelligent joint training method according to any one of claims 1 to 7, characterized in that... include: A multimodal perception unit is used to perform multimodal perception on the user during cognitive training to obtain multimodal perception data. A cross-modal coding unit, connected to the multimodal sensing unit, is used to perform cross-modal coding on the multimodal sensing data, thereby capturing the correlation between cognition, emotion, and movement, and obtaining the corresponding correlation matrix. The biological evolution calculation unit is connected to the cross-modal coding unit to perform biological evolution calculations based on the correlation matrix using a memory reinforcement algorithm and multi-objective evolutionary optimization, thereby obtaining evolution calculation data. The decision-making unit is pre-configured with a three-dimensional decision-making model and is connected to the biological evolution calculation unit and the multimodal perception unit to output the probability distribution of the evolutionary training scheme based on the evolutionary calculation data and the multimodal perception data. A neurobiofeedback unit, connected to the decision unit, is used to collect task execution data and multimodal perception data of the user completing the evolutionary training program, so as to dynamically optimize the evolutionary training program and provide neurobiofeedback to the user. The scheme evolution unit is connected to the neurobiofeedback unit to perform feature stratification based on the user's cognitive impairment characteristics according to the dynamically optimized evolution training scheme, so as to adjust the training weights of the three dimensions of cognitive load, emotional awareness and motor coordination, thereby forming a personalized cognitive training scheme for the user. Furthermore, the scheme evolution unit is also connected to the multimodal perception unit, so that the multimodal perception unit can perform multimodal perception of the user based on the updated personalized cognitive training scheme.

9. The intelligent joint training device as described in claim 8, characterized in that: The multimodal sensing unit is a wearable device.

Images