A multi-person cooperative task closed-loop adjustment method and system based on joint state

Abstract

This invention discloses a closed-loop regulation method and system for multi-person collaborative tasks based on joint states, belonging to the field of multimodal human-computer interaction technology. It includes: synchronously acquiring EEG signals, functional near-infrared spectral signals, and motion posture signals of each member during a multi-person collaborative task as multimodal signals, and extracting multimodal features to construct the individual joint states and individual state reliability of each member; constructing a collaborative relationship matrix using individual joint states as nodes, individual state reliability as node weights, and task stages or role relationships as edge constraints, and generating a group collaborative joint index; identifying the type of collaborative mismatch when the group collaborative joint index is abnormal, generating a corresponding feedback regulation strategy, and executing feedback and updates to form a closed-loop regulation. This invention can effectively improve team collaboration efficiency and task execution quality during multi-person collaborative task execution, and is applicable to various application scenarios such as group collaborative training, multi-person collaborative tasks, and human-computer collaborative operations.

Description

Technical Field

[0001] This invention belongs to the field of multimodal human-computer collaborative interaction technology, specifically relating to a closed-loop adjustment method and system for multi-person collaborative tasks based on joint state. Background Technology

[0002] Multi-person collaborative tasks are widely used in scenarios such as group training, interactive teaching, team decision-making, rehabilitation interaction, human-computer collaboration, multi-person gamified training, and brain-computer interface training. In these tasks, the success depends not only on the attention level, cognitive load, or motor state of individual members, but also on the information exchange, role division, movement rhythm, and degree of neural activity coordination among multiple members. Therefore, it is difficult to comprehensively and accurately evaluate the overall collaborative state in multi-person collaborative tasks based solely on the physiological or behavioral state of a single individual.

[0003] Existing single-person brain-computer interface or physiological monitoring systems typically focus on analyzing individual EEG, blood oxygen, or behavioral signals, enabling single-person status display, training feedback, or evaluation reports. However, they struggle to depict the collaborative relationships among team members. While multi-person hyperscanning technology can simultaneously acquire brain activity signals from multiple members and analyze neural activity coupling relationships between different individuals by calculating indicators such as inter-brain correlation, coherence, or blood oxygen synchronicity, most solutions remain at the offline analysis or status display level. They lack a real-time closed-loop regulation mechanism that uses the group's collaborative state as a control variable, making it difficult to continuously optimize the task process based on changes in collaborative state.

[0004] Furthermore, existing multi-person collaborative evaluation methods suffer from modality limitations. For example, using only EEG signals to calculate brain synchronization can reflect the consistency of rapid neural electrical activity, but it is difficult to reflect slower blood oxygen metabolism responses and actual action execution. Using only functional near-infrared spectroscopy (fNIRS) signals to calculate blood oxygen synchronization can describe hemodynamic changes during interaction, but it cannot capture attentional fluctuations and changes in movement rhythm in a timely manner. Evaluating collaboration solely based on motor or task performance cannot distinguish between factors such as neural state consistency, uneven cognitive load, or interference from motor artifacts.

[0005] Furthermore, EEG, fNIRS, and motor posture signals have different time scales, signal characteristics, and physiological meanings. EEG has millisecond-level temporal resolution, fNIRS reflects a relatively slow blood oxygen dynamic response, and motor posture not only represents behavioral rhythm but can also be a source of artifacts in EEG and near-infrared signals. Simply weighting these three types of signals after independent calculation can easily lead to problems such as inconsistent time scales, abnormal quality affecting indicators, uninterpretable modal contributions, and misjudgment of coordinated states.

[0006] Existing closed-loop feedback adjustment methods typically employ simple threshold strategies, such as reducing task difficulty when the team's performance score is below a threshold and increasing task difficulty when the score is above a threshold. However, the decline in multi-person collaboration can be caused by a variety of factors, including inconsistent attentional states, uneven distribution of cognitive load, asynchronous movement rhythms, inconsistencies between brain activity and behavioral performance, or abnormal signal quality. If the system cannot further identify the type of collaboration mismatch and relies solely on a single score for software prompts or task difficulty adjustments, the feedback may lack specificity, or even lead to erroneous interventions when there are signal anomalies or artifacts.

[0007] Therefore, there is an urgent need for a technical solution for multi-person collaborative tasks that can simultaneously collect EEG, fNIRS, and motion posture signals from multiple members, unify the multimodal data into a unified EEG-near-infrared-motion posture joint state, construct a group collaborative joint index, identify different types of collaborative mismatch causes, and provide software prompts and adaptive adjustments to task difficulty based on the group collaborative state and mismatch type, thereby achieving real-time assessment, accurate feedback, and closed-loop optimization of multi-person collaborative state. Summary of the Invention

[0008] In view of the above, the purpose of this invention is to provide a closed-loop adjustment method and system for multi-person collaborative tasks based on joint state. By synchronously acquiring multimodal signals, constructing individual joint states, and analyzing group collaborative joint indicators, and combining a quality-gated dynamic fusion mechanism based on dynamic fusion weights and identification of collaborative mismatch types, it can realize real-time evaluation and anomaly diagnosis of the group's collaborative state during multi-person collaborative tasks, and generate feedback adjustment strategies that match the mismatch types accordingly. This effectively improves the team collaboration efficiency and task execution quality during the execution of multi-person collaborative tasks, and is applicable to various application scenarios such as group collaborative training, multi-person collaborative tasks, and human-machine collaborative operations.

[0009] To achieve the above-mentioned objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a closed-loop adjustment method for multi-person collaborative tasks based on joint state, comprising the following steps: The system simultaneously collects EEG signals, functional near-infrared spectral signals, and motion posture signals of each member during a multi-person collaborative task as multimodal signals, and extracts multimodal features to construct the individual joint state and individual state reliability of each member. A collaborative relationship matrix is ​​constructed using individual joint states as nodes, individual state reliability as node weights, and task phases or role relationships as edge constraints. A group collaborative joint index is generated based on the collaborative relationship matrix, and the type of collaborative mismatch is identified when the group collaborative joint index is abnormal. Based on the type of coordination mismatch, a corresponding feedback regulation strategy is generated and executed. After the feedback is executed, multimodal signals are collected and the group coordination joint index is updated to form a closed-loop regulation.

[0010] Preferably, the extraction of multimodal features to construct the joint state and individual state reliability of each member includes: Multimodal features are obtained by feature extraction from multimodal signals. After task-stage normalization, the quality confidence of each modality feature is calculated. The intensity of motion artifacts is calculated based on the motion posture signal to correct the quality confidence of EEG modality and functional near-infrared spectroscopy modality. After mapping each modal feature to a unified state space, dynamic fusion weights are calculated based on the quality confidence. The dynamic fusion weights are then used to weight and fuse each modal feature to obtain the individual joint state. Finally, the reliability of the individual state is output based on the quality confidence of each modal feature and the dynamic fusion weights.

[0011] Preferably, the step of calculating the dynamic fusion weights based on the quality confidence level includes: The quality confidence scores and base weights of each modality in the current task are obtained for the EEG modality, functional near-infrared spectroscopy modality, and motion posture modality. The product of the base weight and quality confidence score of each modality is divided by the sum of the products of the base weight and quality confidence scores of all modalities to obtain the dynamic fusion weight of each modality.

[0012] Preferably, the generation of group collaborative joint indicators based on the collaborative relationship matrix includes: The group's collaborative consistency and members' collaborative contribution are calculated based on the collaborative relationship matrix. The balance of members' contributions is calculated based on the members' collaborative contribution. The group's collaborative stability and task performance are obtained and aggregated into a joint index of group collaboration. Among them, group synergy consistency is calculated based on the quality-weighted average of all effective synergy relationships in the synergy relationship matrix; member synergy contribution is calculated based on the quality-weighted average of the synergy relationships between each member and all other members in the synergy relationship matrix; member contribution balance is calculated based on the ratio of the standard deviation to the mean of the member synergy contribution; group synergy stability is calculated based on the degree of fluctuation of group synergy consistency; and task performance is obtained by normalized aggregation of accuracy, reaction time, and task completion status.

[0013] Preferably, identifying the type of collaboration mismatch when the group collaboration joint index is abnormal includes: When the group's collaborative joint index is below the target range or fluctuates abnormally, the mismatch type identification process is initiated. The collaborative mismatch type is identified based on modal collaborative consistency, contribution of each modality, balance of member contributions, task performance, reliability of individual states, and intensity of motion artifacts in motion posture signals. Modal coherence is calculated based on the modal features of each member in the EEG modality, functional near-infrared spectroscopy modality, and motor posture modality. The feature similarity between pairs of members in the same modality is calculated, and the feature similarity of all valid member pairs is obtained by quality-weighted averaging. The modal contribution is calculated based on the state vectors of each member in the EEG modality, functional near-infrared spectroscopy modality, and motor posture modality after unified state space mapping. The amplitude of the state vector of each modality is calculated after dynamic fusion weighting, and the amplitude is divided by the sum of the weighted amplitudes of all modalities of that member.

[0014] Preferably, the type of cooperative mismatch includes at least one of the following: Signal quality anomalies are manifested as individual state reliability being lower than the first threshold, motion artifact intensity being higher than the second threshold, or group quality confidence being lower than the third threshold; this type is prioritized for judgment, and if signal quality anomalies are determined, the following other mismatch types will not be judged. Attention is insufficient coordination, which is manifested as modal coordination consistency of EEG modalities being lower than the fourth threshold and modal contribution of EEG modalities being higher than the fifth threshold; Uneven cognitive load is manifested as modal coherence of functional near-infrared spectral modes being lower than the sixth threshold or member contribution balance being lower than the seventh threshold, and modal contribution of functional near-infrared spectral modes being higher than the eighth threshold. Asynchronous behavior rhythm is manifested as modal coordination consistency of motion posture modality being lower than the ninth threshold and modal contribution of motion posture modality being higher than the tenth threshold. Brain-behavior mismatch manifests as a discrepancy between group coordination and task performance; Role imbalance is manifested in the fact that the balance of member contributions is lower than the eleventh threshold and there are members whose collaborative contributions are lower than the twelfth threshold.

[0015] Preferably, the step of generating and executing a corresponding feedback adjustment strategy based on the type of cooperative mismatch includes: When the type of coordination mismatch is signal quality abnormality, a wear check or recalibration prompt is generated, and the weight of abnormal members or abnormal modes in the subsequent group coordination joint index calculation is reduced without immediately adjusting the task difficulty. When the type of coordination mismatch is insufficient attention coordination, uneven cognitive load, asynchronous behavioral rhythm, brain-behavior mismatch, or role imbalance, software prompts, rhythm prompts, division of labor adjustments, rule prompts, or role reassignment strategies corresponding to the mismatch type are generated according to the preset correspondence between the coordination mismatch type and the feedback strategy.

[0016] Secondly, embodiments of the present invention also provide a multi-person collaborative task closed-loop adjustment system based on joint state, used to implement the above-mentioned multi-person collaborative task closed-loop adjustment method based on joint state, including: The signal synchronization acquisition module is used to simultaneously acquire the EEG signals, functional near-infrared spectral signals and motion posture signals of each member during a multi-person collaborative task as multimodal signals; The individual state construction module is used to extract multimodal features based on multimodal signals to construct the joint individual state and individual state reliability of each member; The group collaboration analysis module is used to construct a collaboration relationship matrix with individual joint states as nodes, individual state reliability as node weights, and task stages or role relationships as edge constraints. Based on the collaboration relationship matrix, a group collaboration joint index is generated, and when the group collaboration joint index is abnormal, the type of collaboration mismatch is identified. The closed-loop feedback regulation module is used to generate and execute corresponding feedback regulation strategies based on the type of coordination mismatch. After the feedback is executed, it continues to collect multimodal signals and update the group coordination joint indicators to form a closed-loop regulation.

[0017] Thirdly, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described closed-loop adjustment method for multi-person collaborative tasks based on joint state when executing the computer program.

[0018] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the aforementioned closed-loop adjustment method for multi-person collaborative tasks based on joint state.

[0019] Compared with the prior art, the beneficial effects of the present invention include at least the following: (1) This invention collects the EEG signals, functional near-infrared spectral signals and motion posture signals of each member, generates individual joint states and individual state reliability and constructs a collaborative relationship matrix, then generates group collaborative joint indicators and identifies collaborative mismatch types, and finally executes feedback adjustment strategies, realizing a closed loop of the entire process of collection-fusion-evaluation-feedback, supporting group synchronous mode and group asynchronous mode, so as to adjust the task process in real time according to the group collaborative state, and is suitable for various scenarios such as group collaborative training, individual differentiated training and scientific research paradigm verification.

[0020] (2) This invention extracts multimodal features, maps neural electrical activity, blood oxygen dynamics changes and behavioral rhythms to the same state space, corrects the quality confidence of each modality based on the intensity of motion artifacts, calculates dynamic fusion weights based on quality confidence, realizes quality-gated dynamic fusion, effectively suppresses the interference of low-quality data on fusion results, and improves the reliability of individual joint state and group collaborative joint index calculation.

[0021] (3) Based on the collaborative relationship matrix, this invention constructs a group collaborative joint index. While outputting the overall collaborative score, it also outputs multi-dimensional evaluation information such as pairwise collaborative relationships, member collaborative contribution, modal contribution, collaborative stability and task performance, making the collaborative evaluation interpretable. On this basis, when the group collaborative joint index is abnormal, different collaborative mismatch types are identified, and corresponding feedback adjustment strategies are generated accordingly, which significantly improves the robustness and practicality of the system. At the same time, it can retain complete data storage and report export functions, which is convenient for training evaluation and scheme optimization. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the closed-loop adjustment method for multi-person collaborative tasks based on joint state provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the framework of the multi-person collaborative task closed-loop adjustment method based on joint state provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of the structure of a multi-person collaborative task closed-loop regulation system based on joint state provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0025] The inventive concept of this invention is as follows: Addressing the problems in existing technologies such as time scale mismatch, quality anomalies contaminating fusion results, unexplainable modal contributions, and lack of targeted feedback or even misjudgment in closed-loop adjustment due to failure to distinguish specific causes of mismatch, this invention provides a method and system for closed-loop adjustment of multi-person collaborative tasks based on joint states. Through a quality-gated dynamic fusion mechanism, the dynamic fusion weights are adjusted in real time according to the quality confidence of each modal signal to construct a reliable individual joint state. Based on the individual joint states, a collaborative relationship matrix is ​​constructed, and a group collaborative joint index is generated. Furthermore, before closed-loop adjustment, the type of collaborative mismatch is identified, and a corresponding feedback adjustment strategy is generated based on the mismatch type. This invention effectively avoids misjudgment problems in multi-modal fusion and collaborative evaluation, achieving accurate assessment and targeted closed-loop adjustment of multi-person collaborative states, significantly improving team collaboration efficiency and task execution quality.

[0026] like Figure 1 and Figure 2 As shown in the embodiment, a closed-loop adjustment method for multi-person collaborative tasks based on joint state is provided, including the following steps: S1 synchronously collects the EEG signals, functional near-infrared spectral signals, and motion posture signals of each member during a multi-person collaborative task as multimodal signals, and extracts multimodal features to construct the individual joint state and individual state reliability of each member.

[0027] S1.1, Synchronous acquisition of multimodal signals.

[0028] In the embodiment, there are a total of [number] people in the multi-person collaborative task. Member Name, Member Index Using the sliding analysis window as the basic calculation unit, the first... window Represented as: , in, For window length, For the first The start time of each window. Task events or task phases are represented as... This includes one or more of the following: task start, task switching, stimulus presentation, role switching, feedback output, difficulty change, successful collaboration, failed collaboration, or manual annotation.

[0029] Each member is equipped with at least one multimodal detection headband, which integrates an EEG acquisition unit, a near-infrared acquisition unit, and a motion posture acquisition unit. The EEG acquisition unit acquires changes in neural electrical activity, obtaining EEG signals; the near-infrared acquisition unit acquires cerebral oxygenation dynamics information such as oxyhemoglobin (HbO) and / or deoxyhemoglobin (HbR), obtaining functional near-infrared spectroscopy (fNIRS) signals; the motion posture acquisition unit acquires head or body movement information, including acceleration, angular velocity, posture angle, movement amplitude, or movement rhythm, obtaining motion posture signals. The multimodal signal is composed of the EEG signal, fNIRS signal, and motion posture signal.

[0030] The user software receives data from the acquisition terminal via Bluetooth and checks the device's battery level, connection status, wearing status, and signal quality. The control software receives data uploaded by one or more user terminals via a local area network and manages the task plan uniformly in either a group synchronous mode or a group asynchronous mode. To ensure that data from multiple users can be used for collaborative analysis, the system writes a device number, user number, sampling time, data type, and sequence number to each data packet.

[0031] The multimodal signals of all members are written to the same task timeline. The task system synchronously writes event markers when task start, task switch, task stage change, prompt output, difficulty adjustment, successful collaboration, failed collaboration, or manual annotation occurs. For data with different sampling rates, resampling, windowing, and buffer alignment are employed to enable EEG signals, fNIRS signals, and motion attitude signals to participate in joint state calculation within the same analysis window.

[0032] EEG signals were processed using short time windows to capture rapid attentional fluctuations, fNIRS signals were processed using time windows that accounted for blood oxygenation response lag, and motion posture signals were processed using time windows matched to the task's rhythm. The windowed features of each modality were normalized to comparable feature vectors for the same task phase. For asynchronous group scenarios at different task phases, feature vectors were determined based on event markers. Instead of forcing a comparison of all members at the same moment in absolute time, different members should be aligned by stage or role.

[0033] S1.2, Multimodal Feature Extraction.

[0034] In this embodiment, signal preprocessing and feature extraction are performed according to the methods listed in Table 1: Table 1 Signal Preprocessing and Feature Extraction

[0035] S1.3, Construction of Individual Joint State.

[0036] In this embodiment, the individual joint state is not a simple concatenation or fixed weighting of EEG indicators, fNIRS indicators and motion posture indicators. Instead, within a unified time window, it comprehensively considers blood oxygen response delay, individual baseline differences, task stage differences, motion artifact effects and the quality of each modality signal to form an individual state vector that can be used for subsequent multi-person collaborative analysis.

[0037] No. Members in The EEG features, fNIRS features, and motor posture features within each window are represented as follows: , in, Indicates modal index, Represents a set of modes. Indicates EEG mode, Indicates the fNIRS mode. This represents the motion posture mode. The three types of inputs mentioned above are first aligned within a unified timeline and task event window, and then proceed to the individual joint state construction process.

[0038] Because the blood oxygenation response of fNIRS lags physiologically relative to the neural electrical activity of EEG, this invention first performs delay compensation on the fNIRS features before fusion, and unifies the three modal features as follows: , in, For the first Blood oxygen response delay compensation parameters for each member. When using window indexing for calculations, this parameter is converted into the corresponding number of analysis windows; it can be preset by the system or determined by members. The time difference between EEG activation changes, task event markers, and HbO / HbR response peaks during the baseline calibration or historical training phases is adaptively determined.

[0039] To mitigate the impact of individual baseline differences, task phase differences, and differences in different modal dimensions, based on the current task event or task phase... Individual normalization is performed on the three types of features: , in, The features are normalized. and They represent the first Members during the mission phase Next The baseline mean and standard deviation of modal characteristics, To prevent constants with zero denominators, each member, task stage, and modal characteristic is converted into comparable relative changes.

[0040] After time compensation and normalization are completed, the raw quality confidence of each mode within the current window is calculated. : , in, A larger value indicates that the mode is more reliable. Indicates the first The first class modality The abnormal indicators are normalized to a value in the range of [0,1], and the larger the value, the more severe the abnormality. This represents the influence weight of the corresponding abnormal indicator. It is a non-negative value and can be preset based on the degree of influence of each abnormal indicator on signal quality or determined from historical data. This represents a cutoff function used to restrict input values ​​to the interval [0,1]: 0 for input values ​​less than 0, 1 for input values ​​greater than 1, and otherwise unchanged. For EEG, anomalies include abnormal electrode contact, power frequency interference, amplitude saturation, and artifacts in electrooculography or electromyography; for fNIRS, anomalies include excessively low light intensity or saturation, sudden changes in light density, reduced signal-to-noise ratio, and data packet loss; for motion posture, anomalies include attitude sensor packet loss, abnormal jumps, or unstable attitude estimation.

[0041] Motion posture signals play a dual role in this invention: firstly, motion posture serves as information for behavioral rhythm and action synchronization, entering the individual's joint state; secondly, motion posture is also a basis for determining the source of artifacts in EEG and fNIRS. The first... Motion artifact intensity of the current window of the member : , in, and These represent the acceleration and angular velocity within the current window, respectively. and These represent the mean and standard deviation of exercise intensity at baseline or under stable wearing conditions, respectively. This represents the root mean square norm of the triaxial acceleration or triaxial angular velocity vector sequence within the current window; the sum of these two values ​​represents the overall motion intensity of the current window. The baseline mean and standard deviation are calculated using the same method for overall motion intensity. To prevent the standard deviation from being zero, a positive number is set. Clip(·,0,1) is used to restrict the standardized result to the interval [0,1]. This is used to normalize the intensity of motion artifacts. The larger the value, the more obvious the head or body movement is within the current window, and the more likely it is to cause artifact interference to EEG and fNIRS.

[0042] Therefore, utilizing Correcting the raw quality confidence levels for EEG and fNIRS: , in, The corrected effective quality confidence level; middle, and Let represent the influence coefficients of motion artifacts on the confidence scores of EEG and fNIRS quality (e.g., values ​​of 0.6 and 0.5 respectively). For motion attitude modes, we can let ,Right now .

[0043] To map EEG, fNIRS, and motion pose to a fusionable common state space, state space mapping is performed on the normalized features of each modality: , in, and They represent the first The mapping matrix and bias terms for the modalities. After mapping, What it represents , and Located in the same state space, they respectively represent the EEG state, blood oxygen state, and motor behavior state within the current window.

[0044] After obtaining the state representations of each modality, the quality-gated dynamic fusion weights are calculated based on the effective quality confidence scores. : , in, Indicates the first The basic weight of a modality in the current task. The effective quality confidence score when a modality exhibits contact anomalies, light intensity anomalies, head movement artifacts, or missing data. Reduce, corresponding fusion weight Automatically decreases; when the modal quality is stable, its contribution to the joint state of individuals increases accordingly.

[0045] Based on this, calculate the initial joint state of individuals within the current window. : , To avoid abrupt state changes caused by transient noise within a single window, further time smoothing is performed to obtain the final output joint state of the individuals. : , in, This represents the time smoothing coefficient.

[0046] At the same time, output members Individual state reliability within the current window : , This indicator reflects The credibility of this. Because Simultaneously affected by the quality of the original signal, the intensity of motion artifacts, and the fusion weights, the influence of a member in subsequent group collaborative computation will be reduced when a member's head movement is too strong, the fNIRS light intensity is abnormal, or the EEG contact is abnormal.

[0047] To improve interpretability, the contribution of each modality to the joint state of the individual is also output: , in, , and Represent the effects of EEG, fNIRS, and motion posture on the first... The degree of contribution of each member to the current state of the alliance. The second norm of a vector; This represents the modal index to be calculated. This represents the modal index traversed during summation. The numerator of the formula represents the modal set consisting of EEG, fNIRS, and motion posture; the numerator of the formula represents the first... The magnitude of the modal state vector after dynamic fusion weighting, where the denominator represents the sum of the weighted magnitudes of all modalities. A positive number is used to prevent the denominator from being zero. Therefore, this contribution represents the first... The relative contribution of a modality to the current joint state of individuals, taking values ​​in the interval [0,1], is... When the contribution of the three modes is negligible, the sum of their contributions is 1.

[0048] In summary, this process, through steps such as blood oxygenation delay compensation, individual normalization, quality confidence calculation, motion artifact correction, unified state space mapping, quality-gated dynamic fusion, and time-smoothed output, generates an individual EEG-blood oxygenation-motor combined state for each member. This state vector simultaneously contains information on neural electrical activity, blood oxygen metabolism response, and motor behavior rhythm. Furthermore, a quality gating mechanism reduces the impact of abnormal modalities on the fusion results, thus providing reliable input for subsequent calculation of group collaborative joint indicators, identification of collaborative mismatch types, and closed-loop feedback control.

[0049] S2 constructs a collaborative relationship matrix with individual joint states as nodes, individual state reliability as node weights, and task stages or role relationships as edge constraints. Based on the collaborative relationship matrix, a group collaborative joint index is generated. When the group collaborative joint index is abnormal, the type of collaborative mismatch is identified.

[0050] S2.1, Construction of joint indicators for group collaboration.

[0051] In this embodiment, after obtaining the individual joint state of each member, the consistency of the joint state among multiple members is further calculated. Unlike directly calculating interbrain synchronization from a single EEG or fNIRS modality, this invention uses individual joint states... As a node in a group collaborative network, As a measure of node reliability, task phases or role relationships are used as edge weight constraints to construct quality-weighted group collaboration relationships.

[0052] For the Name and No. For each member, the collaborative comparison weight within the current window is first determined based on the task event, plan stage, or role relationship: , When two members are in the same task phase, have a direct collaborative relationship, or need to perform synchronous comparisons... Higher; when the two are at different stages of a task, in different roles, or do not need to be directly compared, It is low or zero. Therefore, it can simultaneously support synchronous group training, asynchronous group training, and role-based collaborative tasks.

[0053] Further define the joint state similarity between two members : , in, The joint state similarity function can be represented by cosine similarity, correlation coefficient, dynamic time warp similarity, coherence, or mutual information, etc. and Each represents a member and members The individual joint state.

[0054] Based on this, the quality-weighted collaborative relationship between the two members Represented as: , in, and Each represents a member and members Individual state reliability is used to reduce the impact of members with abnormal signal quality on pairwise cooperation. If a member's excessive head movement leads to... If it rises, then its , , Consequently, all those containing that member decrease. It will also be demoted.

[0055] The collaboration matrix of the current window is composed of all pairwise collaboration relationships. : , In this matrix, the elements represent the cooperative strength among different members based on their individual joint states. The number of group members is represented by the collaboration matrix. matrix.

[0056] Calculate group cohesion consistency based on the cohesion relationship matrix. : , This metric represents the overall consistency among all active collaborators within the current window. Unlike a simple average of all pairwise similarities, The system takes into account both task phase relationships and member status reliability, so members at different phases will not be incorrectly compared, and low-quality members will not cause excessive interference to group collaborative judgment.

[0057] To determine whether any individual members have deviated from the group, the calculation of the... Member collaborative contribution : , in, The lower the value, the better. The weaker the collaborative contribution of a member relative to other relevant members, the lower the synergy. Based on all members... Calculate the balance of member contributions : , in, This represents the standard deviation of the above sequence of collaborative contributions. This represents the arithmetic mean of the above sequence of collaborative contributions from the members. A positive number set to prevent the denominator from being zero; the ratio of standard deviation to mean represents the relative dispersion of member contributions. The calculation results are restricted to the interval [0,1]. The higher the value, the more balanced the contributions of each member to group collaboration; The lower the value, the more likely there are members falling behind, uneven role loads, or unbalanced collaboration relationships.

[0058] To avoid misjudgment caused by noise in a single window, calculate continuous Cooperative stability within a window : , in, This indicates the maximum permissible fluctuation scale obtained from system presets or historical statistics. The higher the value, the more stable the group's collaborative state.

[0059] Task performance Accuracy During the reaction Number of errors Completion Time Completion status at current difficulty level Normalization of the indicators yields: , In this study, all performance components for each task are normalized to a unidirectional index, with higher values ​​indicating better performance; reaction time, number of errors, and completion time are inversely normalized. All weights are non-negative. If certain application scenarios do not include explicit task performance, then... The aggregate weight is set to 0, and the remaining weights are renormalized; the task performance can still be used only in the reporting and mismatch identification stages.

[0060] Ultimately, a group collaborative joint indicator is generated. : , in, It should be noted that, This reflects the overall consistency, stability, member balance, and task performance of the group's collaborative state within the current task window. Quality factors have been... Enter , and .

[0061] It can also output the population quality confidence level. : , in, Mainly used to determine the current Whether it is reliable, and whether it can prioritize the identification of signal quality anomalies in subsequent mismatch identification, rather than repeating them as [other methods]. Independent weighted terms in the main formula.

[0062] S2.2, Identification of Cooperative Mismatch Types.

[0063] In this embodiment, it is first determined whether mismatch identification needs to be initiated. The mismatch type identification process proceeds when any of the following conditions are met: , in, Indicators representing group collaboration The lower threshold (e.g., 0.6). This indicates the allowed mutation magnitude (e.g., 0.15). This represents the threshold for cooperative stability (e.g., 0.7). If none of the above conditions are met, the current window can be determined to be in a normal cooperative state.

[0064] To ensure that mismatch types correspond to cooperative structures, a similarity score is defined between each modal state. : , in, This represents the similarity function.

[0065] And calculate the modal cooperative consistency of each mode at the group level: , in, , and These represent the group-level coherence of EEG, fNIRS, and motion posture, respectively.

[0066] (1) Mismatch identification prioritizes judging abnormal signal quality. If the reliability of the individual state... Lower motion artifact strength High or quality confidence Lower, meets the requirements: , in, The first threshold is set, for example, 0.6; The second threshold is, for example, 0.7; The third threshold is, for example, 0.65.

[0067] Then, the mismatch type should be determined first. for: , in, This indicates an abnormal signal quality. This judgment must be performed first, as head movement, improper wearing, light intensity saturation, poor contact, or data loss can cause false low signal levels. If quality anomalies are not ruled out first, signal problems may be misdiagnosed as coordination problems, and the task difficulty may be incorrectly reduced.

[0068] If no obvious signal quality anomalies are found, the specific mismatch type is further identified based on modal cooperation consistency, modal contribution, member contribution balance, and task performance. For ease of description, the group average modal contribution is defined. : , (2) When EEG modal contribution The overall level is relatively high, but the group coherence at the EEG level is relatively low. And the intensity of motion artifacts If the threshold is not exceeded, it can be determined as insufficient attentional coordination. : , in, The fourth threshold is, for example, 0.55; The fifth threshold is, for example, 0.35. This type indicates that group members do not exhibit consistency in their attentional states, alertness levels, or neural electrical activity rhythms, rather than due to obvious motor artifacts.

[0069] (3) When fNIRS mode contribution High modal coherence at the fNIRS level Low, or uneven distribution of member contributions A decrease can be identified as an uneven distribution of cognitive load. : , in, This represents the sixth threshold, for example, 0.55; This represents the seventh threshold, for example, 0.7; This indicates the eighth threshold, for example, 0.35. This type indicates that some members may be under a higher cognitive load, or that there are significant differences in blood oxygenation response intensity, recovery speed, and task load status among different members.

[0070] (4) When the motion posture mode contributes High modal coherence at the motion posture level Lower, but motion artifact strength If the quality anomaly threshold is not reached, it can be determined that the behavior rhythm is out of sync. : , in, This represents the ninth threshold, for example, 0.55; This indicates the tenth threshold, for example, 0.35. This type indicates inconsistencies in movement rhythm, reaction time, operation interval, or posture changes among members. It's important to distinguish between these. and The former reflects whether the behavioral rhythm is synchronized, while the latter reflects whether the movement posture causes EEG / fNIRS artifacts. If Exceeding the threshold should be prioritized as Rather than a mismatch in behavior rhythm.

[0071] (5) When the group's joint state appears to be coordinated but the task performance is poor, or the task performance is acceptable but the joint state coordination is obviously insufficient, it can be judged as brain-behavior mismatch. : , in, Indicates group coordination and consistency The upper limit threshold, for example, 0.7; Indicates task performance The lower threshold is set, for example, 0.6. This type is used to identify inconsistencies between EEG, blood oxygenation, and motor coordination status and explicit task performance, avoiding unilateral judgments based solely on neural synergy or task performance.

[0072] (6) When the contributions of members are balanced A significant decrease in collaborative contribution, where one member's contribution is significantly lower than that of other members, or where a member's current task role is mismatched with their collaborative contribution, can be considered an imbalance in members or roles. : , Then we have: , in, This represents the eleventh threshold, for example, 0.7; Defined as the twelfth threshold that adaptively changes with the distribution of contributions from group members, it is the current average contribution of group members minus the deviation from the decision coefficient by one standard deviation; if necessary, the result is truncated to the [0,1] interval. This dynamic threshold can be denoted as the role contribution threshold. Indicates the first Member collaborative contribution of each member This represents the average contribution of each member. This represents the deviation judgment coefficient, with a value of 1.0. This type is mainly used to identify situations in multi-person collaborative tasks where individual members continuously fall behind, participate insufficiently, have a mismatch between role workload and actual contribution, or have an unbalanced division of labor within the team.

[0073] In practical task systems, It can be a single type or a combination of multiple labels. If multiple mismatch conditions are met simultaneously, the priority can be set as follows: abnormal signal quality first, followed by asynchronous behavioral rhythm, uneven cognitive load, insufficient attentional coordination, brain-behavior mismatch, and role imbalance; the priority can also be adjusted according to the task type.

[0074] S3 generates and executes a corresponding feedback regulation strategy based on the type of coordination mismatch. After the feedback is executed, it continues to collect multimodal signals and update the group coordination joint index to form a closed-loop regulation.

[0075] In determining the type of mismatch Then, based on the group synergy joint indicators Mismatch type Task Phase Task performance Individual state reliability Member contribution and historical feedback strategy u ( k -1), Generate the feedback strategy for the current window. : , in, This represents the function that generates the feedback strategy.

[0076] To ensure a clear correspondence between feedback strategies and types of cooperation mismatch, This can be further expressed as: , in, This indicates a wear check, recalibration, pausing difficulty changes, or reducing abnormal modal / member weights; This indicates attention to synchronized prompts, shared goal prompts, or a unified breathing / observation rhythm; This indicates a redistribution of tasks, reduction of task complexity for high-load members, or adjustment of roles; This indicates a rhythmic prompt, a need to extend reaction time, or a reduction in required operating speed. This indicates a rule prompt, communication prompt, or task phase transition; This indicates a reassignment of roles, adjustment of task allocation, increasing the participation of low-participation members, or reducing the task complexity of high-load members; Indicates the current Maintain the current task state while within the target range.

[0077] It should be noted that when At that time, do not lower This can be directly interpreted as a decline in team collaboration. This type of mismatch is typically caused by the reliability of individual states. Reduce motion artifact intensity Increased, effective mass of EEG Reduce or fNIRS effective quality This is caused by a decrease. Especially when motion attitude signals are the source of artifacts, they are caused by… The effective quality confidence of EEG and fNIRS is adjusted, and the individual state reliability is further affected. This reduces the number of abnormal members. and The impact on the calculation. Therefore, at this point, priority is given to outputting wear checks, recalibrating, pausing difficulty changes, or reducing abnormal modal / member weights, rather than immediately reducing the task difficulty.

[0078] To avoid frequent fluctuations in task difficulty, a target coordination interval is set. (e.g., [0.60, 0.80]), number of consecutive windows Task performance threshold (e.g., 0.60), Cooperative stability threshold (e.g., 0.70) and minimum holding time (For example, 30 seconds). When continuously The difficulty of the task can be increased when multiple windows satisfy the following formula: , And recently If frequent feedback switching does not occur within a certain time, strategies to increase task difficulty, advance stages, or increase training complexity will be generated.

[0079] When continuous When a window satisfies the following formula, it enters the low-coordination processing flow: , If at this time Then execute This includes wearing checks, recalibrating, pausing changes in difficulty, or reducing abnormal modalities / member weights; if Based on the mismatch type, corresponding software prompts, rhythm prompts, role adjustments, rule prompts, or task phase switches will be output. If after continuous feedback... It has not yet recovered to the target range, and the task performance... If the response time remains below the threshold, the task difficulty can be further reduced or the reaction time extended.

[0080] when And task performance If no abnormal decrease occurs, maintain the current task state and output. After the feedback is executed, the system continues to collect EEG, fNIRS, and motion attitude signals for the next window, and recalculates the individual joint state. Individual state reliability Group collaboration joint indicators and coordination mismatch type This forms a closed-loop process of data collection, fusion, collaborative evaluation, mismatch identification, feedback adjustment, and re-collection.

[0081] Furthermore, the control unit can display each member's brain activation, focus, relaxation, real-time heart rate, and device signal quality in real time in the training view. It can also view 3D brain heatmaps, full-band EEG power spectra, raw waveforms, time-frequency analysis, frequency domain analysis, and motion posture data in the multimodal view. Building upon the aforementioned views, this invention further enhances visualization at the multi-person collaboration level, including Group Collaboration Index (GCI) curves, pairwise collaboration matrices, member contributions, modal contributions, periods of low collaboration, collaboration mismatch types, software prompt records, task difficulty change trajectories, and recovery status after feedback.

[0082] After the task is completed, the raw data, preprocessed features, event markers, GCI curves, feedback logs, and task performance data are stored uniformly, and CSV data and evaluation / training reports can be exported. The report content may include statistical results, raw waveforms, time-frequency analysis, frequency domain analysis, trend analysis, population cooperative changes, main mismatch types, and closed-loop feedback effects.

[0083] In one application scenario, a group of 2-15 members is selected. The control terminal chooses a group synchronization mode, binding each member to their corresponding user device. The user device guides members to wear multimodal detection headbands and complete signal quality calibration. The control terminal selects training programs such as group breathing, group body scanning, group cooperation, or group competition, and distributes these programs to all user devices. During training, EEG, fNIRS, and head movement posture data of each member are collected synchronously. The control terminal calculates the joint state of each member in real time and outputs group coordination indicators. .when When the readings are below the target range, first determine the source of the low coordination: if there are significant differences in EEG attention-related features, it suggests synchronized attention and breathing rhythm across the entire group; if fNIRS shows excessive workload for a particular member, it suggests task reassignment or reducing cognitive complexity; if movement postures show asynchronous rhythm, it provides rhythmic cues; if signal quality is abnormal, it suggests checking the wearer's equipment without immediately changing the task difficulty. Continue collecting and comparing data after receiving feedback. Recovery status.

[0084] Based on the same inventive concept, such as Figure 3As shown, this embodiment of the invention also provides a multi-person collaborative task closed-loop adjustment system 300 based on joint state, including: a signal synchronization acquisition module 310, an individual state construction module 320, a group collaborative analysis module 330, and a closed-loop feedback adjustment module 340.

[0085] The signal synchronization acquisition module 310 is used to synchronously acquire the EEG signals, functional near-infrared spectral signals and motion posture signals of each member during a multi-person collaborative task as multimodal signals.

[0086] The individual state construction module 320 is used to extract multimodal features based on multimodal signals to construct the joint individual state and individual state reliability of each member.

[0087] The group collaboration analysis module 330 is used to construct a collaboration relationship matrix with individual joint states as nodes, individual state reliability as node weights, and task stages or role relationships as edge constraints. Based on the collaboration relationship matrix, a group collaboration joint index is generated, and when the group collaboration joint index is abnormal, the type of collaboration mismatch is identified.

[0088] The closed-loop feedback regulation module 340 is used to generate and execute corresponding feedback regulation strategies based on the type of coordination mismatch. After the feedback is executed, it continues to collect multimodal signals and update the group coordination joint index to form a closed-loop regulation.

[0089] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described closed-loop adjustment method for multi-person collaborative tasks based on joint state when executing the computer program.

[0090] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described closed-loop adjustment method for multi-person collaborative tasks based on joint state.

[0091] It should be noted that the multi-person collaborative task closed-loop adjustment system, electronic device, and computer-readable storage medium based on joint state provided in the above embodiments all belong to the same inventive concept as the multi-person collaborative task closed-loop adjustment method based on joint state. For details of their specific implementation process, please refer to the embodiments of the multi-person collaborative task closed-loop adjustment method based on joint state, which will not be repeated here.

[0092] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A closed-loop adjustment method for multi-person collaborative tasks based on joint state, characterized in that, Includes the following steps: The system simultaneously collects EEG signals, functional near-infrared spectral signals, and motion posture signals of each member during a multi-person collaborative task as multimodal signals, and extracts multimodal features to construct the individual joint state and individual state reliability of each member. A collaborative relationship matrix is ​​constructed using individual joint states as nodes, individual state reliability as node weights, and task phases or role relationships as edge constraints. A group collaborative joint index is generated based on the collaborative relationship matrix, and the type of collaborative mismatch is identified when the group collaborative joint index is abnormal. Based on the type of coordination mismatch, a corresponding feedback regulation strategy is generated and executed. After the feedback is executed, multimodal signals are collected and the group coordination joint index is updated to form a closed-loop regulation.

2. The closed-loop adjustment method for multi-person collaborative tasks based on joint state as described in claim 1, characterized in that, The extraction of multimodal features to construct the joint state and individual state reliability of each member includes: Multimodal features are obtained by feature extraction from multimodal signals. After task-stage normalization, the quality confidence of each modality feature is calculated. The intensity of motion artifacts is calculated based on the motion posture signal to correct the quality confidence of EEG modality and functional near-infrared spectroscopy modality. After mapping each modal feature to a unified state space, dynamic fusion weights are calculated based on the quality confidence. The dynamic fusion weights are then used to weight and fuse each modal feature to obtain the individual joint state. Finally, the reliability of the individual state is output based on the quality confidence of each modal feature and the dynamic fusion weights.

3. The closed-loop adjustment method for multi-person collaborative tasks based on joint state as described in claim 1 or 2, characterized in that, The calculation of dynamic fusion weights based on quality confidence includes: The quality confidence scores and base weights of each modality in the current task are obtained for the EEG modality, functional near-infrared spectroscopy modality, and motion posture modality. The product of the base weight and quality confidence score of each modality is divided by the sum of the products of the base weight and quality confidence scores of all modalities to obtain the dynamic fusion weight of each modality.

4. The closed-loop adjustment method for multi-person collaborative tasks based on joint state according to claim 1 or 2, characterized in that, The generation of group collaborative joint indicators based on the collaborative relationship matrix includes: The group's collaborative consistency and members' collaborative contribution are calculated based on the collaborative relationship matrix. The balance of members' contributions is calculated based on the members' collaborative contribution. The group's collaborative stability and task performance are obtained and aggregated into a joint index of group collaboration. Among them, group synergy consistency is calculated based on the quality-weighted average of all effective synergy relationships in the synergy relationship matrix; member synergy contribution is calculated based on the quality-weighted average of the synergy relationships between each member and all other members in the synergy relationship matrix; member contribution balance is calculated based on the ratio of the standard deviation to the mean of the member synergy contribution; group synergy stability is calculated based on the degree of fluctuation of group synergy consistency; and task performance is obtained by normalized aggregation of accuracy, reaction time, and task completion status.

5. The closed-loop adjustment method for multi-person collaborative tasks based on joint state according to claim 4, characterized in that, The method for identifying the type of collaboration mismatch when the group's collaborative joint indicators are abnormal includes: When the group's collaborative joint index is below the target range or fluctuates abnormally, the mismatch type identification process is initiated. The collaborative mismatch type is identified based on modal collaborative consistency, contribution of each modality, balance of member contributions, task performance, reliability of individual states, and intensity of motion artifacts in motion posture signals. Modal coherence is calculated based on the modal features of each member in the EEG modality, functional near-infrared spectroscopy modality, and motor posture modality. The feature similarity between pairs of members in the same modality is calculated, and the feature similarity of all valid member pairs is obtained by quality-weighted averaging. The modal contribution is calculated based on the state vectors of each member in the EEG modality, functional near-infrared spectroscopy modality, and motor posture modality after unified state space mapping. The amplitude of the state vector of each modality is calculated after dynamic fusion weighting, and the amplitude is divided by the sum of the weighted amplitudes of all modalities of that member.

6. The closed-loop adjustment method for multi-person collaborative tasks based on joint state according to claim 5, characterized in that, The type of cooperative mismatch includes at least one of the following: Signal quality anomalies are manifested as individual state reliability being lower than the first threshold, motion artifact intensity being higher than the second threshold, or group quality confidence being lower than the third threshold; this type is prioritized for judgment, and if signal quality anomalies are determined, the following other mismatch types will not be judged. Attention is insufficient coordination, which is manifested as modal coordination consistency of EEG modalities being lower than the fourth threshold and modal contribution of EEG modalities being higher than the fifth threshold; Uneven cognitive load is manifested as modal coherence of functional near-infrared spectral modes being lower than the sixth threshold or member contribution balance being lower than the seventh threshold, and modal contribution of functional near-infrared spectral modes being higher than the eighth threshold. Asynchronous behavior rhythm is manifested as modal coordination consistency of motion posture modality being lower than the ninth threshold and modal contribution of motion posture modality being higher than the tenth threshold. Brain-behavior mismatch manifests as a discrepancy between group coordination and task performance; Role imbalance is manifested in the fact that the balance of member contributions is lower than the eleventh threshold and there are members whose collaborative contributions are lower than the twelfth threshold.

7. The closed-loop adjustment method for multi-person collaborative tasks based on joint state according to claim 6, characterized in that, The step of generating and executing a corresponding feedback adjustment strategy based on the type of cooperative mismatch includes: When the type of coordination mismatch is signal quality abnormality, a wear check or recalibration prompt is generated, and the weight of abnormal members or abnormal modes in the subsequent group coordination joint index calculation is reduced without immediately adjusting the task difficulty. When the type of coordination mismatch is insufficient attention coordination, uneven cognitive load, asynchronous behavioral rhythm, brain-behavior mismatch, or role imbalance, software prompts, rhythm prompts, division of labor adjustments, rule prompts, or role reassignment strategies corresponding to the mismatch type are generated according to the preset correspondence between the coordination mismatch type and the feedback strategy.

8. A multi-person collaborative task closed-loop adjustment system based on joint state, used to implement the multi-person collaborative task closed-loop adjustment method based on joint state as described in any one of claims 1 to 7, characterized in that, include: The signal synchronization acquisition module is used to simultaneously acquire the EEG signals, functional near-infrared spectral signals and motion posture signals of each member during a multi-person collaborative task as multimodal signals; The individual state construction module is used to extract multimodal features based on multimodal signals to construct the joint individual state and individual state reliability of each member; The group collaboration analysis module is used to construct a collaboration relationship matrix with individual joint states as nodes, individual state reliability as node weights, and task stages or role relationships as edge constraints. Based on the collaboration relationship matrix, a group collaboration joint index is generated, and when the group collaboration joint index is abnormal, the type of collaboration mismatch is identified. The closed-loop feedback regulation module is used to generate and execute corresponding feedback regulation strategies based on the type of coordination mismatch. After the feedback is executed, it continues to collect multimodal signals and update the group coordination joint indicators to form a closed-loop regulation.

9. An electronic device comprising a memory and one or more processors, the memory for storing a computer program, characterized in that, The processor is used to implement the multi-person collaborative task closed-loop adjustment method based on joint state as described in any one of claims 1 to 7 when executing a computer program.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a computer, it implements the multi-person collaborative task closed-loop adjustment method based on joint state as described in any one of claims 1 to 7.

Images