Multi-agent multimodal behavioral cognition screening and risk assessment system and method

Abstract

This invention discloses a multi-agent, multimodal behavioral cognition screening and risk assessment system and method, comprising: collecting and preprocessing gait, facial video, and voice signals of subjects; performing feature analysis on the preprocessed gait, facial video, and voice signals to generate corresponding gait, facial, and audio structured evidence units; calculating cross-modal evidence consistency scores and evidence conflict degrees based on the multimodal structured evidence set, performing cross-modal semantic fusion and multi-domain risk reasoning to generate comprehensive behavioral risk results and sub-domain risk results for the emotional, cognitive, and social domains; introducing human-in-the-loop verification when the risk assessment results meet preset trigger conditions to confirm or correct the risk assessment results; and generating a standardized assessment report by combining the fusion reasoning results and the human-in-the-loop verification results. This invention improves the stability, security, and interpretability of risk assessment in scenarios with modality loss, local anomalies, or inconsistent multimodal conclusions.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, multimodal behavior analysis and risk assessment, and in particular to a multi-agent multimodal behavior cognitive screening and risk assessment system and method. Background Technology

[0002] With the development of artificial intelligence technology, the automated assessment of behavioral and emotional health is gradually shifting from single-modal analysis to multimodal collaborative analysis. A person's emotional state, cognitive function, and social behavior are often simultaneously reflected in movement patterns, facial expression dynamics, and speech features. In scenarios such as community mental health and screening for cognitive impairment in the elderly, clinical and home-based assessments typically require non-invasive, repeatable, and remote technologies. Gait videos, facial videos, and audio signals, due to their simple acquisition methods and low dependence on the environment, have become the main feasible input sources for multimodal behavioral assessment.

[0003] To address the aforementioned needs, various AI-based evaluation schemes have been proposed in the existing technology, but the following problems still exist: (1) Single-modal methods are difficult to form stable comprehensive judgment results. Existing methods usually establish recognition models based on a single signal, such as emotion recognition based on facial video, emotion or depression recognition based on speech, and gait video analysis of motor retardation or Parkinson's-like gait. These methods are low in implementation cost, but the conclusions are easily affected by factors such as lighting, speaking style, and walking environment, and cannot cover states with heterogeneous symptoms in clinical practice such as depression, anxiety, and cognitive decline. Therefore, they are difficult to use as an independent comprehensive assessment tool.

[0004] (2) Multimodal fusion methods lack a unified structured evidence representation mechanism. Some multimodal methods jointly model video, audio or text features through end-to-end models, which can improve recognition performance to some extent. However, such methods usually fuse directly at the feature layer or decision layer. The output results from different modalities are still heterogeneous and lack a unified standardized intermediate evidence form. Therefore, it is difficult to support the unified organization, comparison, tracing and subsequent expansion of results from different modalities.

[0005] (3) Existing multi-branch or multi-agent methods lack cross-modal conflict handling and risk aggregation mechanisms. Some methods establish branch networks for different modalities or have multiple agents process different information sources, and then perform feature splicing, attention fusion, or text summarization at a higher level to output a single label, score, or diagnostic suggestion. However, such methods usually do not explicitly model the credibility differences, consistency relationships, and conflict levels between the outputs of different modalities, making it difficult to obtain stable overall risk results in the case of modality missing, local anomalies, or inconsistent multimodal conclusions.

[0006] Existing technologies (Zhou, Yucheng, Lingran Song, and Jianbing Shen. "MAM: Modular Multi-Agent Framework for Multi-Modal Medical Diagnosis via Role-Specialized Collaboration." arXiv preprint arXiv:2506.19835 (2025).) employ multiple pre-defined roles, with each agent analyzing medical images or case texts, ultimately generating diagnostic text suggestions from a master agent. However, these approaches primarily target collaborative diagnosis using medical images or case texts. The information exchanged between agents is mostly free text analysis results or task role outputs, lacking standardized structured evidence representations for behavioral signals, and failing to establish a cross-subdomain risk aggregation mechanism based on evidence credibility, consistency, and conflict.

[0007] Furthermore, existing large language models (LLMs) and multi-agent systems are mainly geared towards medical diagnostic scenarios, focusing on the collaboration of multi-source medical data or multi-step diagnostic processes. They emphasize the understanding and generation of multi-source medical information, but do not construct a unified structured evidence intermediate module for behavioral signals such as gait videos, facial expressions, and voice. They also lack collaborative risk reasoning and gating calibration mechanisms for the emotional, cognitive, and social domains. Summary of the Invention

[0008] This invention provides a multi-agent multimodal behavioral cognitive screening and risk assessment system and method based on structured evidence modeling and risk gating, which is used to solve the problems of heterogeneous multimodal behavioral signal analysis results, difficulty in unified organization and collaborative aggregation of cross-modal evidence, and lack of traceable security control for automatic assessment results in the prior art.

[0009] The technical solution of the present invention is as follows: A method for screening and assessing the cognitive behavior of multiple agents using multiple modalities, comprising the following steps: Step S1: Acquire and preprocess the subject's gait video, facial video, and voice signals; Step S2: Use the gait analysis sub-agent, the face analysis sub-agent, and the audio analysis sub-agent to perform feature analysis on the preprocessed gait video, face video, and voice signal, and generate corresponding gait, face, and audio structured evidence units. Step S3: Store each structured evidence unit in the structured evidence intermediate module to form a multimodal structured evidence set; Step S4: Using the fusion reasoning agent module, calculate the cross-modal evidence consistency score and evidence conflict degree based on the multimodal structured evidence set, perform cross-modal semantic fusion and multi-domain risk reasoning, and generate comprehensive behavioral risk results as well as sub-domain risk results in the emotion domain, cognitive domain, and social domain. Step S5: When the risk assessment results meet the preset trigger conditions, a human-in-the-loop review is introduced to confirm or correct the risk assessment results; Step S6: Generate a standardized evaluation report from the reasoning results of the fusion reasoning agent module and the human-in-the-loop verification results.

[0010] The core of this invention lies in encapsulating the analysis results of gait, facial, and audio modalities into structured evidence units with a unified field structure, and using these structured evidence units as cross-modal risk calculation objects. The fusion reasoning agent performs shared space mapping, consistency calculation, conflict degree calculation, subdomain risk mapping, and low-confidence gating triggering based on these structured evidence units, enabling modality missing, modality conflict, and low-quality inputs to be identified, corrected, or trigger manual review within the same risk assessment closed loop.

[0011] Specifically, the gait analysis sub-agent, facial analysis sub-agent, and audio analysis sub-agent perform feature modeling on their respective modal data, outputting structured evidence units. The structured evidence intermediate module verifies the field integrity, modal label, availability status, and time index of each structured evidence unit, and organizes and stores them according to a unified field structure to form a multimodal structured evidence set. The fusion reasoning agent performs shared space mapping, consistency calculation, conflict detection, comprehensive behavioral risk result generation, and subdomain risk reasoning based on the multimodal structured evidence set. When high-risk, high-uncertainty, low-confidence evidence clusters or system anomalies are detected, human-in-the-loop verification is triggered and the verification results are written back, thereby forming a closed-loop processing mechanism for multimodal behavioral cognition screening.

[0012] Through the above technical solution, the present invention transforms the traditional black-box unified prediction into a computable, aggregateable, and traceable structured evidence processing process, thereby improving the stability, security, and interpretability of risk assessment in scenarios with missing modalities, local anomalies, or inconsistent multimodal results.

[0013] In step S1, the preprocessing includes at least: time alignment, quality screening, data format unification, and effective segment extraction. Specifically, for video data, frame sampling, resolution unification, invalid frame removal, and time window division are performed; for audio data, noise reduction, silent segment processing, duration truncation, and sampling rate unification are performed.

[0014] After preprocessing, standardized input data suitable for processing by the corresponding modality analysis sub-agent is obtained, so as to reduce the error propagation caused by acquisition environment, timing misalignment or low quality input in subsequent analysis.

[0015] Preferably, each structured evidence unit for a modality includes at least a modality semantic feature vector, a modality risk representation value, a modality credibility parameter, a modality label, and key evidence information; wherein, the key evidence information further includes evidence location information, anomaly label set, explanatory summary, and fine-grained evidence unit set.

[0016] Preferably, in step S2, generating gait structured evidence units includes: Step S21-1: Extract frame-level gait features from the temporal image sequence of the preprocessed gait video; Step S21-2: The frame-level gait features are weighted and aggregated using a temporal attention mechanism to obtain a gait semantic feature vector; Step S21-3: Calculate the gait modality risk representation value based on the gait semantic feature vector; Step S21-4: Calculate the gait mode confidence parameters based on the gait input quality score, gait feature stability score, and model output confidence score; Step S21-5: Extract key evidence information on gait modalities related to the current gait risk outcome; Step S21-6: Encapsulate the gait semantic feature vector, gait modality risk representation value, gait modality credibility parameter, gait modality label, and gait modality key evidence information into a gait structured evidence unit.

[0017] Key gait modality evidence information includes gait evidence location information. Keyframe Index Set Abnormal gait label set Explanatory summary Gait fine-grained evidence unit set : ;

[0018] in, This indicates gait evidence location information; It is a set of keyframe indices, determined by the evidence scores corresponding to each sampled frame; This represents a set of key video clips; This represents the set of fine-grained evidence units related to gait.

[0019] For the Each gait video sample frame is weighted based on the temporal weights of the gait video. and frame-level anomaly saliency score Calculate frame-level evidence scores for gait videos. : ; in, By the Frame-level features of gait video sampling frames Obtained through the anomaly scoring function; Select the frame-level evidence with the highest score Each sampled frame and its corresponding index constitutes a keyframe index set. .

[0020] The preprocessed gait video is divided into several local time windows. For each time window, a window-level evidence score is calculated based on the temporal weight and anomaly significance score, and time windows with scores higher than a preset threshold are selected. High-scoring windows that are temporally adjacent or overlapping are merged to obtain a set of key video segments. .

[0021] abnormal gait label set It is obtained from the gait semantic feature vector through the anomaly label mapping function.

[0022] Gait mode risk characterization value Gait mode reliability parameters Keyframe Index Set Key video clips collection and abnormal gait label set Structured evidence fields are obtained through structuring. ; Structured evidence fields The input language generation module generates an explanatory summary based on a preset prompt template. .

[0023] Preferably, in step S2, generating facial structured evidence units includes: Step S22-1: Extract frame-level expression features from the facial image sequence of the preprocessed facial video; Step S22-2: The frame-level facial expression features are weighted and aggregated using a temporal attention mechanism to obtain an emotion semantic feature vector; Step S22-3: Calculate the facial modality risk representation value based on the aforementioned emotion semantic feature vector; Step S22-4: Calculate the facial modality confidence parameter based on the facial input quality score, facial feature stability score, and model output confidence score; Step S22-5: Extract key facial modal evidence information related to the current emotional risk outcome; Step S22-6: Encapsulate the emotion semantic feature vector, facial modality risk representation value, facial modality credibility parameter, facial modality label, and key facial modality evidence information into a facial structured evidence unit.

[0024] Preferably, in step S2, generating the audio structured evidence unit includes: Step S23-1: Transcribe the preprocessed speech signal to obtain a transcribed text sequence; Step S23-2: Compare the transcribed text sequence with the preset reference text to obtain a semantic consistency score, and extract language fluency feature vectors and language organization feature vectors from the transcribed text sequence and the original speech signal. Step S23-3: Merge the semantic consistency score, language fluency feature vector, and language organization feature vector into a cognitive semantic feature vector; Step S23-4: Calculate the audio modality risk representation value based on the cognitive semantic feature vector; Step S23-5: Calculate the audio modal confidence parameters based on the audio input quality score, audio feature stability score, and model output confidence score; Step S23-6: Extract key audio modal evidence information related to the current cognitive risk outcome; Step S23-7: Encapsulate the cognitive semantic feature vector, audio modality risk representation value, audio modality credibility parameter, audio modality label, and audio modality key evidence information into an audio structured evidence unit.

[0025] Audio modality key evidence information Including audio evidence location information Abnormal audio tag set Explanatory summary Audio fine-grained evidence unit set : ;

[0026] in, This indicates the location information of the audio evidence; A collection of indexes for key audio time segments; This is a set of abnormal transcribed text fragments corresponding to key audio time segments; This represents a set of labels indicating language organization anomalies. Indicates an explanatory summary; This represents a set of fine-grained audio evidence units.

[0027] Explanatory summary Audio modal risk characterization value Audio modal reliability parameters Key time segment index set abnormal semantic fragment set and a set of language organization anomaly tags Generate textual explanations of the main evidence for current perceived risk outcomes.

[0028] In step S3, the structured evidence intermediate module is used to uniformly encapsulate, organize and store the analysis results generated by each sub-agent, thereby forming a multimodal structured evidence set with a unified field structure that can be directly called by the fusion reasoning agent.

[0029] Preferably, in step S4, calculating the cross-modal evidence consistency score and the degree of evidence conflict includes: Step S4-i: Based on the set of fine-grained evidence units in each modal structured evidence unit. Constructing a fine-grained evidence risk vector And utilize the preset shared space mapping matrix Perform a shared space mapping on the fine-grained evidence risk vector to obtain a unified evidence representation. ; Step S4-ii: Calculate the cross-modal evidence consistency score between any two modalities based on the availability indicator, credibility parameter, risk characterization value, and similarity between the uniform evidence representations of each modality. Step S4-iii: Calculate the global cross-modal evidence consistency score based on the cross-modal evidence consistency scores of all valid modal pairs, and use the difference between the global cross-modal evidence consistency score and 1 as the degree of evidence conflict.

[0030] In step S4-ii, the mode Availability indicator When the mode The original data exists, and after preprocessing, it meets the minimum quality requirements, and the corresponding sub-agent successfully outputs structured evidence units. At that time, I recorded Otherwise, remember Any two modes and modality The constituent mode pairs, whose availability indicator is defined as follows: ,when and hour ,otherwise .

[0031] Preferably, in step S4-ii, the cross-modal evidence consistency score between any two modalities Calculated using the following formula: ; in, Representing modes and modality Availability indicators between; and Representing modes With mode Credibility parameters; and Representing modes With mode Risk characterization value; and It is a non-negative adjustment parameter; Representing modes With mode The similarity between unified evidence representations.

[0032] Preferably, in step S4-iii, the global cross-modal evidence consistency score The number of effective mode pairs is calculated using the following method. for: ; when At that time, the global cross-modal evidence consistency score Calculate using the following formula: ; And define the degree of conflict of evidence. ; when When this condition is met, it indicates that there are no valid modal pairs available for cross-modal comparison in the current sample. The system does not calculate the cross-modal evidence conflict degree, nor does it classify this situation as a cross-modal evidence conflict. Instead, it generates a single-modal degradation marker.

[0033] Preferably, in step S4, cross-modal semantic fusion and multi-domain risk reasoning are performed to generate a comprehensive behavioral risk result and sub-domain risk results for the emotion domain, cognitive domain, and social domain, including: Step S4-I: Calculate the initial comprehensive risk value based on the weighted average of the risk characterization values ​​and credibility parameters in each modal structured evidence unit. : ; in, It is a smoothing constant; Step S4-II: Based on the degree of conflict of evidence and initial comprehensive risk value Calculate the overall risk status : ; Step S4-III: Based on the preset contribution weights and contribution mapping functions, map the structured evidence units of each modality to the emotional, cognitive, and social domains, and calculate the risk outcome for each subdomain. The risk outcome for each subdomain is contributed by evidence from at least two modalities. ; in, For the first Subdomain risk outcomes It indicates the emotional domain, cognitive domain, or social domain; Representing modes subdomain Contribution mapping function; Representing modes subdomain The contribution weight; when only a single available mode exists, the risk result of the corresponding subdomain is estimated from that available mode, and a low confidence mark is added to the output result;

[0034] Step S4-IV: Comprehensive behavioral risk results based on overall risk status The risk results of each subdomain are generated together.

[0035] Preferably, in step S5, the preset triggering condition includes at least one of the following: The overall risk status has reached the preset high-risk threshold; At least one subdomain risk result reaches the corresponding preset high-risk threshold; The proportion of low-credibility evidence reaches a preset threshold, whereby the proportion of low-credibility evidence is defined as follows: ,in , This represents the set of fine-grained evidence units corresponding to the current sample; This represents the total number of fine-grained evidence units; Indicates the confidence level of the evidence unit; Indicates the credibility threshold; Indicates an indicator function; The degree of conflict of evidence has reached a preset threshold; The system is malfunctioning.

[0036] Preferably, in step S5, the introduction of human presence verification includes: Presents the results of automatic reasoning, key evidence information, triggering reasons, and operation logs; The reviewers confirm, correct, or mark the structured evidence units, comprehensive behavioral risk results, or subdomain risk results; Record the manual review process, output the confirmed, corrected or marked results and write them to the audit log.

[0037] Preferably, step S6 includes: generating a standardized assessment report from the comprehensive behavioral risk results, subdomain risk results, key evidence information, credibility markers, single-modal degradation markers, no valid evidence markers, triggering causes, and review status.

[0038] This invention also provides a multi-agent multimodal behavioral cognitive screening and risk assessment system, comprising: The data acquisition module collects and preprocesses the subject's gait video, facial video, and voice signals; The multimodal analysis sub-agent module includes a gait analysis sub-agent, a face analysis sub-agent, and an audio analysis sub-agent, which perform feature analysis on the preprocessed gait video, face video, and voice signal, and generate corresponding gait, face, and audio structured evidence units. The structured evidence intermediate module receives and stores each structured evidence unit to form a multimodal structured evidence set; The fusion reasoning agent module calculates cross-modal evidence consistency scores and evidence conflict degrees based on a multimodal structured evidence set, performs cross-modal semantic fusion and multi-domain risk reasoning, and generates comprehensive behavioral risk results as well as sub-domain risk results in the emotion domain, cognitive domain, and social domain. The Human-in-the-Loop Verification Module introduces human-in-the-loop verification when the risk assessment results meet the preset trigger conditions, to confirm or correct the risk assessment results; The risk report output module generates a standardized assessment report by combining the reasoning results from the fusion reasoning agent module and the human-in-the-loop verification results.

[0039] Preferably, the gait analysis sub-agent includes: Gait feature encoder extracts frame-level gait features from the temporal image sequence of the preprocessed gait video; The gait temporal aggregation unit uses a temporal attention mechanism to weighted aggregate the frame-level gait features to obtain a gait semantic feature vector; The gait risk mapping unit calculates the gait modality risk representation value based on the gait semantic feature vector. The gait confidence calculation unit calculates gait modality confidence parameters based on the gait input quality score, gait feature stability score, and model output confidence score. The gait key evidence extraction unit extracts key gait modality evidence information related to the current gait risk outcome. The gait information encapsulation unit encapsulates gait semantic feature vectors, gait modality risk representation values, gait modality credibility parameters, gait modality labels, and key gait modality evidence information into gait structured evidence units.

[0040] Preferably, the facial analysis sub-agent includes: A facial feature encoder extracts frame-level expression features from facial image sequences in a preprocessed facial video. The facial temporal aggregation unit uses a temporal attention mechanism to weight and aggregate the frame-level facial expression features to obtain an emotion semantic feature vector. The facial risk mapping unit calculates the facial modality risk representation value based on the aforementioned emotion semantic feature vector. The facial confidence calculation unit calculates facial modality confidence parameters based on the facial input quality score, facial feature stability score, and model output confidence score. The facial key evidence extraction unit extracts key facial modal evidence information related to the current emotional risk outcome. The facial information encapsulation unit encapsulates the emotion semantic feature vector, facial modality risk representation value, facial modality credibility parameter, facial modality label, and key facial modality evidence information into a structured facial evidence unit.

[0041] Preferably, the audio analysis sub-agent includes: An automatic speech recognition model transcribes preprocessed speech signals to obtain a transcribed text sequence. The cognitive semantic analysis unit compares the transcribed text sequence with a preset reference text to obtain a semantic consistency score, and extracts language fluency feature vectors and language organization feature vectors from the transcribed text sequence and the original speech signal. The cognitive feature fusion unit fuses the semantic consistency score, language fluency feature vector, and language organization feature vector into a cognitive semantic feature vector. The cognitive risk mapping unit calculates the audio modality risk representation value based on the cognitive semantic feature vector. The cognitive credibility calculation unit calculates audio modal credibility parameters based on the audio input quality score, audio feature stability score, and model output confidence score. The cognitive key evidence extraction unit extracts key audio modal evidence information related to the current cognitive risk outcome; The cognitive information encapsulation unit encapsulates the cognitive semantic feature vector, audio modality risk representation value, audio modality credibility parameter, audio modality label, and key audio modality evidence information into an audio structured evidence unit.

[0042] Preferably, the fusion reasoning agent module includes: Shared spatial mapping unit, based on the set of fine-grained evidence units in each modal structured evidence unit. Constructing a fine-grained evidence risk vector And utilize the preset shared space mapping matrix Perform a shared space mapping on the fine-grained evidence risk vector to obtain a unified evidence representation. ; The evidence consistency calculation unit calculates the evidence consistency score between any two modalities based on the availability indicator, credibility parameter, risk characterization value, and similarity between the uniform evidence representations of each modality. The global consistency calculation unit calculates the global cross-modal evidence consistency score based on the evidence consistency scores of all valid modal pairs, and uses the difference between the global cross-modal evidence consistency score and 1 as the degree of evidence conflict. The initial comprehensive risk calculation unit calculates the initial comprehensive risk value based on the weighted average of the risk characterization values ​​and credibility parameters in each modal structured evidence unit; The conflict correction unit multiplies the degree of evidence conflict by the conflict correction coefficient and adds it to the initial comprehensive risk value to obtain the overall risk status. The multi-domain risk mapping unit maps the structured evidence units of each modality to the emotion domain, cognitive domain, and social domain according to the preset contribution weights and contribution mapping functions, and calculates the risk results of each subdomain respectively. When at least two modalities are available and meet the quality requirements, the risk results of each subdomain are jointly contributed by evidence from at least two modalities. When only a single available modality exists, the risk results of the corresponding subdomain are estimated by the available modality, and a low confidence mark or a single modality degradation mark is added to the output results.

[0043] Preferably, the human-in-the-loop verification module includes: The supervisory trigger signal generation unit determines whether the risk assessment result output by the fusion reasoning agent module meets the preset trigger conditions. When the preset trigger conditions are met, a trigger signal with a value of 1 is generated. When the trigger signal is true, the review interface unit presents the automatic reasoning results, key evidence information, triggering reasons and operation logs, and receives confirmation, correction or marking of the structured evidence unit, comprehensive behavioral risk results or subdomain risk results from the reviewers. The audit log unit records the manual review process and output results.

[0044] Through the collaborative work of the aforementioned modules, this invention enables interpretable analysis and cross-domain risk assessment of multimodal behavioral signals. This invention transforms multimodal behavioral cognitive screening from a black-box unified prediction process into a computable, aggregateable, and traceable structured evidence processing procedure, thereby improving the stability, security, and interpretability of risk assessment in scenarios with missing modalities, local anomalies, or inconsistent multimodal conclusions.

[0045] Compared with the prior art, the beneficial effects of the present invention are as follows: First, by unifying the analysis results of different modalities into structured evidence units and introducing evidence consistency analysis and conflict degree calculation mechanisms, the problem of heterogeneous multimodal behavioral signals and difficulty in unified aggregation can be solved, thereby improving the stability of cross-modal risk assessment.

[0046] Second, through a credibility-based risk aggregation and conflict correction mechanism, more robust overall risk results can be obtained in the case of modality missing, local anomalies, or inconsistent multimodal conclusions, reducing the interference of single modality anomalies on the final judgment.

[0047] Third, by separating risk calculation from language interpretation generation, the system can not only output the overall risk value and modal risk characterization value, but also output the key evidence index and explanatory information corresponding to the risk results, thereby improving the interpretability and traceability of the results.

[0048] Fourth, by setting up a gating trigger mechanism based on overall risk, degree of evidence conflict, and proportion of low credibility, the system can automatically enter the manual review process in high uncertainty scenarios, thereby improving the system security and reliability in high-risk application scenarios.

[0049] Fifth, since each modal analysis unit is decoupled from the unified evidence representation mechanism, the system does not need to reconstruct the entire system when adding a new modality, adding a new subdomain task, or replacing a single analysis unit, thus having good modular expansion capabilities. Attached Figure Description

[0050] Figure 1 A structural block diagram of a multimodal behavioral cognitive screening and risk assessment system; Figure 2 A flowchart illustrating the multimodal behavioral cognition screening and risk assessment method; Figure 3 A flowchart illustrating the process of generating structured evidence units for sub-agent modules; Figure 4 This is a schematic diagram of the closed-loop process of the human-in-the-loop verification module. Detailed Implementation

[0051] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not limit it in any way.

[0052] This invention provides a multimodal behavioral cognitive screening and risk assessment system based on multi-agent collaboration, such as... Figure 1 As shown, the system includes: ① Multimodal data acquisition and preprocessing module, used to acquire and preprocess gait video, facial video and voice signals; ② The multimodal analysis sub-agent module includes a gait analysis sub-agent, a face analysis sub-agent, and an audio analysis sub-agent, which are used to perform feature analysis on the corresponding modalities and generate structured evidence units, respectively; ③ The structured evidence intermediate module is used to receive and store the structured descriptions and evaluation processes output by each sub-agent; ④ The fusion reasoning agent module is used to perform cross-modal semantic fusion and multi-domain risk reasoning based on the multimodal structured evidence set of the structured evidence intermediate module, and generate comprehensive behavioral risk results and sub-domain risk results of the emotion domain, cognitive domain, and social domain. ⑤ The human-in-the-loop verification module is used to introduce manual verification when the system's automatic assessment results meet preset trigger conditions, to confirm or correct the automatically generated risk results.

[0053] ⑥ The risk report output module is used to generate a standardized assessment report based on the risk results output by the fusion reasoning agent and the results after review, confirmation, correction or marking by the trigger person in the loop.

[0054] The methodology and process for behavioral cognitive screening and risk assessment based on a multimodal behavioral cognitive screening and risk assessment system are as follows: Figure 2 As shown, the details are as follows: Step S1: Multimodal data acquisition and preprocessing In this embodiment, gait video, facial video, and speech signals of the subject are first collected. The gait video is used to characterize the subject's posture and movement patterns during movements such as walking, turning, or standing up; the facial video is used to characterize the subject's facial dynamics, changes in facial movement units, and facial response characteristics; and the speech signals are used to characterize the subject's speech fluency, semantic organization ability, and cognitive-related speech behavior characteristics.

[0055] The aforementioned multimodal data undergoes standardized preprocessing. Preprocessing includes at least: temporal alignment, quality screening, data format standardization, and extraction of valid segments. Specifically, for video data, frame sampling, resolution standardization, invalid frame removal, and time window segmentation are performed; for audio data, denoising, silent segment processing, duration truncation, and sampling rate standardization are performed. After preprocessing, standardized input data suitable for processing by the corresponding modality analysis sub-agent is obtained, reducing error propagation caused by the acquisition environment, temporal misalignment, or low-quality input during subsequent analysis.

[0056] Let the three modalities of gait, face, and audio be denoted as follows: The structured evidence units output by each sub-agent are denoted as: ; in, Represents modal semantic feature vectors. This represents the modal risk characterization value. This represents the modal confidence parameter. Indicates modal label, This indicates key evidentiary information. Including evidence location information Exception tag set Explanatory summary and fine-grained evidence unit sets ,Right now: ; in, This represents a set of fine-grained evidence units for this modality. Each fine-grained evidence unit includes an indicator name, its subdomain, indicator value, confidence level, and risk score, used to support subsequent cross-modal shared space mapping, subdomain risk calculation, and manual review. The subdomains include the emotional domain, cognitive domain, and social domain. Emotional domain evidence units include one or more of the following: facial expression activity, facial response speed, emotional fluctuation stability, and degree of facial movement unit abnormality; cognitive domain evidence units include one or more of the following: semantic consistency, language fluency, narrative coherence, degree of semantic deviation, and motor execution stability; social domain evidence units include one or more of the following: gaze persistence, interaction response delay, facial orientation stability, verbal response coherence, and insufficient nonverbal interaction response.

[0057] Step S2: Perform feature analysis on the multimodal data and generate structured evidence units. Each sub-agent according to Figure 3 The process shown performs feature analysis on the corresponding modal data and generates structured evidence units.

[0058] ②-1 Gait Analysis Sub-Agent The gait analysis sub-agent receives the preprocessed gait video temporal image sequence. The system extracts frame-level gait features and performs temporal aggregation, calculates gait modal risk characterization values ​​and modal confidence parameters, extracts key evidence information, and outputs gait structured evidence units. The processing flow includes the following steps: Step S21-1: Gait video input and temporal sampling The preprocessed gait video is input into the gait analysis sub-agent, denoted as... ,in, Indicates the first The image obtained after decoding each sampled frame is input; Indicates the number of sampled frames. This is a standardized temporal image sequence obtained by uniformly sampling the original gait video and removing invalid frames.

[0059] In this step, the gait analysis sub-agent performs uniform sampling on the original gait video according to a preset sampling interval, and performs size unification, invalid frame removal, and temporal window division on the sampled frames to obtain a standardized gait temporal image sequence. If necessary, the sampled video frames can be subjected to size unification, invalid frame removal, and temporal window division to form a standardized gait image sequence suitable for visual encoder input.

[0060] Step S21-2: Gait feature extraction.

[0061] The time-series image sequence obtained in step S21-1 Input the visual feature encoder to extract the frame-level feature sequence corresponding to the sampled frame: , Indicates the first The visual feature vector corresponding to the frame image; Represents the dimension of visual features.

[0062] Step S21-3: Temporal aggregation and gait semantic feature generation.

[0063] Through time series aggregation functions Weighted aggregation of frame-level feature sequences yields an overall gait behavior representation. : ; in, Indicates the first Temporal weights for each sampled frame. Output from a temporal attention mechanism, used for frame-level feature sequences. Perform weighted aggregation to obtain a gait semantic feature vector that characterizes the overall gait behavior of the subject. .

[0064] Step S21-4: Gait modality risk characterization value and key evidence generation.

[0065] Based on the gait semantic feature vector obtained in steps 21-3 Calculate the risk characterization value corresponding to the gait mode: ; in, and For mapping parameters, This is the risk mapping function. Gait mode risk characterization value. This is used to characterize the risk level of subjects in terms of motor activity, gait stability, body coordination, balance control, or bradykinesia. Simultaneously, based on the gait video input quality, the proportion of effective keyframes, feature extraction stability, and model output confidence level, the confidence parameter corresponding to the gait modality is calculated. : ; in, This represents the gait input quality score, which reflects the proportion of effective keyframes, the clarity of human targets, and the visibility of motion. The gait feature stability score reflects the continuity and consistency of feature changes in adjacent time segments. This indicates the model output confidence score; , and These are weighting coefficients. Credibility parameter. Used to characterize the reliability of the current gait analysis results.

[0066] In this invention, This represents a truncation function used to truncate variables. Limited to the range within; when When 0 is taken, Time to take ,when Time to take .

[0067] Furthermore, gait input quality score It can be calculated using the following formula: ; in, Used to Limited to Inside; This represents the gait input quality score obtained during the preprocessing stage; This indicates the percentage of valid frames or the percentage of valid keyframes. The human target sharpness score is used to characterize the degree to which the subject's human figure is recognizable in an image; The motion visibility score is used to characterize whether the subject's key motion behaviors are fully presented in the field of vision, and whether there is severe occlusion, out-of-frame, or abnormal viewpoint. , , and The preset non-negative weighting coefficients are determined by offline calibration and are called at fixed intervals during system operation.

[0068] Gait feature stability score It can be calculated using the following formula: ; in, Indicates the first The aggregated gait features of a local time window can be obtained by averaging the frame-level features within that window. Indicates the number of local time windows; This represents the Euclidean distance between aggregated gait features in two adjacent local time windows, used to characterize the magnitude of feature changes; This represents a normalization constant used to make the differences between windows of different subjects and under different shooting conditions comparable.

[0069] Model output confidence score It can be calculated using the following formula: ; in, Indicates the first The confidence level of a fine-grained evidence unit of gait can be given by classification probability, similarity score or confidence score after rule mapping; This indicates the number of fine-grained evidence units related to gait.

[0070] Furthermore, based on the gait semantic feature vector Extract key evidence information relevant to the current risk outcome. Key Evidence Information Including gait evidence location information abnormal gait label set and explanatory summary Gait fine-grained evidence unit : ; in, This indicates gait evidence location information; It is a set of keyframe indices, determined by the evidence scores corresponding to each sampled frame; This represents a set of key video clips; This represents the set of fine-grained evidence units related to gait. For the ... Each sampling frame, based on temporal weights and frame-level anomaly saliency score Calculate frame-level evidence scores : ; in, Indicates the first The importance of frames in overall gait semantic modeling; Indicates the first The anomaly significance score corresponding to the frame. Frame-level features Obtained through an anomaly scoring function. The top scorers are selected. Frames, and their corresponding indices constitute a keyframe index set. .

[0071] Key video clip collection It is obtained by aggregating frame-level evidence scores within a continuous time window. The gait video is divided into several local time windows. , of which Window-level evidence scores for each time window Calculate using the following formula: ; in, The corresponding time window is designated as the high evidence window; adjacent or overlapping high evidence windows are merged to obtain the set of key video segments. ; This is the threshold for window-level evidence scores.

[0072] abnormal gait label set Gait semantic feature vector The abnormal label mapping function is used to map gait semantic features to a preset set of abnormal labels. These labels are used to characterize one or more abnormal types, including shortened stride, abnormal trunk swing, unstable motor rhythm, slowed gait, abnormal balance control, or decreased body coordination.

[0073] Explanatory summary Gait mode risk characterization value Credibility parameters Keyframe Index Set Key video clips collection and abnormal gait label set This is used to generate a textual description of the main evidence for the current risk outcome. In a preferred embodiment, structured evidence fields are used. The input language generation module generates an explanatory summary based on a preset prompt template. .

[0074] Step S21-5: Generation of gait structured evidence units.

[0075] Based on the outputs of steps S21-3 and S21-4, construct the gait structured evidence unit: ; in, Represents gait semantic features; This represents the gait mode risk characterization value; This represents the gait mode confidence parameter; Indicates gait modality label; This indicates key evidence information about gait.

[0076] ②-2 Facial Analysis Sub-Agent The facial analysis sub-agent receives the preprocessed facial image sequence. The system extracts frame-level facial expression features and performs temporal aggregation, calculates facial modality risk representation values ​​and modality credibility parameters, extracts key facial modality evidence information, and outputs structured facial evidence units. .

[0077] Step S22-1: Facial video input and keyframe extraction.

[0078] The preprocessed facial video is input into the facial analysis sub-agent, denoted as: ; in, Indicates the first One time frame image; This indicates the number of frames in the facial video. This is a standardized facial image sequence obtained by performing temporal sampling, face detection, facial region cropping, and invalid frame removal on the original facial video.

[0079] In this step, the original facial video can be temporally sampled uniformly, and face detection, facial region cropping, and size standardization can be performed on the video frames to obtain a sequence of facial images that can characterize the subject's expression changes, facial movements, and emotional responses. If necessary, keypoint stability detection and invalid frame removal can also be performed on the video frames to improve the stability of subsequent expression feature extraction.

[0080] Step S22-2: Facial visual feature extraction.

[0081] The facial image sequence obtained in step S22-1 is input into the facial visual encoder to extract frame-by-frame facial expression features. ,get: ; in, Indicates the first Visual feature representation of a frame of facial images.

[0082] Step S22-3: Emotion semantic analysis and description generation.

[0083] Temporal modeling and feature aggregation are performed on the frame-by-frame facial features obtained in step S22-2 to obtain facial emotion semantic feature vectors: ; in, Indicates the first Temporal weights corresponding to facial features in a frame. Temporal weights Outputted by a temporal attention mechanism and used for processing frame-by-frame facial feature sequences. Perform weighted aggregation. Through the above temporal aggregation, an emotional semantic feature vector representing the overall emotional state and facial behavior patterns of the subject can be obtained. .

[0084] Step S22-4: Generate structured evidence units.

[0085] Based on the emotion semantic feature vector obtained in step S22-3 Calculate the facial modality risk characterization value: ; in, and For mapping parameters; This is the risk mapping function. Modal risk representation value. This is used to characterize the risk level of subjects in terms of emotional expression activity, facial reaction speed, emotional fluctuation patterns, or tendency towards depression. Simultaneously, based on factors such as facial detection stability, keypoint tracking success rate, effective video frame ratio, and model output confidence, the reliability parameter corresponding to the facial modality is calculated. : ; in, This represents the facial input quality score, which reflects the stability, occlusion, and clarity of face detection. The facial feature stability score reflects the continuity of facial action units, expression features, and gaze features in adjacent temporal segments. This indicates the model output confidence score; , and These are the weighting coefficients.

[0086] Furthermore, facial input quality score It can be calculated using the following formula: ; in, This represents the facial input quality score obtained during the preprocessing stage; The face detection stability score reflects whether faces can be consistently and accurately detected in adjacent video frames. The score represents the degree of occlusion, reflecting the impact of factors such as masks, hand coverings, hair coverings, and side face coverings on the visibility of the facial area. The facial image sharpness score represents the degree to which facial texture, expression details, and key areas are recognizable. This indicates the percentage of valid frames or valid keyframes. , , , and These are non-negative weighting coefficients used to control the impact of each component on... The extent of their contribution.

[0087] Facial feature stability score It can be calculated using the following formula: ; in, The continuity score of the action unit is used to reflect whether the intensity of the action unit changes smoothly over time and whether there are any abnormal jumps. The fixation continuity score reflects the stability of gaze direction, fixation pattern, and head orientation in adjacent time segments. The expression response continuity score reflects the naturalness of the temporal sequence of expression intensity, expression transitions, and emotional responses. , and These are non-negative weighting coefficients used to characterize the importance of three types of stability indicators: action unit, gaze pattern, and facial expression response.

[0088] Model output confidence score It can be calculated using the following formula: ; in, Indicates the first The confidence level of a fine-grained facial evidence unit can be given by the probability of abnormal recognition of action unit, the confidence level of judging abnormal gaze, and the confidence level of judging abnormal facial expression response. This indicates the number of facial fine-grained evidence units.

[0089] Furthermore, from the emotion semantic feature vector Extract key evidence information related to current emotional risk outcomes Key Evidence Information Including facial evidence location information Abnormal facial tag set Explanatory summary Facial fine-grained evidence unit : ; in, This indicates the location information of facial evidence; It is a set of keyframe indices, determined by the evidence scores corresponding to each sampled frame; This represents a set of key video clips; This represents a set of fine-grained evidence units related to facial features. Specifically, it is first based on temporal weights. The degree of abnormality in the action unit, the degree of abnormality in gaze, and the degree of abnormality in facial expression response are used to calculate the degree of abnormality in the action unit. Frame Evidence Importance Score : ; in, Indicates the degree of abnormality of the action unit; Indicates the degree of gaze abnormality; Indicates the degree of abnormality in facial expression response; , , , Weights were calculated for the importance of facial evidence, corresponding to temporal weights, the degree of abnormality in motor units, the degree of abnormality in gaze, and the degree of abnormality in facial expression response. These weights were then used to determine the importance of the evidence based on its score. Select high-contribution frame indices to form the keyframe index set. High-contribution frames that are temporally consecutive are merged to form a set of key time segments. Interpretive summary according to , , , and abnormal emoji tag set Generates using a templated language generation module.

[0090] Step S22-5: Generation of facial structured evidence units.

[0091] Based on the outputs of steps S22-3 and S22-4, construct facial structured evidence units: ; in, Represents semantic features of facial emotions; This represents the facial modality risk characterization value; This represents the facial modality confidence parameter; Represents facial modality tags; This indicates key evidentiary information. In this step, the facial structured evidence unit is... The data is written into the structured evidence intermediate module as the basic input for subsequent cross-modal consistency analysis, evidence conflict calculation, overall risk aggregation, and interpretation generation.

[0092] ②-3 Audio Analysis Sub-Agent The audio analysis sub-agent is used to perform content understanding and cognitive representation analysis on the subject's speech data. It extracts structured indicators related to cognitive state from three levels: speech expression, language organization, and semantic consistency, and generates structured evidence units for cognitive domain risk assessment.

[0093] Step S23-1: Audio preprocessing.

[0094] The audio analysis sub-agent first receives the subject's raw speech data. The raw speech data is then standardized to form a unified audio input representation. Standardization includes audio segmentation, duration unification, noise suppression, and acoustic feature representation construction, ensuring that speech data acquired under different acquisition conditions can be input into the audio analysis sub-agent in a consistent format. After processing, the standardized audio representation is obtained: ; in, Represents the raw speech data. This represents the audio normalization mapping function. This represents the standardized audio input representation.

[0095] Step S23-2: Speech transcription and speech content acquisition.

[0096] The audio analysis sub-agent uses an automatic speech recognition model to represent standardized audio. Speech-to-text transcription yields a text sequence: ; in, This represents a speech coding function used to extract temporal semantic features from audio. This represents a speech decoding function, used to output the corresponding text based on the encoded features; This represents a sequence of speech-to-text transcripts generated by the audio analysis sub-agent. Through this process, the audio analysis sub-agent converts continuous speech input into a linguistic content representation that can be subjected to cognitive semantic analysis.

[0097] Step S23-3: Cognitive semantic analysis and feature extraction In obtaining the transcribed text Subsequently, the audio analysis sub-agent further extracts cognitive-related features from three aspects: semantic consistency, language fluency, and organizational coherence.

[0098] First, transcribe the text. With preset reference text The semantic consistency score is obtained by comparison: ; in, This represents a text semantic encoding function. Represents the semantic similarity function. The semantic consistency score represents the degree of consistency between the subject's language expression and the semantics of the target task. The lower the consistency, the higher the likelihood that the language content deviates from the target task or that there is an abnormality in cognitive processing.

[0099] Secondly, the audio analysis sub-agent extracts language fluency features and language organization features from the transcribed text and speech rhythm. Let the language fluency feature vector be... The language organization feature vector is Then it can be written as ; in, The fluency feature extraction module is used to extract indicators such as pause frequency, speech rate fluctuation, and sentence interruption. The language organization feature extraction module is used to extract indicators such as syntactic integrity, contextual coherence, semantic jump degree, and narrative coherence.

[0100] Furthermore, semantic consistency, linguistic fluency, and language organization ability are jointly mapped into a cognitive semantic representation: ; in, This represents the cognitive semantic vector output by the audio analysis sub-agent; This represents the cognitive feature fusion function. Cognitive semantic vectors are used to characterize a subject's semantic retention ability, language organization ability, and expression stability in the current task, and serve as the main input for cognitive domain risk assessment.

[0101] This invention further defines the modal risk representation value of the audio analysis sub-agent for the current sample as: ; in, and These represent the audio cognitive risk mapping parameters, Represents the normalized mapping function. This represents the modal risk representation value output by the audio analysis sub-agent. This score reflects the degree to which the subject exhibits signs of cognitive abnormalities at the language expression level.

[0102] Step S23-4: Generate a structured semantic description of the audio.

[0103] The audio analysis sub-agent uses cognitive semantic vectors and modal risk characterization value Further calculate the credibility parameters corresponding to the audio modalities.

[0104] ; in, This represents the audio input quality score, which reflects the signal-to-noise ratio, effective speech percentage, sampling integrity, and speech intelligibility. The audio feature stability score reflects the continuity and consistency of speech rate, pause patterns, and language organization features in adjacent time segments. This indicates the model output confidence score; , and These are the weighting coefficients.

[0105] Furthermore, audio input quality score It can be determined as follows: ; in, This indicates the overall quality assessment results of the audio preprocessing stage; The signal-to-noise ratio score reflects the relative strength between effective speech and background noise. This indicates the percentage of effective speech, reflecting whether the remaining speech information is sufficient to support subsequent analysis after removing silent and invalid segments. The speech intelligibility score represents the clarity of speech, the intelligibility of pronunciation, and the parsability of automatic speech recognition. , , and These are non-negative weighting coefficients used to control the impact of each component on... The extent of their contribution.

[0106] Audio feature stability score It can be calculated using the following formula: ; in, The continuity score represents the transcription quality, semantic continuity, and the naturalness of the transitions between audio segments within adjacent time windows. It indicates the degree of fluctuation in speech rate and is used to measure the dispersion of speech rate changes between different time windows; The normalized maximum reference value for the degree of speech rate fluctuation; It represents the proportion of sentence interruptions, used to measure the degree of occurrence of abnormal expressions such as incomplete short sentences, sudden interruptions, and repeated restarts in the speech stream; , , These are non-negative weighting coefficients used to adjust the effects of segment continuity, speech rate fluctuations, and sentence breaks on speech flow. The extent of their contribution.

[0107] Model output confidence score It can be calculated using the following formula: ; in, Indicates the first Confidence level of each speech recognition segment; This indicates the number of fine-grained audio evidence units.

[0108] Extract key evidence information related to current cognitive risk outcomes. Key Evidence Information This may include audio evidence location information. Abnormal audio tag set Explanatory summary Audio fine-grained evidence unit set: ; in, This indicates the location information of the audio evidence; A collection of indexes for key audio time segments; This is a set of abnormal transcribed text fragments corresponding to key audio time segments; This represents a set of labels indicating language organization anomalies. Indicates an explanatory summary; This represents a set of fine-grained audio evidence units. Each fine-grained audio evidence unit... It should include at least the name of the audio or language behavior indicator, its subdomain, indicator value, risk score, confidence level, and the location of the corresponding audio time segment or transcribed text segment. The name of the audio or language behavior indicator may include one or more of the following: pause frequency, speech rate fluctuation, sentence interruption ratio, semantic consistency score, semantic deviation degree, narrative coherence, and contextual cohesion.

[0109] For the first A local time window is defined to determine window-level anomaly scores. : ; in, This indicates the abnormality of the pause ratio, used to characterize the frequency of abnormal pauses within this window; Normalized speech rate fluctuation anomaly is used to characterize the degree of speech rate instability within this window; Statement interruption anomaly degree is used to characterize the degree to which statements within this window are incomplete, repeatedly start, or interrupted. Local semantic consistency score between the transcribed text and the preset reference text; , , , is a weighting coefficient used to adjust the contribution of different anomaly sources to the window-level anomaly score.

[0110] when At that time, the corresponding time window will be included in the key time segment index set. A collection of abnormal semantic fragments It consists of transcribed fragments with local semantic similarity below a preset threshold. (A set of language organization anomaly tags) Generated through a rule-based mapping function, it includes at least one or more of the following: syntactic incompleteness, contextual coherence anomalies, semantic jumps, and decreased narrative coherence. Interpretive summary. according to , , , and Generates using a templated language generation module.

[0111] Based on the above results, an audio structured evidence unit is constructed: ; in, This represents the credibility score of the sub-agent's output. This represents the audio modal risk characterization value; Indicates modal label; This indicates key audio evidence information.

[0112] Step S23-5: Write the structured evidence intermediate module The structured evidence units generated by the audio analysis sub-agent, along with subject identifiers, time information, and modality types, are written into the structured evidence intermediate module to achieve standardized storage of speech-cognition related features, which can then be uniformly retrieved by the fusion reasoning agent in cross-modal semantic fusion and multi-domain risk reasoning.

[0113] Step S3: Unify the encapsulation, organization, and storage of the structured evidence units generated by each sub-agent. The structured evidence intermediate module is used to uniformly encapsulate, organize, and store the analysis results generated by each sub-agent, thereby forming a multimodal structured evidence set with a unified field structure that can be directly invoked by the fusion reasoning agent. In this invention, the structured evidence intermediate module uniformly represents the outputs of different modalities as a multimodal structured evidence set.

[0114] The gait analysis sub-agent, facial analysis sub-agent, and audio analysis sub-agent each output structured evidence units. , and : ; ; ; in, Represent modal semantic features; This represents the modal risk characterization value; Indicates the modal confidence parameter; Indicates modal label; This indicates key evidentiary information.

[0115] The multimodal structured evidence set is composed of structured evidence units from various modalities. ; When the raw data for a certain modality is missing, the preprocessed data does not meet the minimum quality requirements, key parameters are missing, or the corresponding sub-agent fails to output structured evidence units, the corresponding modality is marked as unavailable, and an availability indicator for that modality is set. The data is written into the structured evidence intermediate module in the form of empty placeholder evidence entries, unavailable markers, or abnormal state markers; this modality does not participate in subsequent cross-modal consistency calculations, evidence conflict degree calculations, and risk-weighted aggregation. When the original data of a certain modality exists, and the corresponding sub-agent can output structured evidence units, but the modality has low input quality, insufficient feature stability, or low model output confidence scores, the corresponding modality is marked as a low-confidence state, and an availability indicator for that modality is set. And through a lower modal confidence parameter It participates in subsequent risk-weighted aggregation, and a low-confidence marker is appended to the output. The standardized representation refers to the encapsulation, organization, and storage of a unified field structure, and does not involve cross-modal shared space mapping.

[0116] Step S4: The fusion reasoning agent completes cross-modal consistency analysis, evidence conflict calculation, overall risk aggregation, and subdomain risk reasoning based on the multimodal structured evidence set. To achieve comparability of semantic features across different modalities, the fusion reasoning agent module uses fine-grained evidence unit sets from the structured evidence units of each modality. Constructing a fine-grained evidence risk vector And utilize the preset shared space mapping matrix Perform a shared space mapping on the fine-grained evidence risk vector to obtain a unified evidence representation. The shared space mapping function is a pre-constructed cross-modal shared semantic mapping function in this invention, used to uniformly project heterogeneous evidence from gait, facial, and audio modalities into the same shared semantic space.

[0117] In the implementation, for modes Structured evidence unit Let its fine-grained evidence unit risk vector be: ; in, Representing modes The Middle Normalized risk scores for each fine-grained unit of evidence; Representing modes The number of fine-grained evidence units. The shared space mapping matrix is... Then, a unified evidence representation The Vickers can be represented as: ; in, Representing modes The number of fine-grained evidence units, Indicates the first Each fine-grained evidence unit in the shared semantic space Dimensional contribution weights This is a smoothing constant. Therefore, heterogeneous evidence from different modalities that were originally not directly comparable can be uniformly transformed into a unified evidence representation in a shared semantic space. This is used for subsequent evidence consistency analysis and cross-modal risk aggregation.

[0118] make Representing modes Availability indicator. When modal The original data exists, and after preprocessing, it meets the minimum quality requirements, and the corresponding sub-agent successfully outputs structured evidence units. At that time, I recorded Otherwise, remember Any two modes and The modality pair availability indicator is defined as follows: When both modes are available ,otherwise .

[0119] Cross-modal evidence consistency scores can be calculated based on the similarity relationships between semantic features of different modalities: ; in, Representing modes With mode The consistency score of evidence between them; This indicates modal availability. and Representing modes With mode Credibility parameters; and Representing modes With mode Modal risk characterization value; | | Represents a mode With mode The degree of difference in risk assessment; and Here, is a non-negative adjustment parameter, where Used to control the inhibitory strength of risk difference items on consistent scores. To smooth out the adjustment parameters and avoid the denominator being 0; Representing modes With mode The similarity measurement function in the unified evidence space takes values ​​ranging from... .

[0120] Based on the pairwise evidence consistency score between each modality pair Let the number of effective mode pairs be: ; in, Representing modes With mode Whether the composed mode pairs can be used for cross-modal comparison. When When this occurs, it indicates that there are no valid modal pairs available for cross-modal comparison in the current sample. The system does not calculate the cross-modal evidence conflict degree, nor does it classify this situation as a cross-modal evidence conflict; instead, it generates a single-modal degradation marker. At that time, the global cross-modal evidence consistency score Calculate using the following formula: ; in, This indicates the global consistency level of evidence for the current sample at the multimodal level; the more effective modal pairs involved in the calculation, and the more semantically similar, risk-judgment consistent, and credible the pairs, the higher the level of consistency. The larger the value, the greater the degree of evidence conflict. The difference between the global cross-modal evidence consistency score and 1 is used as the degree of evidence conflict. ; in, This is used to characterize the degree of conflict between different modal outputs; the degree of evidence conflict increases when there are significant inconsistencies between different modalities in risk characterization values, uniform evidence representation, or credibility levels. The global cross-modal evidence consistency score is also used. and degree of conflict of evidence All are calculated by the fusion reasoning agent based on the multimodal structured evidence set, and serve as input for subsequent overall risk aggregation, anomaly detection, and triggered review; the structured evidence intermediate module is only responsible for the unified encapsulation, organization, and storage of structured evidence units, and does not perform the consistency calculation and conflict degree calculation.

[0121] The fusion reasoning agent module is used to complete overall risk aggregation and interpretation generation based on a multimodal structured evidence set. In this embodiment, the initial comprehensive risk value is first calculated based on the modal risk representation value and credibility parameter in each modal structured evidence unit: ; in, Representing modes Modal risk characterization value; Representing modes Credibility parameters; To prevent the smoothing constant from having a denominator of 0, its value range is preferably [missing information]. to ;when or all modes When this occurs, no numerical low-risk conclusion is generated. Instead, the overall risk status is recorded as unassessable, and a marker indicating no valid evidence is generated, triggering a system anomaly review.

[0122] Furthermore, the results calculated by the fusion reasoning agent module... After conflict correction of the basic risk value, the overall risk status is obtained: ; in, This is a non-negative conflict correction coefficient, used to control the strength of the correction for the degree of evidence conflict on the overall risk status; This represents the truncation function, used to define the overall risk status. Limited to Within the range; This is used to characterize the subject's comprehensive behavioral and cognitive risk level. After obtaining the overall risk status, the fusion reasoning agent further analyzes the multimodal structured evidence set. Consistency score Conflict level and overall risk status The system generates subdomain risk results, calculating them separately for the emotion, cognition, and social domains. Under normal multimodal availability, each subdomain risk result is preferentially contributed by evidence from at least two relevant modalities. When the number of relevant available modalities is 1, the system enters a single-modal degradation mode, still calculating according to the subdomain risk calculation formula, but only the structured evidence unit, contribution mapping function, contribution weight, and confidence parameter of the single available modality are used in the calculation, and a single-modal degradation label and a low confidence label are added to the output result. When the number of relevant available modalities is 0, the system does not generate a numerical risk result for that subdomain, but generates a label indicating no valid evidence or low coverage verification.

[0123] The contribution of different modalities to the subdomain can be determined by weighting based on a preset mapping function, evidence relevance, and credibility parameters.

[0124] Let the first The risk result of each subdomain is denoted as Modality Key Evidence Information Includes a set of fine-grained evidence units .in, Representing modes The Each fine-grained unit of evidence includes at least the indicator name, its subdomain, indicator value, confidence level, risk score, and the location of the evidence source. For any given fine-grained unit of evidence... , This indicates the risk subdomain to which the fine-grained evidence unit belongs, and its values ​​include the emotional domain, cognitive domain, or social domain. This represents the normalized risk score corresponding to the fine-grained evidence unit, used to characterize the degree to which the evidence unit contributes to the risk of its subdomain.

[0125] make Representing modes Belongs to a subfield The set of fine-grained evidence units, then modality subdomain The contribution mapping function can be defined as: ; in, Representing modes Belongs to a subfield The number of fine-grained evidence units. Contribution mapping function. By mode Belongs to a subfield Fine-grained evidence unit set The risk scores are obtained by average aggregation or weighted aggregation.

[0126] The risk results for each subdomain are calculated using the following formula: ; in, Representing modes subdomain Contribution weights are used to characterize modes. subdomain The relative contribution intensity; Representing modes subdomain Contribution mapping function; Representing modes Structured evidence units.

[0127] Contribution mapping function Used to convert modes The middle belongs to the target subdomain The fine-grained evidence units are aggregated into corresponding subdomain contribution values. In one embodiment, the contribution mapping function can be obtained by performing average aggregation and weighted average aggregation on the risk scores of fine-grained evidence units belonging to the target subdomain. The contribution weights... These are preset configurable parameters used to characterize the relative contribution of different modalities to the risk results of subdomains; the contribution weights can be preset based on prior knowledge or determined by calibration using an offline calibration dataset, and are called as fixed configuration parameters during system operation.

[0128] Under normal multimodal availability, the risk outcome of each subdomain is contributed by evidence from at least two modalities. The degree of contribution of different modalities to the subdomain can be determined by weighting according to a preset mapping function, evidence relevance, and confidence parameters. When only a single available modality exists, the risk outcome of the corresponding subdomain is estimated from that available modality, and a low confidence marker or a single-modal degradation marker is appended to the output to indicate that the current subdomain risk outcome is generated in a degradation mode.

[0129] In one implementation, the risk outcome in the emotional domain is primarily contributed by evidence of abnormal facial emotional expression and evidence of decreased gait vitality; the risk outcome in the cognitive domain is primarily contributed by evidence of abnormal language organization, semantic deviation, slow facial response, and slow gait movement; and the risk outcome in the social domain is primarily contributed by evidence of abnormal facial gaze, insufficient emotional interaction, abnormal language participation, fluency of speech communication, and decreased gait vitality.

[0130] The overall behavioral risk outcome is based on the overall risk status. Subdomain Risk Result Set The system generates a comprehensive behavioral risk result based on the corresponding risk level and explanatory text summary, and outputs the comprehensive behavioral risk result to the human-in-the-loop verification module and the risk report output module.

[0131] Step S5: Triggered Human-in-the-Loop Review The human-in-the-loop review module receives monitoring trigger signals, automatic reasoning results, key evidence information, trigger reasons, and operation logs. When the monitoring trigger signal is set to 1, the human-in-the-loop review module presents the above information and receives input from reviewers, performing confirmation, correction, or marking on structured evidence units, comprehensive behavioral risk results, or subdomain risk results. When the overall risk status... When a preset high-risk threshold is reached, or when the risk result of at least one subdomain reaches the corresponding preset high-risk threshold, the system automatically triggers the human-in-the-loop monitoring process. The trigger-based review and optimization includes two levels: sub-level review judgment and comprehensive review judgment.

[0132] Determine whether a single mode needs to undergo a controlled reanalysis. For any mode... Secondary verification of this modality is triggered when at least one of the following conditions is met: Alternatively, the mode is associated with at least one other available mode. The resulting mode pairs were classified as low-consistency mode pairs. For modal confidence parameters, This is a low confidence threshold. Low consistency mode pairs are those that satisfy... and modal pairs, where, Representing modes With mode The modality of availability indicator Representing modes With mode The consistency score of evidence between them This represents the pairwise consistency threshold. If the modality... If a mode participates in at least one low-consistency mode pair, then the mode will be... Mark it as a conflict-related modality and trigger a secondary verification of that modality.

[0133] Based on the overall risk score The system anomaly is marked as Anomalies are marked as 1. System anomaly status markers are automatically generated by the operation monitoring module based on the system's operating status. Preset anomaly events include at least one or more of the following: missing or corrupted input data, preprocessing failure, feature extraction failure, model inference timeout, failure to successfully generate structured evidence units, and missing or out-of-bounds output results. The supervision trigger signal can be represented as: ; in, For indicator functions; , and These represent risk outcomes in the emotional domain, cognitive domain, and social domain, respectively. Indicates the overall risk status threshold; , and These represent the high-risk trigger thresholds for the emotional, cognitive, and social domains, respectively. This indicates the threshold for the proportion of low-credibility evidence. Indicates the degree of conflict of evidence; Indicates the threshold for the degree of conflict of evidence; Indicates an abnormal system operating state; when the number of valid mode pairs... When determining the degree of conflict of evidence Has the preset threshold for the degree of conflict of evidence been reached? ;when At that time, the evidence conflict trigger condition is not enabled, and the system determines whether to enter the human-in-the-loop review process based on the single-modal degradation marker, low coverage marker, no valid evidence marker, or abnormal system operation status. Indicates logical OR; This indicates a logical AND operation.

[0134] The proportion of low-credibility evidence is defined as follows: : ; ; in, This represents the set of fine-grained evidence units corresponding to the current sample; Indicates the total number of evidence units, when When this happens, the system will mark the current sample as having no valid evidence and trigger system anomaly or low coverage review conditions; Indicates the confidence level of the evidence unit; Indicates the credibility threshold; Indicates an indicator function.

[0135] The supervisor's responsibility is to monitor, inspect, and verify the system results, without interfering with the normal operation of the multi-agent system. When the system directly outputs the comprehensive behavioral risk results and subdomain risk results; when When the process begins, the system enters the human-in-the-loop review process, presenting the automatic reasoning results, key evidence information, triggering reasons, and operation logs. Reviewers then confirm, correct, or mark the relevant structured evidence units, comprehensive behavioral risk results, or subdomain risk results, and finally output the confirmed, corrected, or marked risk results.

[0136] Figure 4This demonstrates the closed-loop process of human-centered review. After receiving the automatic assessment results, the regulatory interface first determines whether the preset trigger conditions are met. If not, it directly outputs the comprehensive behavioral risk result. If the conditions are met, it presents the automatic reasoning result, key evidence information, trigger reasons, and log information, which are then confirmed, corrected, or marked by human reviewers. Finally, the comprehensive behavioral risk result is output and written to the audit log.

[0137] The risk report output module generates a standardized behavioral and cognitive risk assessment report based on the output results of the fusion reasoning agent module and the human-in-the-loop review module. The risk assessment report includes at least the emotional domain risk results, cognitive domain risk results, social domain risk results, comprehensive behavioral risk results, key evidence information, credibility markers, single-modal degradation markers, no valid evidence markers, triggering reasons, review status information, and audit log index.

[0138] To ensure the auditability of the system, this invention logs the operation process of the multi-agent system, saving the output of each sub-agent, the fusion reasoning results, the triggering reasons, and the supervision conclusions, thereby achieving interpretability, traceability, and verifiability of the system behavior.

[0139] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for screening and assessing the cognitive behavior of multiple agents using multiple modalities, characterized in that, Includes the following steps: Step S1: Acquire and preprocess the subject's gait video, facial video, and voice signals; Step S2: Use the gait analysis sub-agent, the face analysis sub-agent, and the audio analysis sub-agent to perform feature analysis on the preprocessed gait video, face video, and voice signal, and generate corresponding gait, face, and audio structured evidence units. Step S3: Store each structured evidence unit in the structured evidence intermediate module to form a multimodal structured evidence set; Step S4: Using the fusion reasoning agent module, calculate the cross-modal evidence consistency score and evidence conflict degree based on the multimodal structured evidence set, perform cross-modal semantic fusion and multi-domain risk reasoning, and generate comprehensive behavioral risk results as well as sub-domain risk results in the emotion domain, cognitive domain, and social domain. Step S5: When the overall behavioral risk results meet the preset trigger conditions, a human-in-the-loop review is introduced to confirm or correct the risk assessment results; Step S6: Generate a standardized evaluation report from the reasoning results of the fusion reasoning agent module and the human-in-the-loop verification results.

2. The multi-agent multimodal behavioral cognition screening and risk assessment method according to claim 1, characterized in that, In step S2, generating gait structured evidence units includes: Step S21-1: Extract frame-level gait features from the temporal image sequence of the preprocessed gait video; Step S21-2: The frame-level gait features are weighted and aggregated using a temporal attention mechanism to obtain a gait semantic feature vector; Step S21-3: Calculate the gait modality risk representation value based on the gait semantic feature vector; Step S21-4: Calculate the gait mode confidence parameters based on the gait input quality score, gait feature stability score, and model output confidence score; Step S21-5: Extract key evidence information on gait modalities related to the current gait risk outcome; Step S21-6: Encapsulate the gait semantic feature vector, gait modality risk representation value, gait modality credibility parameter, gait modality label, and gait modality key evidence information into a gait structured evidence unit.

3. The multi-agent multimodal behavioral cognition screening and risk assessment method according to claim 1, characterized in that, In step S2, generating facial structured evidence units includes: Step S22-1: Extract frame-level expression features from the facial image sequence of the preprocessed facial video; Step S22-2: The frame-level facial expression features are weighted and aggregated using a temporal attention mechanism to obtain an emotion semantic feature vector; Step S22-3: Calculate the facial modality risk representation value based on the aforementioned emotion semantic feature vector; Step S22-4: Calculate the facial modality confidence parameter based on the facial input quality score, facial feature stability score, and model output confidence score; Step S22-5: Extract key facial modal evidence information related to the current emotional risk outcome; Step S22-6: Encapsulate the emotion semantic feature vector, facial modality risk representation value, facial modality credibility parameter, facial modality label, and key facial modality evidence information into a facial structured evidence unit.

4. The multi-agent multimodal behavioral cognition screening and risk assessment method according to claim 1, characterized in that, In step S2, generating audio structured evidence units includes: Step S23-1: Transcribe the preprocessed speech signal to obtain a transcribed text sequence; Step S23-2: Compare the transcribed text sequence with the preset reference text to obtain a semantic consistency score, and extract language fluency feature vectors and language organization feature vectors from the transcribed text sequence and the original speech signal. Step S23-3: Merge the semantic consistency score, language fluency feature vector, and language organization feature vector into a cognitive semantic feature vector; Step S23-4: Calculate the audio modality risk representation value based on the cognitive semantic feature vector; Step S23-5: Calculate the audio modal confidence parameters based on the audio input quality score, audio feature stability score, and model output confidence score; Step S23-6: Extract key audio modal evidence information related to the current cognitive risk outcome; Step S23-7: Encapsulate the cognitive semantic feature vector, audio modality risk representation value, audio modality credibility parameter, audio modality label, and audio modality key evidence information into an audio structured evidence unit.

5. The multi-agent multimodal behavioral cognition screening and risk assessment method according to claim 1, characterized in that, In step S4, calculating the cross-modal evidence consistency score and the degree of evidence conflict includes: Step S4-i: Construct a fine-grained evidence risk vector based on the set of fine-grained evidence units in each modal structured evidence unit, and perform shared space mapping on the fine-grained evidence risk vector using a preset shared space mapping matrix to obtain a unified evidence representation; Step S4-ii: Calculate the cross-modal evidence consistency score between any two modalities based on the availability indicator, credibility parameter, risk characterization value, and similarity between the uniform evidence representations of each modality. Step S4-iii: Calculate the global cross-modal evidence consistency score based on the cross-modal evidence consistency scores of all valid modal pairs.

6. The multi-agent multimodal behavioral cognition screening and risk assessment method according to claim 5, characterized in that, In step S4-ii, the mode Availability indicator When the mode The original data exists, and after preprocessing, it meets the minimum quality requirements, and the corresponding sub-agent successfully outputs structured evidence units. At that time, I recorded Otherwise, remember Any two modes and modality The constituent mode pairs, whose availability indicator is defined as follows: ,when and hour ,otherwise .

7. The multi-agent multimodal behavioral cognition screening and risk assessment method according to claim 6, characterized in that, In step S4-ii, the cross-modal evidence consistency score between any two modalities. Calculated using the following formula: ； in, Representing modes and modality Availability indicators between; and Representing modes With mode Credibility parameters; and Representing modes With mode Risk characterization value; and To adjust the parameters; Representing modes With mode The similarity between the unified evidence representations, with a value range of .

8. The multi-agent multimodal behavioral cognition screening and risk assessment method according to claim 7, characterized in that, In step S4, cross-modal semantic fusion and multi-domain risk reasoning are performed to generate comprehensive behavioral risk results and sub-domain risk results for the emotion domain, cognitive domain, and social domain, including: Step S4-I: Calculate the initial comprehensive risk value by performing a confidence-weighted average of the risk characterization values ​​for each modality. : ； in, It is a smoothing constant; Step S4-II: Based on the degree of conflict of evidence and initial comprehensive risk value Calculate the overall risk status : ； in, This represents the truncation function, used to define the overall risk status. Limited to Within the range; These are non-negative conflict correction coefficients; Step S4-III: Based on the preset contribution weights and contribution mapping functions, map the structured evidence units of each modality to the emotional, cognitive, and social domains, and calculate the risk outcome for each subdomain. The risk outcome for each subdomain is contributed by evidence from at least two modalities. ； in, For the first Subdomain risk outcomes It indicates the emotional domain, cognitive domain, or social domain; Representing modes subdomain Contribution mapping function; Representing modes subdomain The contribution weight; when only a single available mode exists, the risk result of the corresponding subdomain is estimated from that available mode, and a low confidence mark is added to the output result; Step S4-IV: Comprehensive behavioral risk results based on overall risk status The risk results of each subdomain are generated together.

9. The multi-agent multimodal behavioral cognition screening and risk assessment method according to claim 8, characterized in that, In step S5, the preset triggering condition includes at least one of the following: The overall risk status has reached the preset high-risk threshold; At least one subdomain risk result reaches the corresponding preset high-risk threshold; The proportion of low-credibility evidence reaches a preset threshold, whereby the proportion of low-credibility evidence is defined as follows: ,in This represents the set of fine-grained evidence units corresponding to the current sample; This represents the total number of fine-grained evidence units; Indicates the confidence level of the evidence unit; Indicates the credibility threshold; Indicates an indicator function; The degree of conflict of evidence has reached a preset threshold; The system is malfunctioning.

10. A multi-agent, multimodal behavioral cognitive screening and risk assessment system, characterized in that, For performing the method according to any one of claims 1-9, comprising: The data acquisition module collects and preprocesses the subject's gait video, facial video, and voice signals; The multimodal analysis sub-agent module includes a gait analysis sub-agent, a face analysis sub-agent, and an audio analysis sub-agent, which perform feature analysis on the preprocessed gait video, face video, and voice signal, and generate corresponding gait, face, and audio structured evidence units. The structured evidence intermediate module receives and stores each structured evidence unit to form a multimodal structured evidence set; The fusion reasoning agent module calculates cross-modal evidence consistency scores and evidence conflict degrees based on a multimodal structured evidence set, performs cross-modal semantic fusion and multi-domain risk reasoning, and generates comprehensive behavioral risk results as well as sub-domain risk results in the emotion domain, cognitive domain, and social domain. The Human-in-the-Loop Verification Module introduces human-in-the-loop verification when the comprehensive behavioral risk results meet the preset trigger conditions, to confirm or correct the risk assessment results; The risk report output module generates a standardized assessment report by combining the reasoning results from the fusion reasoning agent module and the human-in-the-loop verification results.

Images