A multi-modal data labeling processing and evaluation application system for psychological assessment

Abstract

This invention discloses a multimodal data annotation, processing, and evaluation application system for psychological assessment, relating to the field of data processing. The system includes: a multimodal data access module for receiving raw data streams from an external data acquisition device that have achieved time synchronization; the raw data streams include facial motion unit intensity data streams, heart rate data streams, and fixation point coordinate data streams; and a cross-modal association cleaning module for performing joint quality cleaning on the raw data streams according to preset cross-modal association cleaning rules. The beneficial effects of this invention are: it is the first to integrate multimodal data access, cross-modal association cleaning, spatiotemporal alignment visualization, human-computer collaborative annotation and management, and psychological assessment applications into a unified system, filling the gap in the technical chain from data acquisition to versioned annotation and then to psychological assessment applications, significantly improving research and evaluation efficiency.

Description

Technical Field

[0001] This invention relates to the field of data processing, specifically a multimodal data annotation, processing, and assessment application system for psychological evaluation. Background Technology

[0002] With the in-depth development of affective computing and psychological assessment technologies, the comprehensive assessment of an individual's psychological state through multimodal signals (such as facial expressions, heart rate, and eye movements) has become a trend. Existing technologies have enabled real-time synchronous acquisition of multimodal data based on a monocular camera.

[0003] However, the entire technology chain, from data collection to the final application used for psychological assessment, has the following prominent problems:

[0004] 1. Lack of post-data processing. Researchers typically need to use multiple independent software programs to process different modalities of data, followed by manual alignment and annotation. This is inefficient and prone to introducing time synchronization errors, and there is a lack of one-stop post-processing tools.

[0005] 2. Lack of cross-modal collaboration in data cleaning. Existing cleaning techniques are mostly designed for single-modal data and lack a joint assessment mechanism for cross-modal data quality in psychological assessment scenarios. For example, when a subject's gaze deviates from the stimulus material, their facial expressions and physiological data lose analytical significance, but existing tools cannot automatically correlate and process this, resulting in low-quality data being mixed into subsequent analyses.

[0006] 3. Isolated annotation dimensions lead to loss of multimodal correlation information. Existing annotation tools typically only support labeling operations on a single data dimension, making it difficult to create composite annotations covering multimodal signal features for the same psychological state event within the same time interval. This results in the loss of correspondences between multimodal data, significantly reducing the value of the annotated data.

[0007] 4. A disconnect exists between annotation and application. Existing technical solutions generally suffer from a disconnect between data annotation and practical application. Even after data annotation is completed, there is a lack of a mechanism to systematically transform the annotated dataset into a publicly available psychological assessment service. In practical application scenarios such as initial screening for mental health in schools, a complete technical loop is urgently needed, from multimodal data collection to the construction of annotated datasets, and then to the implementation of assessment applications.

[0008] In summary, existing technologies suffer from four core problems: fragmented processing procedures, insufficient data quality control, fragmented annotation information, and a disconnect between research and application. These problems severely restrict the industrial application of multimodal psychological assessment technologies and require improvement. Summary of the Invention

[0009] The purpose of this invention is to provide a multimodal data annotation, processing, and assessment application system for psychological evaluation, so as to solve the problems mentioned in the background art.

[0010] To achieve the above objectives, the present invention provides the following technical solution:

[0011] A multimodal data annotation, processing, and assessment application system for psychological evaluation includes:

[0012] The multimodal data access module is used to receive raw data streams that have been synchronized with time from an external data acquisition device. The raw data streams include facial motion unit intensity data streams, heart rate data streams, and gaze point coordinate data streams.

[0013] The cross-modal correlation cleaning module is used to perform joint quality cleaning on the original data stream according to the preset cross-modal correlation cleaning rules, and generate multimodal data with quality labeling information. The cross-modal correlation cleaning rules stipulate that when any modality among multiple modalities is judged to be of abnormal quality within a certain time period, the judgment result is selectively applied to the concurrent data of other modalities to realize cross-modal linkage quality labeling.

[0014] The spatiotemporal alignment visualization module is used to load the cleaned multimodal data with the corresponding stimulus materials and display them synchronously in multiple views along a unified timeline in the graphical user interface.

[0015] The human-computer collaborative annotation and management module is used to receive operation instructions from users on the graphical user interface, perform cross-modal association annotation, visualization data correction and annotation version management operations, and generate a versioned annotation dataset containing multimodal association labels;

[0016] The psychological testing application module is used to build a sample library based on versioned labeled datasets according to psychological assessment dimensions, train a multimodal psychological state assessment model, and provide psychological testing services to the outside world through API interfaces;

[0017] The output of the multimodal data access module is connected to the input of the cross-modal association and cleaning module. The output of the cross-modal association and cleaning module is connected to the first input of the spatiotemporal alignment and visualization module. The output of the spatiotemporal alignment and visualization module is connected to the input of the human-computer collaborative annotation and management module. The first output of the human-computer collaborative annotation and management module is connected to the input of the psychological assessment application module. The second output of the human-computer collaborative annotation and management module is connected to the second input of the spatiotemporal alignment and visualization module.

[0018] As a further aspect of the present invention: in the cross-modal association cleaning module, the cross-modal association cleaning rules include:

[0019] Attention association judgment rule based on fixation point: The visual attention state of the subject is judged according to the relative positional relationship between the fixation point coordinates and the display area of ​​the stimulus material. When the fixation point deviates from the display area for more than a preset time, attention state labels are generated for all modal data within that time period.

[0020] As a further aspect of the present invention: in the cross-modal association cleaning module, the cross-modal association cleaning rules include:

[0021] Data linkage labeling rules based on signal quality: According to the quality indicators of each modality signal (such as face tracking confidence and heart rate signal-to-noise ratio), time periods with signal quality below a preset threshold are labeled, and according to the selection strategy configured by the user, the label is synchronously applied to the concurrent data of other modalities.

[0022] As a further aspect of the present invention: in the spatiotemporal alignment visualization module, the graphical user interface includes:

[0023] The stimulus material playback area is used to synchronously replay video or image sequences viewed by the subjects.

[0024] At least one facial motion unit (AU) intensity curve is used to show the intensity changes of each facial motion unit with time as the horizontal axis;

[0025] Heart rate signal waveform is used to share a unified time axis with facial motion unit intensity curve to show changes in the subject's heart rate;

[0026] The fixation point display area is used to show the superimposed trajectory or heat map of the fixation point on the stimulus material screen;

[0027] A unified timeline control component is used to synchronously control the playback, pause, zoom, and time positioning of the stimulus material playback area, facial motion unit intensity curve, heart rate signal waveform, and fixation point display area.

[0028] As a further aspect of the present invention: the human-machine collaborative annotation and management module includes:

[0029] The associated annotation unit is used to generate an initial composite annotation record in response to the user's selection operation of the target time interval on a unified time axis and the user's selection operation of the target modality data features; the initial composite annotation record includes: time interval information, annotation label text, and data feature information of at least two modalities within the time interval;

[0030] The visualization correction unit allows users to correct the labeled time boundaries and data points on the facial motion unit intensity curve or gaze point display area through preset interactive operations (such as dragging and clicking), generating corrected composite annotation records.

[0031] The version management unit is used to store the original data, cleaning results, initial composite annotation records, corrected composite annotation records and all correction operation history in a versioned manner. It supports the comparison, backtracking and export of different annotation versions, and finally generates a versioned annotation dataset.

[0032] The output of the associated annotation unit is connected to the input of the visualization correction unit, the output of the visualization correction unit is connected to the input of the version management unit, the first output of the version management unit constitutes the first output of the human-computer collaborative annotation and management module, and the second output of the version management unit constitutes the second output of the human-computer collaborative annotation and management module.

[0033] As a further aspect of the present invention: the psychological assessment application module includes:

[0034] The labeled dataset construction unit is used to classify, organize, and store versioned labeled datasets according to preset psychological evaluation dimensions to form a trainable sample library. The psychological evaluation dimensions include: emotional stability, attention level, and stress response degree.

[0035] The evaluation model training unit is used to train a multimodal psychological state evaluation model using a sample library. The input of the model is multimodal physiological behavior data (facial action unit intensity data, heart rate data, and gaze point coordinate data), and the output is a quantitative score or classification result of a preset psychological evaluation dimension.

[0036] The assessment service interface unit is used to provide an application programming interface (API) for external systems to call the trained multimodal mental state assessment model;

[0037] The input of the labeled dataset construction unit constitutes the input of the psychological assessment application module. The output of the labeled dataset construction unit is connected to the input of the evaluation model training unit, and the output of the evaluation model training unit is connected to the input of the assessment service interface unit.

[0038] As a further aspect of the present invention, the application programming interface includes a annotation tool invocation function.

[0039] Compared with the prior art, the beneficial effects of the present invention are: the present invention integrates multimodal data access, cross-modal association cleaning, spatiotemporal alignment visualization, human-computer collaborative annotation and management, and psychological assessment application into a unified system for the first time, filling the gap in the technical chain from data collection to versioned annotation to psychological assessment application, and significantly improving research and evaluation efficiency;

[0040] By using cross-modal association cleaning rules, invalid data caused by continuous attention deviation and abnormal single-modal signal quality is removed from the data logic level, realizing cross-modal linkage quality labeling. This solves the cross-modal data quality problems that traditional single-modal cleaning cannot detect, ensuring the quality of data entering the labeling and training stages.

[0041] Through the cross-modal association annotation mechanism, the generated versioned annotation dataset naturally contains composite annotation records corresponding to the same psychological state event, and fully preserves the spatiotemporal correspondence between multimodal signals; compared with traditional single-modal labels, it can provide richer and more causal inference-valued supervision information for downstream multimodal psychological state assessment models;

[0042] By connecting the annotation platform with the psychological assessment application module, a complete link is established from the construction of versioned annotated datasets and the training of multimodal psychological state assessment models to application programming interface (API) services. The high-quality versioned annotated datasets produced by the annotation platform directly support the training and iteration of multimodal psychological state assessment models. The application programming interface supports both annotation tool invocation and psychological assessment service modes, realizing a seamless transformation from research tools to practical application products. It can be widely used in large-scale psychological assessment scenarios such as initial screening of school mental health. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the architecture of a multimodal data annotation, processing, and assessment application system for psychological evaluation.

[0044] Figure 2 This is an application flowchart for a multimodal data annotation, processing, and assessment system for psychological evaluation.

[0045] Figure 3 This is a schematic diagram of a graphical user interface.

[0046] Figure 4 This is a schematic diagram of the architecture of the psychological assessment application module. Detailed Implementation

[0047] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0048] Please see Figure 1 and Figure 2 A multimodal data annotation, processing, and assessment application system for psychological evaluation, comprising:

[0049] The multimodal data access module 1 is used to receive raw data streams that have been synchronized with time from an external data acquisition device. The raw data streams include facial motion unit intensity data streams, heart rate data streams, and gaze point coordinate data streams.

[0050] The cross-modal correlation cleaning module 2 is used to perform joint quality cleaning on the original data stream according to the preset cross-modal correlation cleaning rules, and generate multimodal data with quality labeling information. The cross-modal correlation cleaning rules stipulate that when any modality among multiple modalities is judged to be of abnormal quality within a certain time period, the judgment result is selectively applied to the concurrent data of other modalities to realize cross-modal linkage quality labeling.

[0051] The spatiotemporal alignment visualization module 3 is used to load the cleaned multimodal data with the corresponding stimulus materials and display them synchronously in multiple views along a unified timeline in the graphical user interface.

[0052] The human-computer collaborative annotation and management module 4 is used to receive operation instructions from users on the graphical user interface, perform cross-modal association annotation, visualization data correction and annotation version management operations, and generate a versioned annotation dataset containing multimodal association labels.

[0053] The psychological assessment application module 5 is used to build a sample library based on versioned labeled datasets according to psychological assessment dimensions, train a multimodal psychological state assessment model, and provide psychological assessment services to the outside world through API interface;

[0054] The output of the multimodal data access module 1 is connected to the input of the cross-modal association and cleaning module 2. The output of the cross-modal association and cleaning module 2 is connected to the first input of the spatiotemporal alignment and visualization module 3. The output of the spatiotemporal alignment and visualization module 3 is connected to the input of the human-computer collaborative annotation and management module 4. The first output of the human-computer collaborative annotation and management module 4 is connected to the input of the psychological assessment application module 5. The second output of the human-computer collaborative annotation and management module 4 is connected to the second input of the spatiotemporal alignment and visualization module 3.

[0055] In a specific embodiment, the system runs on a computing device and interacts with the user through a graphical user interface. The raw multimodal data to be processed comes from a prior multimodal real-time synchronous acquisition device, which has already timestamped the facial action unit (AU) intensity, heart rate, and gaze coordinates of the subjects during the experiment. When the system is running as a whole, the multimodal data access module 1 first receives the raw data stream that has been synchronized in time, automatically verifies the timestamp alignment accuracy between the data streams of each modality, and transmits it to the cross-modal association and cleaning module 2 after the verification is passed; the cross-modal association and cleaning module 2 performs joint quality cleaning and generates multimodal data with quality labels, which is then transmitted to the spatiotemporal alignment and visualization module 3 for multi-view synchronous rendering; the user sends operation instructions to the human-computer collaborative annotation and management module 4 through the graphical user interface to complete the cross-modal association annotation, visualization data correction, and annotation version management operations, generating a versioned annotation dataset; the versioned annotation dataset is transmitted to the psychological assessment application module 5 for model training and service deployment, and is also fed back to the spatiotemporal alignment and visualization module 3 through a two-way interactive channel for display updates, ultimately forming a complete technical closed loop from data access to psychological assessment services.

[0056] In this embodiment: Please refer to Figure 1 In the cross-modal association cleaning module 2, the cross-modal association cleaning rules include:

[0057] Attention association judgment rule based on fixation point: The visual attention state of the subject is judged according to the relative positional relationship between the fixation point coordinates and the display area of ​​the stimulus material. When the fixation point deviates from the display area for more than a preset time, attention state labels are generated for all modal data within that time period.

[0058] When the gaze point is detected to deviate from the display area for more than the user-configurable preset duration, the period is determined to be an inattentive period. An inattentive state marker is automatically generated for all modal data (such as facial motion unit intensity data and heart rate data) within the period, realizing cross-modal linkage quality marking. The inattentive state marker will be automatically identified and removed in subsequent annotation and training stages to ensure that subsequent analysis only uses valid data from the period when the subject's attention is focused.

[0059] In this embodiment: Please refer to Figure 1 In the cross-modal association cleaning module 2, the cross-modal association cleaning rules include:

[0060] Data linkage labeling rules based on signal quality: According to the quality indicators of each modality signal (such as face tracking confidence and heart rate signal-to-noise ratio), time periods with signal quality below a preset threshold are labeled, and according to the selection strategy configured by the user, the label is synchronously applied to the concurrent data of other modalities.

[0061] During the execution of the data linkage labeling rules based on signal quality, the cross-modal association and cleaning module 2 monitors the quality indicators of each modality signal in real time, including face tracking confidence and heart rate signal-to-noise ratio. When the signal quality indicator of a certain modality is lower than the preset threshold, the system automatically generates a low-quality label for the corresponding modality data in that time period, and applies the low-quality label to the concurrent data of other modalities according to the linkage strategy pre-configured by the user. Users can flexibly configure different linkage strategies according to experimental needs, such as only linking the face tracking low-quality label to the heart rate data, or linking it to all modality data at the same time, so as to realize the linkage control of single-modal quality problems to multi-modal, and avoid low-quality data from being mixed into the subsequent labeling and training stages.

[0062] Attention association judgment rules based on foveation points and data linkage labeling rules based on signal quality can be used individually or in combination.

[0063] In this embodiment: Please refer to Figure 3 In the spatiotemporal alignment visualization module 3, the graphical user interface includes:

[0064] The stimulus material playback area is used to synchronously replay video or image sequences viewed by the subjects.

[0065] At least one facial motion unit (AU) intensity curve is used to show the intensity changes of each facial motion unit with time as the horizontal axis;

[0066] Heart rate signal waveform is used to share a unified time axis with facial motion unit intensity curve to show changes in the subject's heart rate;

[0067] The fixation point display area is used to show the superimposed trajectory or heat map of the fixation point on the stimulus material screen;

[0068] A unified timeline control component is used to synchronously control the playback, pause, zoom, and time positioning of the stimulus material playback area, facial motion unit intensity curve, heart rate signal waveform, and fixation point display area.

[0069] When the spatiotemporal alignment visualization module 3 runs, it first loads the cleaned multimodal data and the corresponding stimulus material files, and precisely aligns all the data according to a unified timestamp. Then, in the graphical user interface, it renders the stimulus material playback area, facial action unit (AU) intensity curve, heart rate signal waveform, and gaze point display area, all of which share a unified time axis. The gaze point display area supports two display modes: overlay trajectory and heat map, which users can switch freely on the interface. When the user performs playback, pause, zoom, or time positioning operations through the unified time axis control component, the system synchronously sends time positioning signals to all visualization areas, so that each area is updated to the display content of the corresponding time point. Moreover, the zoom operation will be synchronously applied to the time axis of all areas, making it easier for users to observe data details or global change trends, and realizing the spatiotemporal fusion display of multimodal data.

[0070] In this embodiment: Please refer to Figure 1 The human-machine collaborative annotation and management module 4 includes:

[0071] The associated annotation unit is used to generate an initial composite annotation record in response to the user's selection operation of the target time interval on a unified time axis and the user's selection operation of the target modality data features; the initial composite annotation record includes: time interval information, annotation label text, and data feature information of at least two modalities within the time interval;

[0072] The visualization correction unit allows users to correct the labeled time boundaries and data points on the facial motion unit intensity curve or gaze point display area through preset interactive operations (such as dragging and clicking), generating corrected composite annotation records.

[0073] The version management unit is used to store the original data, cleaning results, initial composite annotation records, corrected composite annotation records and all correction operation history in a versioned manner. It supports the comparison, backtracking and export of different annotation versions, and finally generates a versioned annotation dataset.

[0074] The output of the associated annotation unit is connected to the input of the visualization correction unit, the output of the visualization correction unit is connected to the input of the version management unit, the first output of the version management unit constitutes the first output of the human-computer collaborative annotation and management module 4, and the second output of the version management unit constitutes the second output of the human-computer collaborative annotation and management module 4.

[0075] When the human-computer collaborative annotation and management module 4 is running, users can replay stimulus materials and simultaneously observe changes in multimodal data through a graphical user interface. When a psychological state event worthy of annotation is found, the playback is paused, and the time interval of the event is selected by dragging on a unified time axis. The associated annotation unit receives the user's selection operation and the selection operation of the target modal data features, automatically extracts data features such as the intensity value of the main facial action units, the trend of heart rate changes, and the distribution area of ​​fixation points within the interval, and generates an initial composite annotation record. The initial composite annotation record is transmitted to the visualization correction unit, where users can correct the annotated time boundaries and data points through interactive operations such as dragging and clicking, for example, on the face. The precise start and end boundaries of events can be directly dragged and adjusted on the intensity curve of the action unit, or the drifted gaze point data can be corrected by clicking on the gaze point display area to generate a corrected composite annotation record. The corrected composite annotation record is transmitted to the version management unit, which simultaneously receives the original data and cleaning results, and stores all data, annotation records and correction operation history in a versioned manner. It supports users to create multiple independent annotation versions, compare differences between different versions, and export the final selected version as a standard format data file, ultimately generating a versioned annotation dataset, which is connected to the psychological assessment application module 5 and the spatiotemporal alignment visualization module 3 through bidirectional output terminals.

[0076] In this embodiment: Please refer to Figure 4 The psychological assessment application module 5 includes:

[0077] The labeled dataset construction unit is used to classify, organize, and store versioned labeled datasets according to preset psychological evaluation dimensions to form a trainable sample library. The psychological evaluation dimensions include: emotional stability, attention level, and stress response degree.

[0078] The evaluation model training unit is used to train a multimodal psychological state evaluation model using a sample library. The input of the model is multimodal physiological behavior data (facial action unit intensity data, heart rate data, and gaze point coordinate data), and the output is a quantitative score or classification result of a preset psychological evaluation dimension.

[0079] The assessment service interface unit is used to provide an application programming interface (API) for external systems to call the trained multimodal mental state assessment model;

[0080] The input end of the labeled dataset construction unit constitutes the input end of the psychological assessment application module 5. The output end of the labeled dataset construction unit is connected to the input end of the evaluation model training unit, and the output end of the evaluation model training unit is connected to the input end of the assessment service interface unit.

[0081] When the psychological assessment application module 5 runs, the labeled dataset construction unit first receives the versioned labeled dataset from the human-computer collaborative labeling and management module 4. It then categorizes, organizes, and stores the data according to preset psychological assessment dimensions such as emotional stability, attention level, and stress response level. Specifically, data labeled "distracted attention" and "concentrated attention" are assigned to the attention level sample library; data labeled "emotionally agitated" and "emotionally calm" are assigned to the emotional stability sample library; and data labeled "stress response" and related physiological changes are assigned to the stress response level sample library, forming a trainable sample library. The assessment model training unit uses a supervised learning algorithm, with data from the sample library that has been manually labeled... The labeled psychological state tags serve as supervisory signals, learning the mapping relationship between multimodal data and psychological states to train a multimodal psychological state assessment model. This model takes facial motion unit intensity data, heart rate data, and gaze point coordinate data as input and outputs quantitative scores or classification results for each psychological assessment dimension. The trained model is transmitted to the assessment service interface unit and encapsulated as a standardized application programming interface (API). The API supports the calling of the assessment model for external systems. Application users, such as school mental health screening systems, can submit students' multimodal physiological behavior data through this interface to obtain students' scores on each assessment dimension, assisting in mental health screening decisions.

[0082] In this embodiment: Please refer to Figure 4 The application programming interface includes annotation tool invocation functionality.

[0083] The application programming interface (API) supports two independent modes: annotation tool invocation and assessment model invocation, meeting the differentiated needs of different user groups. When the annotation tool invocation function is running, research users can send requests to the system through the API, uploading raw multimodal data to be processed. The system automatically calls the cross-modal association cleaning module 2 to perform joint quality cleaning on the uploaded data, and then calls the human-computer collaborative annotation and management module 4 to perform cross-modal association annotation, visual data correction, and annotation version management operations. After processing, the system returns a standardized versioned annotation dataset to the research user through this interface, meeting the research user's needs for batch processing and annotation of multimodal data, and realizing the cloudification and service-oriented nature of the annotation tool. The assessment model invocation interface, on the other hand, is geared towards application-end users, providing standardized psychological state assessment services.

[0084] This invention is the first to integrate multimodal data access, cross-modal association and cleaning, spatiotemporal alignment visualization, human-computer collaborative annotation and management, and psychological assessment applications into a unified system, filling the gap in the technology chain from data collection to versioned annotation and then to psychological assessment applications, and significantly improving research and evaluation efficiency.

[0085] By using cross-modal association cleaning rules, invalid data caused by continuous attention deviation and abnormal single-modal signal quality is removed from the data logic level, realizing cross-modal linkage quality labeling. This solves the cross-modal data quality problems that traditional single-modal cleaning cannot detect, ensuring the quality of data entering the labeling and training stages.

[0086] Through the cross-modal association annotation mechanism, the generated versioned annotation dataset naturally contains composite annotation records corresponding to the same psychological state event, and fully preserves the spatiotemporal correspondence between multimodal signals; compared with traditional single-modal labels, it can provide richer and more causal inference-valued supervision information for downstream multimodal psychological state assessment models;

[0087] By connecting the annotation platform with the psychological assessment application module 5, a complete link is established from the construction of versioned annotated datasets and the training of multimodal psychological state assessment models to application programming interface (API) services. The high-quality versioned annotated datasets produced by the annotation platform directly support the training and iteration of multimodal psychological state assessment models. The application programming interface supports both annotation tool invocation and psychological assessment service modes, realizing a seamless transformation from research tools to practical application products. It can be widely used in large-scale psychological assessment scenarios such as initial screening of school mental health.

[0088] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and not restrictive.

[0089] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A multimodal data annotation, processing, and assessment application system for psychological evaluation, characterized in that, The multimodal data annotation, processing, and assessment application system for psychological evaluation includes: The multimodal data access module is used to receive raw data streams that have been synchronized with time from an external data acquisition device. The raw data streams include facial motion unit intensity data streams, heart rate data streams, and gaze point coordinate data streams. The cross-modal correlation cleaning module is used to perform joint quality cleaning on the original data stream according to the preset cross-modal correlation cleaning rules, and generate multimodal data with quality labeling information. The cross-modal correlation cleaning rules stipulate that when any modality among multiple modalities is judged to be of abnormal quality within a certain time period, the judgment result is selectively applied to the concurrent data of other modalities to realize cross-modal linkage quality labeling. The spatiotemporal alignment visualization module is used to load the cleaned multimodal data with the corresponding stimulus materials and display them synchronously in multiple views along a unified timeline in the graphical user interface. The human-computer collaborative annotation and management module is used to receive operation instructions from users on the graphical user interface, perform cross-modal association annotation, visualization data correction and annotation version management operations, and generate a versioned annotation dataset containing multimodal association labels; The psychological testing application module is used to build a sample library based on versioned labeled datasets according to psychological assessment dimensions, train a multimodal psychological state assessment model, and provide psychological testing services to the outside world through API interfaces; The output of the multimodal data access module is connected to the input of the cross-modal association and cleaning module. The output of the cross-modal association and cleaning module is connected to the first input of the spatiotemporal alignment and visualization module. The output of the spatiotemporal alignment and visualization module is connected to the input of the human-computer collaborative annotation and management module. The first output of the human-computer collaborative annotation and management module is connected to the input of the psychological assessment application module. The second output of the human-computer collaborative annotation and management module is connected to the second input of the spatiotemporal alignment and visualization module.

2. The multimodal data annotation, processing, and assessment application system for psychological evaluation according to claim 1, characterized in that, In the cross-modal association cleaning module, the cross-modal association cleaning rules include: Attention association judgment rule based on fixation point: The visual attention state of the subject is judged according to the relative positional relationship between the fixation point coordinates and the display area of ​​the stimulus material. When the fixation point deviates from the display area for more than a preset time, attention state labels are generated for all modal data within that time period.

3. The multimodal data annotation, processing, and assessment application system for psychological evaluation according to claim 1 or 2, characterized in that, In the cross-modal association cleaning module, the cross-modal association cleaning rules include: Data linkage marking rules based on signal quality: According to the quality indicators of each modal signal, time periods when the signal quality is lower than a preset threshold are marked, and according to the selection strategy configured by the user, the mark is synchronously applied to the synchronous data of other modalities.

4. The multimodal data annotation, processing, and assessment application system for psychological evaluation according to claim 1, characterized in that, The spatiotemporal alignment visualization module includes the following graphical user interface: The stimulus material playback area is used to synchronously replay video or image sequences viewed by the subjects. At least one facial motion unit intensity curve is used to show the intensity changes of each facial motion unit with time as the horizontal axis. Heart rate signal waveform is used to share a unified time axis with facial motion unit intensity curve to show changes in the subject's heart rate; The fixation point display area is used to show the superimposed trajectory or heat map of the fixation point on the stimulus material screen; A unified timeline control component is used to synchronously control the playback, pause, zoom, and time positioning of the stimulus material playback area, facial motion unit intensity curve, heart rate signal waveform, and fixation point display area.

5. The multimodal data annotation, processing, and assessment application system for psychological evaluation according to claim 1, characterized in that, The human-machine collaborative annotation and management module includes: The associated annotation unit is used to generate an initial composite annotation record in response to the user's selection operation of the target time interval on a unified time axis and the user's selection operation of the target modality data features; the initial composite annotation record includes: time interval information, annotation label text, and data feature information of at least two modalities within the time interval; The visualization correction unit allows users to correct the labeled time boundaries and data points on the facial motion unit intensity curve or gaze point display area through preset interactive operations, generating corrected composite annotation records. The version management unit is used to store the original data, cleaning results, initial composite annotation records, corrected composite annotation records and all correction operation history in a versioned manner. It supports the comparison, backtracking and export of different annotation versions, and finally generates a versioned annotation dataset. The output of the associated annotation unit is connected to the input of the visualization correction unit, the output of the visualization correction unit is connected to the input of the version management unit, the first output of the version management unit constitutes the first output of the human-computer collaborative annotation and management module, and the second output of the version management unit constitutes the second output of the human-computer collaborative annotation and management module.

6. The multimodal data annotation, processing, and assessment application system for psychological evaluation according to claim 1, characterized in that, The psychological assessment application module includes: The labeled dataset construction unit is used to classify, organize, and store versioned labeled datasets according to preset psychological evaluation dimensions to form a trainable sample library. The psychological evaluation dimensions include: emotional stability, attention level, and stress response degree. The evaluation model training unit is used to train a multimodal psychological state evaluation model using a sample library. The input of the model is multimodal physiological and behavioral data, and the output is a quantitative score or classification result of a preset psychological evaluation dimension. The assessment service interface unit is used to provide an application programming interface for external systems to call the trained multimodal mental state assessment model; The input of the labeled dataset construction unit constitutes the input of the psychological assessment application module. The output of the labeled dataset construction unit is connected to the input of the evaluation model training unit, and the output of the evaluation model training unit is connected to the input of the assessment service interface unit.

7. The multimodal data annotation, processing, and assessment application system for psychological evaluation according to claim 6, characterized in that, The application programming interface includes annotation tool invocation functionality.

Images