Method for generating multi-modal training data, training method and electronic equipment
By generating multimodal training data, the problem of insufficient single-modal data in psychological counseling models is solved, and the alignment and segmentation of multimodal data are realized, thereby improving the quality and efficiency of counseling reports.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- DIGITAL NINGXIA CONSTRUCTION & OPERATION CO LTD
- Filing Date
- 2025-12-30
- Publication Date
- 2026-05-12
AI Technical Summary
In existing technologies, the training data for psychological counseling models are mostly single-modal, which makes it impossible for the models to fully capture the emotional state and psychological characteristics of clients. The report analysis dimensions are incomplete, and there is a lack of cross-modal data timestamp alignment mechanism, making it difficult to support the training needs of report generation models.
By acquiring pure text-based synthetic dialogues and combining them with virtual physiological data time series and virtual video target frame descriptions to generate a multimodal synthetic dataset, we can collect multimodal information from real dialogues, process it into a unified annotation format, and achieve alignment, segmentation, and encapsulation of multimodal training data, thereby improving the information content and adaptability of the data.
It significantly improves the information content of training data and the training effect of models, enhances the quality and efficiency of generated consulting reports, and ensures the professionalism and reliability of reports.
Smart Images
Figure CN122020157A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of psychological counseling, and more specifically, the embodiments of this application relate to a method for generating multimodal training data, a training method, and an electronic device. Background Technology
[0002] As large-scale multimodal models (LMMs) are increasingly used in the field of mental health, the automatic generation of professional psychological counseling reports has become an important requirement for improving counseling efficiency. However, the performance of these models is largely affected by the training data.
[0003] The training data provided by the relevant technologies have at least the following technical defects: the use of training data with a single modality results in fewer features that the model can learn, which in turn affects the quality of the psychological counseling reports output by the model. Summary of the Invention
[0004] The purpose of this application is to provide a method for generating multimodal training data, a training method, and an electronic device. The embodiments of this application can provide multimodal training data and can unify synthetic dialogue and real dialogue into a labeling format, thereby improving the fit between the training data and the trained model and improving the training effect of the model.
[0005] In a first aspect, embodiments of this application provide a method for generating multimodal training data, the method comprising: acquiring a synthetic dialogue of plain text type; determining a virtual physiological data time series and a virtual video target frame description adapted to the synthetic dialogue to obtain a multimodal synthetic dataset, wherein the virtual physiological data time series is used to characterize the change feature sequence of physiological indicators matching the synthetic dialogue, and the virtual video target frame description is a textual description of visual information at at least one target moment based on the emotional state of the synthetic dialogue; collecting multimodal information of a real dialogue process to obtain a multimodal real dataset, wherein the multimodal real dataset includes: real dialogue text data, a physiological data time series corresponding to the real dialogue process, and a video target frame description; and obtaining multimodal training data based on the multimodal real dataset and the multimodal synthetic dataset.
[0006] The embodiments of this application can significantly increase the amount of information carried by the obtained training data by setting other modal information for the synthetic dialogue, thereby improving the quality of the training model obtained from the multimodal training data.
[0007] In some embodiments, obtaining multimodal training data based on the multimodal real dataset and the multimodal synthetic dataset includes: aligning the multimodal data included in each dialogue in the multimodal real dataset or the multimodal synthetic dataset to obtain an aligned multimodal data stream; and segmenting and encapsulating the aligned multimodal data stream according to a set duration to obtain the multimodal training data.
[0008] Some embodiments of this application also require aligning multimodal data and segmenting and encapsulating the aligned data to obtain more structured training data that can be directly called, thereby improving the model training speed.
[0009] In some embodiments, both the real multimodal dataset and the synthetic multimodal dataset include: text modal data and non-text modal data; aligning the multimodal data included in each dialogue in the real multimodal dataset or the synthetic multimodal dataset to obtain an aligned multimodal data stream includes: using the time axis corresponding to the text modality as a reference time axis, converting the text modal data into a time series to obtain a reference sequence; extracting feature sequences from the non-text modal data to obtain at least one sequence to be compared; aligning each sequence to be compared with the reference sequence to obtain the aligned multimodal data stream.
[0010] Some embodiments of this application use the time axis corresponding to the text modality data as a reference to align the feature sequences of other modalities with the reference sequence, thereby improving the alignment effect of multimodal data.
[0011] In some embodiments, the non-text modal data includes: video modal data and audio modal data; the step of extracting feature sequences from the non-text modal data to obtain at least one sequence to be compared includes: extracting features from each frame of the video modal data and sorting the extraction results by time to obtain a video feature sequence; extracting audio energy from the audio modal data within a first duration and sorting the extraction results by time to obtain an audio feature sequence, wherein the first duration is a preset duration.
[0012] Some embodiments of this application provide a specific feature type for the feature sequence of video modal data and audio modal data during alignment, thereby improving the consistency of the technical solution.
[0013] In some embodiments, aligning each sequence to be compared with the reference sequence to obtain the aligned multimodal data stream includes: calculating a target distance path between each sequence to be compared and the reference sequence; bending the time axis of the non-reference modality according to the target distance path to achieve alignment, wherein the non-reference modality includes: a video modality corresponding to the video modality data, a physiological modality corresponding to the physiological data time series, and an audio modality corresponding to the audio modality data.
[0014] Some embodiments of this application achieve alignment of multimodal data by comparing the distance path between the sequence to be aligned and the reference sequence, as well as the curvature of the time axis, thereby improving the alignment effect.
[0015] In some embodiments, the step of segmenting and encapsulating the aligned multimodal data stream according to a set duration to obtain the multimodal training data includes: determining the identifier and time interval information of the data blocks corresponding to each set duration; establishing a correspondence between the multimodal data subsets corresponding to each time interval information and the corresponding data block identifiers, wherein the multimodal data subsets are obtained by segmenting the aligned multimodal data stream according to the set duration, and the multimodal data subsets include: audio data subsets, video data subsets, physiological data subsets, and text data subsets, with one data subset corresponding to one set duration; and obtaining the multimodal training data at least according to the correspondence.
[0016] Some embodiments of this application obtain structured training data by segmenting data into blocks and encapsulating them.
[0017] In some embodiments, the audio data subset includes: audio data and the transcribed text of the audio data; the video data subset includes: target video frames and emotion tags corresponding to the target video frames; the physiological data subset includes physiological data time series; and the text data subset includes transcribed text of real consultation dialogues and / or text of synthesized dialogues.
[0018] Some embodiments of this application define a structured data subset pattern to improve the consistency of technical solutions and ensure the uniformity of training data structures.
[0019] In some embodiments, obtaining the multimodal training data based at least on the correspondence includes: extracting multimodal features from each data block, wherein the multimodal features include: text semantic features, audio emotional features, video facial features, and physiological data statistical features; and associating and organizing the multimodal features according to preset modules of the consultation report to obtain the multimodal training data.
[0020] The embodiments of this application accurately associate each modality data with timestamps and complete data structuring according to the diagnostic report generation requirements, thereby improving the adaptability of data and models.
[0021] Secondly, some embodiments of this application provide a method for model training, the method comprising: obtaining multimodal training data according to the method described in the first aspect; and training a consultation report generation model based on the multimodal training data.
[0022] Thirdly, some embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the methods described in the above embodiments.
[0023] Fourthly, some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the multi-sequence alignment retrieval method as described in the first aspect.
[0024] Fifthly, some embodiments of this application provide a computer program product, including computer program instructions, which, when read and executed by a processor, can implement the methods described in the above embodiments. Attached Figure Description
[0025] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0026] Figure 1 An architecture diagram of a system for generating multimodal training data and applying the training data to train a model, as provided in an embodiment of this application;
[0027] Figure 2 One of the flowcharts for a method of generating multimodal training data provided in the embodiments of this application;
[0028] Figure 3 A second flowchart illustrating the method for generating multimodal training data provided in this application embodiment;
[0029] Figure 4 The fourth flowchart of the method for generating multimodal training data provided in the embodiments of this application;
[0030] Figure 5 A block diagram illustrating the components of an apparatus for generating multimodal training data provided in an embodiment of this application;
[0031] Figure 6 This is a schematic diagram illustrating the composition of an electronic device provided in an embodiment of this application. Detailed Implementation
[0032] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.
[0033] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0034] The inventors of this application discovered in their research that the accuracy and comprehensiveness of reports in the field of psychological counseling highly depend on multi-dimensional data support (i.e., a greater quantity and higher quality of training data is required), but existing training data for these models has the following shortcomings:
[0035] 1. Limited Data Modality: Traditional psychological data collection often relies on text notes or single audio recordings, ignoring non-verbal cues in videos such as facial expressions and body language, as well as physiological indicators such as heart rate and blood oxygen. This results in models failing to fully capture the emotional state and psychological characteristics of visitors, leading to incomplete report analysis dimensions.
[0036] 2. Lack of standardization in data acquisition and processing: The selection of data acquisition equipment for various modalities is chaotic and the parameters are inconsistent (such as poor audio noise reduction and insufficient video resolution). The data formats are messy (such as text without fixed templates and video without keyframe markers). At the same time, there is a lack of cross-modal data timestamp alignment mechanism, which makes it difficult to effectively integrate multi-source information and support the training needs of the report generation model.
[0037] 4. Poor data and report requirements: Most of the existing data is raw data that has not been structured for the core modules of the consultation report (such as symptom assessment, intervention process, and effect analysis). The model needs to consume a lot of computing power to parse the data, and key information in the report is prone to being omitted or incorrect.
[0038] 5. Inadequate quality control system: The lack of a two-tiered quality control mechanism that balances format standardization and clinical professionalism means that issues such as errors in clinical terminology, emotional labeling bias, and modal information conflicts in the data cannot be effectively filtered out, directly affecting the professionalism and reliability of the generated reports.
[0039] In summary, existing technologies cannot meet the core needs of mental health counseling for the automatic generation of compliant, standardized, high-quality, and highly adaptable multimodal data. There is an urgent need to build a fully controllable multimodal data collection and production system.
[0040] At least to achieve the above-mentioned technical objectives, embodiments of this application provide a method for multimodal training data. This method generates multimodal information for synthetic dialogue and processes real multimodal data and multimodal synthetic data into training data in a standard format. Training the model based on this training data can improve the quality of the model's output consultation reports for a target domain (e.g., psychological counseling).
[0041] Please refer to Figure 1 , Figure 1 The system for generating multimodal training data and applying the training data to train models, as provided in some embodiments of this application, includes: a multimodal information acquisition device 110, an acquisition server 120, an application server 130, a storage server 140, and an AI training server 150.
[0042] Figure 1 The multimodal information acquisition device 100 includes:
[0043] Audio equipment is used to record audio information from a real conversation (e.g., a counseling session). Audio acquisition equipment is used to record the content of the counseling conversation and can perform the following operations based on the content: sound source localization to distinguish the voices of the counselor and the client; voice quality assurance to provide professional-grade audio quality and reduce environmental noise interference; and emotional tone analysis to provide high-quality voice data for subsequent emotional analysis.
[0044] For example, the audio device can be a combination of a digital recorder and a microphone, with the background noise set to ≤30 dB, a sound sampling rate of 44.1 kHz, a bit depth of 16 bits, and an output WAV / FLAC format. The device's sound quality is tested before recording.
[0045] Video equipment is used to record video during a real conversation (e.g., a psychological counseling conversation). The video capture equipment in this application embodiment can record video of the counseling process, ensuring image clarity. This video can be used for: facial expression capture to record changes in facial expressions of the counselor and client; body language recording to capture nonverbal communication information such as body posture and gestures; and environmental monitoring to record the overall situation of the counseling environment.
[0046] For example, the video device can use a 1080P high-definition camera (45° side angle), a frame rate of 30fps, uniform lighting without shadows, output MP4 / MOV format, and mark keyframes of emotional change (JPG format) using a CV algorithm.
[0047] Text acquisition devices are used to collect data from text related to a real conversation.
[0048] For example, this text acquisition device uses SOAP / BIRP templates to collect medical history, conversation transcription, and progress records, converting them into standardized Markdown format with a transcription accuracy of ≥98%.
[0049] A physiological data acquisition device is used to collect physiological data from the subject of a real conversation. The physiological data acquisition device in this application embodiment can achieve: multi-channel physiological signal acquisition, simultaneously acquiring signals such as electrocardiogram, skin conductance response, and electromyography; real-time physiological monitoring, continuously monitoring changes in the visitor's physiological state; quantification of emotional responses, objectively quantifying emotional fluctuations through physiological indicators; and precise data synchronization, ensuring the temporal consistency of physiological data with other modal data.
[0050] For example, this physiological data acquisition device collects heart rate (1 time, per second) and blood oxygen (1 time, 30 seconds) data from wearable devices and outputs JSON format (including precise timestamps).
[0051] Figure 1 The acquisition server 120 is configured to collect real-time data streams from all acquisition devices, perform preliminary cleaning and format standardization on the raw data, manage data inflow and outflow, prevent data loss, and monitor the operating status of each acquisition device in real time.
[0052] Figure 1 The storage server's functions include: multimodal data storage: centralized storage of video, audio, physiological signals, and text data; data backup management: regular backup of important data to ensure data security; storage resource allocation: optimization of storage strategies based on data type and access frequency; and fast data retrieval: support for efficient data querying and access.
[0053] Figure 1 The AI training server is used for model training computation and can provide GPU computing power to support deep learning model training; multimodal fusion analysis, processing and analyzing cross-modal data association patterns; feature extraction optimization, automatically extracting and optimizing feature representations of multimodal data; model version management: managing and maintaining different versions of AI models.
[0054] Figure 1 The application server's functions include: business logic processing, executing the processing steps of psychological counseling data analysis (i.e., methods for generating multimodal training data); and data processing, implementing data segmentation, alignment, encapsulation, and other processing flows for multimodal data streams.
[0055] Some embodiments of this application also include: API service provision for providing data access and processing interfaces for front-end applications; and system task scheduling for managing and scheduling various processing tasks in the system.
[0056] The following is combined Figure 2An exemplary embodiment of this application provides a method for generating multimodal training data, the method comprising:
[0057] S210, Obtain the synthesized dialogue in plain text type.
[0058] For example, in some embodiments of this application, S210 exemplarily includes: obtaining a structured seed dataset, wherein the seed dataset includes multiple dialogue samples and / or report samples, the dialogue samples having role labels adapted to the target domain, and the report samples having multiple different modules adapted to the target domain; constructing an initial instruction pool based on the seed dataset, wherein the initial instruction pool includes multiple sets of instruction-response pairs, each instruction describing a task requirement adapted to the target domain, and each response being a task output matching the task requirement; generating synthetic instructions through an instruction generation model and the initial instruction pool, and generating synthetic responses corresponding to the synthetic instructions through a response generation model and each response in the initial instruction pool; obtaining target synthetic sample data based on the synthetic instructions and the synthetic responses, wherein the target synthetic sample includes synthetic dialogues as synthetic responses.
[0059] It is understood that those skilled in the art can also use other models to learn the features of real dialogues and then output synthetic dialogues. The embodiments of this application do not limit the specific method of obtaining text-based synthetic dialogues. For example, some embodiments of this application can generate synthetic dialogues based on Self-Instruct and Multi-Agent methods.
[0060] S220, determine the virtual physiological data time series and virtual video target frame description adapted to the synthetic dialogue to obtain a multimodal synthetic dataset.
[0061] It should be noted that, in some embodiments of this application, the virtual physiological data time series is used to characterize the change feature sequence of physiological indicators matching the synthetic dialogue, and the virtual video target frame description is a textual description of visual information at at least one target moment based on the emotional state of the synthetic dialogue.
[0062] It is understood that, in some embodiments of this application, virtual physiological data time series refers to time series data that simulates real physiological responses, generated based on the emotional content of a dialogue. These virtual physiological data time series have the following characteristics: data nature—not actually measured, but simulated data generated based on psychological principles; content composition—including sequences of physiological indicators such as heart rate, skin conductance, and respiration changing over time; generation basis—generated based on the emotional development trajectory and content characteristics of the dialogue.
[0063] In some embodiments of this application, video target frame description (also known as video key frame description) refers to the textual description of key visual moments during the consultation process. These video target frame descriptions have the following characteristics: description content: visual information such as facial expressions, body language, and posture changes; temporal positioning: precisely corresponding to the dialogue content in time; generation method: textual descriptions generated based on dialogue content and emotional state (not real images).
[0064] For example, some embodiments of this application employ Figure 3 The trained multimodal synthesis model yields virtual physiological data time series and virtual video target frame descriptions adapted to the synthesized dialogue. The corresponding training process is as follows: Figure 3 The process includes: S310, collecting real multimodal data, i.e., using multimodal data related to multiple real dialogues; S320, labeling the collected multimodal data to obtain labeled data, for example, labeling the physiological parameter feature types and video information feature types corresponding to different emotions; S330, training a multimodal synthesis model using real multimodal data, and after training, the model outputs a virtual physiological data time series corresponding to a certain synthesized dialogue under the input condition, and outputs a virtual video target frame description corresponding to the video keyframe description of the dialogue.
[0065] For example, some embodiments of this application employ Figure 4 The resulting mapping rule base yields virtual physiological data time series and virtual video target frame descriptions adapted to the synthetic dialogue. For example... Figure 4 As shown, the process of constructing the mapping rule base includes: S410, collecting real multimodal data, i.e., using multimodal data related to multiple real dialogues, including text, audio, video, and physiological data; S420, establishing emotional-physiological mapping relationships and emotional expression relationships; S330, constructing the mapping rule base. Then, by searching the mapping rule base, the physiological data time series matching the relevant moments of a certain synthesized dialogue is obtained to obtain the virtual physiological data time series, and the virtual video target frame description matching the video keyframe description of that dialogue is found.
[0066] S230, Collect multimodal information of the real dialogue process to obtain a multimodal real dataset, wherein the multimodal real dataset includes: real dialogue text data, physiological data time series corresponding to the real dialogue process, and video target frame description.
[0067] It should be noted that the multimodal real-world dataset here is obtained through... Figure 1 The multimodal information acquisition device 110 acquires data from a real dialogue process. Since there are many types of acquisition devices, multimodal data can be obtained for a single real dialogue.
[0068] S240, obtain multimodal training data based on the multimodal real dataset and the multimodal synthetic dataset.
[0069] For example, in some embodiments of this application, it is necessary to align the data in the multimodal real dataset and the multimodal synthetic dataset, and then segment and encapsulate the aligned data. Afterwards, the encapsulated data can be mapped to the various functional modules included in the report of the target domain, thus obtaining training data specifically for training the report generation model of that target domain. For example, the target domain could be the field of psychological counseling.
[0070] The implementation process of the relevant steps is described below.
[0071] For example, in some embodiments of this application, the step S240 of obtaining multimodal training data based on the multimodal real dataset and the multimodal synthetic dataset includes:
[0072] The first step is to align the multimodal data included in each dialogue in the real multimodal dataset or the synthetic multimodal dataset to obtain an aligned multimodal data stream.
[0073] Within the duration corresponding to a single dialogue, the data of each modality corresponding to the single dialogue in the real multimodal dataset or the synthetic multimodal dataset are aligned to obtain an aligned multimodal data stream (for example, in the field of counseling, the aligned multimodal data stream is continuous multimodal data of a single counseling session).
[0074] In some embodiments of this application, both the multimodal real dataset and the multimodal synthetic dataset include: text modal data and non-text modal data. The first step typically includes: using the time axis corresponding to the text modality as a reference time axis, converting the text modal data into a time series to obtain a reference sequence; extracting feature sequences from the non-text modal data to obtain at least one sequence to be compared; aligning each sequence to be compared with the reference sequence to obtain the aligned multimodal data stream.
[0075] For example, in some embodiments of this application, the non-text modal data includes: video modal data, physiological data time series, and audio modal data; then, extracting the feature sequence of the non-text modal data to obtain at least one comparison sequence includes: extracting features from each frame of the video modal data and sorting the extraction results by time to obtain a video feature sequence; extracting the audio energy of the audio modal data within a first duration and sorting the extraction results by time to obtain an audio feature sequence; using the physiological data time series as the comparison sequence, or reusing or interpolating the physiological data time series to obtain the comparison sequence.
[0076] For example, in some embodiments of this application, aligning each sequence to be compared with the reference sequence to obtain the aligned multimodal data stream includes: calculating a target distance path between each sequence to be compared and the reference sequence; bending the time axis of the non-reference modality according to the target distance path to achieve alignment, wherein the non-reference modality includes: a video modality corresponding to the video modality data, a physiological modality corresponding to the physiological data time series, and an audio modality corresponding to the audio modality data.
[0077] For example, in some embodiments of this application, the time series may involve dividing the text into sentences or words and recording the start and end times of each unit. For example, in some embodiments of this application, the time series may involve discretizing the entire conversation timeline to the millisecond level, and then representing each time point with a feature value, such as 1 to indicate the presence of speech, 0 to indicate silence, or more complex features such as speech energy, tone of voice, etc., or focusing on the rhythm and content changes of the dialogue.
[0078] The second step is to segment and encapsulate the aligned multimodal data stream according to a set duration to obtain the multimodal training data.
[0079] It should be noted that if the dialogue is a synthesized dialogue, the duration is the virtual duration allocated to the synthesized object; if the dialogue is a real dialogue, the duration is the duration of the real dialogue.
[0080] The following example illustrates the process of segmenting and encapsulating an aligned multimodal data stream.
[0081] In some embodiments of this application, step S240, which involves segmenting and encapsulating the aligned multimodal data stream according to a set duration to obtain the multimodal training data, includes:
[0082] The first step is to determine the identifier and time interval information of the data blocks corresponding to each set duration.
[0083] The second step is to establish a correspondence between the multimodal data subsets corresponding to each time interval information and the corresponding data block identifiers. The multimodal data subsets are obtained by segmenting the aligned multimodal data stream according to the set duration. The multimodal data subsets include: audio data subsets, video data subsets, physiological data subsets and text data subsets, with each data subset corresponding to a set duration.
[0084] For example, in some embodiments of this application, the audio data subset includes: audio data and the transcribed text of the audio data; the video data subset includes: video target frames and emotion tags corresponding to the video target frames; the physiological data subset includes physiological data time series; and the text data subset includes transcribed text of real consultation dialogues and / or text of synthesized dialogues.
[0085] The third step is to obtain the multimodal training data based at least on the correspondence.
[0086] For example, in some embodiments of this application, the third step of obtaining the multimodal training data at least according to the correspondence includes: extracting multimodal features from each data block, wherein the multimodal features include: text semantic features, audio emotional features, video facial features, and physiological data statistical features; and associating and organizing the multimodal features according to preset modules of the consultation report to obtain the multimodal training data. For example, the preset modules include a basic situation description module, a core problem description module, an intervention process recording module, and an effect evaluation module.
[0087] The following describes a method for generating multimodal training data provided by some embodiments of this application, using psychological counseling as the target field. This method includes:
[0088] Step 1: Establishing and Pre-reviewing the Ethics and Compliance System
[0089] To address compliance risks associated with sensitive data, the following safeguard mechanisms are established:
[0090] Privacy protection measures: Some embodiments of this application adopt a three-level protection of "de-identification of the collection and synthesis end, encryption of transmission and control of storage permissions", stripping sensitive information such as names and ID numbers, and retaining only anonymous IDs; transmission is encrypted with AES-256, and storage is set with a role permission matrix, which only authorizes researchers and ethics committees to access.
[0091] Step 2: Multimodal Standardized Data Acquisition and Seed Data Construction
[0092] 1) Multimodal acquisition specifications:
[0093] Audio: Audio information was collected using a digital recorder and microphone. The background noise was set to ≤30 dB, the sampling rate to 44.1 kHz, the bit depth to 16 bit, and the audio signal output to WAV or FLAC format. The sound quality of the device was tested before the audio signal was collected.
[0094] Video: A 1080P high-definition camera (45° side angle) was used to capture video information at a frame rate of 30fps. The lighting was uniform and shadow-free, and the output was in MP4 / MOV format. Keyframes of emotional change were marked using the CV algorithm (JPG format).
[0095] Text: Medical history, conversation transcription, and progress records were collected using SOAP / BIRP templates and converted into standardized Markdown format, with a transcription accuracy of ≥98%.
[0096] Physiological data: Wearable devices were used to collect data such as heart rate (1 beat per second) and blood oxygen (1 beat per 30 seconds) from the consultants, and the collected data (including precise timestamps) were output in JSON format to obtain the time series of physiological data.
[0097] 2) Seed data construction: Interview psychological counseling experts, collect compliant and authentic dialogues and analysis reports, and organize them in a structured manner according to patient information, doctor information and counseling process as the basic template for data synthesis to ensure that the synthesized data fits the target scenarios such as clinical practice.
[0098] Step 3: Multimodal data synthesis to fill gaps in real data.
[0099] High-quality synthetic data is generated based on seed data through amplification and screening:
[0100] 1) Augment the collected real dialogue data to obtain synthesized dialogue.
[0101] For example, by calling the SOTA model API (DeepSeek-R1, GPT-4o, etc.), a synthetic dialogue with a structure similar to the collected real dialogue is generated. The data covers 12 core psychological disorders (anxiety, depression, etc.), generating 8.0B tokens of initial screening data. After deduplication, 5.0B tokens are retained.
[0102] 2) Refine the synthesized dialogue.
[0103] For example, three collaborative agents can be used to improve the quality of synthesized dialogues.
[0104] The first intelligent agent, Agent1, generates the consultation reasoning chain (CoT), the second intelligent agent, Agent2, refutes and optimizes it (correcting logical loopholes and terminological errors), and the third intelligent agent, Agent3, votes and decides, generating refined data 2.0B tokens (containing CoT information).
[0105] 3) Synthetic data standardization: Adapt the synthesized dialogue (or synthetic report) to the multimodal scenario, that is, supplement the virtual physiological data time series and video key frame descriptions that are adapted to the specific synthesized dialogue, and ensure that they are consistent with the format of the collected multimodal data.
[0106] Step 4: Multimodal Data Processing and Cross-Modal Alignment
[0107] 1) Data preprocessing:
[0108] Data collection: audio noise reduction, video image stabilization, text removal of special characters, and removal of outliers from physiological data (heart rate >180 or <40).
[0109] Synthetic data: Verify format standardization, correct content that does not conform to clinical scenarios, and ensure consistency with the collected data modality.
[0110] 2) Cross-modal timestamp alignment: A unified timeline (accurate to milliseconds) is established based on the consultation start time. The Dynamic Time Warping (DTW) algorithm is used to accurately bind audio segments, video keyframes, text dialogues, physiological data, and synthetic data (the specific processing process can be referred to the description above) to eliminate time deviations between modalities.
[0111] 3) Structured encapsulation: Divide into "chunks" in 5-minute units. Each chunk contains a chunkID and a time interval. Use JSON format to encapsulate multimodal subsets (audio file path + transcribed text, video keyframes + emotion tags, physiological data time series, text records, and synthetic data fragments) to achieve "one consultation - multi-chunk integration".
[0112] Step 5: Full-process quality control, establishing a two-tier mechanism of basic verification and professional evaluation:
[0113] 1) Basic validation: The rule engine validates the data format (e.g., audio is WAV, text is Markdown), completeness (no modal missing chunks), and timestamp continuity, and removes unqualified data.
[0114] 2) Professional assessment: Scoring (1-5 points) is conducted on four dimensions: accuracy of clinical terminology, matching degree of emotion labels, coherence of intervention logic, and consistency of modal information (no conflict between collected and synthesized data). Data with a score ≥4 points are selected. A binary classification quality discrimination model is trained to achieve efficient evaluation of the full dataset, and finally 5.0B tokens of valid data (500,000-1,000,000 samples) are retained.
[0115] Step Six: Data Adaptation and Report Generation Verification
[0116] 1) Data adaptation: Extract multimodal features (text semantics, audio emotion, video facial features, physiological data statistical features) from each chunk, associate features and content according to the report modules (basic information, core issues, intervention process, effect evaluation), and generate structured training data that the model can directly call.
[0117] 2) Effect verification: Based on Qwen-VL (visual), Qwen-Audio (audio) and Qwen2-72B (text), a multimodal report generation model was constructed. The output effect of this method was compared with that of traditional single data. It is expected that the completeness of diagnostic report information will be ≥92%, the accuracy of clinical terminology will be ≥97%, the accuracy of emotion assessment will be ≥93%, and the generation time of a single report will be shortened from 2 hours to 5 minutes.
[0118] Some embodiments of this application provide a method for model training, the method comprising: obtaining multimodal training data according to the above-described method for generating multimodal training data; and training a consultation report generation model based on the multimodal training data.
[0119] Please refer to Figure 5 , Figure 5 The apparatus for generating multimodal training data provided in the embodiments of this application is shown. It should be understood that this apparatus is similar to the one described above. Figure 2 Corresponding to the method embodiments, it is capable of executing the various steps involved in the above method embodiments. The specific functions of the device can be found in the description above; to avoid repetition, detailed descriptions are appropriately omitted here. The device includes at least one software function module that can be stored in a memory or embedded in the device's operating system in the form of software or firmware. The device for generating multimodal training data includes:
[0120] The synthetic dialogue acquisition module 301 is configured to acquire synthetic dialogues of plain text type.
[0121] The multimodal synthesis data acquisition module 302 is configured to determine a virtual physiological data time series and a virtual video target frame description adapted to the synthetic dialogue, thereby obtaining a multimodal synthesis dataset. The virtual physiological data time series is used to characterize the change feature sequence of physiological indicators matching the synthetic dialogue, and the virtual video target frame description is a textual description of visual information at at least one target moment based on the emotional state of the synthetic dialogue.
[0122] The multimodal real data acquisition module 303 is configured to acquire multimodal information of the real dialogue process to obtain a multimodal real dataset, wherein the multimodal real dataset includes: real dialogue text data, physiological data time series corresponding to the real dialogue process, and video target frame description.
[0123] The multimodal training data acquisition module 304 is configured to obtain multimodal training data based on the multimodal real dataset and the multimodal synthetic dataset.
[0124] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.
[0125] Some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, can implement the method for generating multimodal training data as described in the above examples.
[0126] Some embodiments of this application provide a computer program product, including computer program instructions, which, when read and executed by a processor, can implement the method for generating multimodal training data as described in the above embodiments.
[0127] like Figure 6 As shown, some embodiments of this application provide an electronic device 400, which includes a memory 410, a processor 420, and a computer program stored in the memory 410 and executable on the processor 420. When the processor reads and executes the program via a bus 430, it can implement the method for generating multimodal training data as described above.
[0128] Processor 420 can process digital signals and may include various computing architectures. For example, it may be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 420 may be a microprocessor.
[0129] Memory 410 can be used to store instructions executed by processor 420 or data related to the execution of instructions. These instructions and / or data may include code used to implement some or all of the functions of one or more modules described in the embodiments of this application. The processor 420 of the embodiments of this disclosure can be used to execute the instructions in memory 410 to implement… Figure 2 The method shown. Memory 410 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memory well known to those skilled in the art.
[0130] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0131] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.
[0132] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0133] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0134] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0135] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
Claims
1. A method for generating multimodal training data, characterized in that, The method includes: Obtain a text-based synthesized dialogue; A virtual physiological data time series and a virtual video target frame description adapted to the synthetic dialogue are determined to obtain a multimodal synthetic dataset, wherein the virtual physiological data time series is used to characterize the change feature sequence of physiological indicators matched with the synthetic dialogue, and the virtual video target frame description is a textual description of visual information at at least one target moment based on the emotional state of the synthetic dialogue. Multimodal information from real dialogue processes is collected to obtain a multimodal real dataset, wherein the multimodal real dataset includes: real dialogue text data, physiological data time series corresponding to the real dialogue process, and video target frame descriptions; Multimodal training data is obtained based on the real multimodal dataset and the synthetic multimodal dataset.
2. The method as described in claim 1, characterized in that, The process of obtaining multimodal training data based on the multimodal real dataset and the multimodal synthetic dataset includes: Align the multimodal data included in each dialogue in the multimodal real dataset or the multimodal synthetic dataset to obtain an aligned multimodal data stream; The aligned multimodal data stream is segmented and encapsulated according to a set duration to obtain the multimodal training data.
3. The method as described in claim 2, characterized in that, Both the real multimodal dataset and the synthetic multimodal dataset include: text modal data and non-text modal data; The process of aligning the multimodal data included in each dialogue within the real multimodal dataset or the synthetic multimodal dataset to obtain an aligned multimodal data stream includes: Using the time axis corresponding to the text modality as a reference time axis, the text modality data is converted into a time series to obtain a reference sequence; Extract the feature sequences from the non-text modal data to obtain at least one sequence to be compared; Each sequence to be aligned is aligned with the reference sequence to obtain the aligned multimodal data stream.
4. The method as described in claim 3, characterized in that, The non-text modal data includes: video modal data and audio modal data; The step of extracting the feature sequence of the non-text modality data to obtain at least one sequence to be compared includes: Features are extracted from each frame of the video modal data, and the extraction results are sorted by time to obtain a video feature sequence; The audio modal data is extracted for audio energy within a first duration, and the extraction results are sorted by time to obtain an audio feature sequence, wherein the first duration is a preset duration.
5. The method according to any one of claims 3-4, characterized in that, The step of aligning each sequence to be compared with the reference sequence to obtain the aligned multimodal data stream includes: Calculate the target distance path between each sequence to be compared and the reference sequence; Alignment is achieved by bending the time axis of the non-reference modality according to the target distance path, wherein the non-reference modality includes: the video modality corresponding to the video modality data, the physiological modality corresponding to the physiological data time series, and the audio modality corresponding to the audio modality data.
6. The method as described in claim 2, characterized in that, The step of segmenting and encapsulating the aligned multimodal data stream according to a set duration to obtain the multimodal training data includes: Determine the identifier and time interval information of the data blocks corresponding to each set duration; A correspondence is established between the multimodal data subsets corresponding to each time interval information and the corresponding data block identifiers. The multimodal data subsets are obtained by segmenting the aligned multimodal data stream according to the set duration. The multimodal data subsets include: audio data subsets, video data subsets, physiological data subsets and text data subsets. Each data subset corresponds to a set duration. The multimodal training data is obtained at least based on the aforementioned correspondence.
7. The method as described in claim 6, characterized in that, The audio data subset includes: audio data and the transcribed text of the audio data; the video data subset includes: video target frames and emotion tags corresponding to the video target frames; the physiological data subset includes physiological data time series; and the text data subset includes transcribed text of real consultation dialogues and / or text of synthesized dialogues.
8. The method according to any one of claims 6-7, characterized in that, The process of obtaining the multimodal training data based at least on the correspondence includes: Multimodal features are extracted from each data block, including: text semantic features, audio emotional features, video facial features, and physiological data statistical features; the multimodal features are associated and organized according to the preset modules of the consultation report to obtain the multimodal training data.
9. A method for training a model, characterized in that, The method includes: Multimodal training data is obtained according to any one of claims 1-8; The consultation report generation model is trained based on the multimodal training data.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the program, it can implement the method as described in any one of claims 1-9.