Digital structured interview processing method based on multimodal fusion and large language model
By employing a digital structured interview method that combines multimodal fusion and a large language model, we have addressed the issues of insufficient resources and emotional instability in traditional interviews. This approach enables personalized and dynamically adaptive interview strategies, thereby improving interview quality and security.
Patent Information
- Application Number
- CN202410881068.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-03
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-03
AI Technical Summary
Traditional structured interviews suffer from a shortage of qualified counselors, leading to unstable emotions among interviewees, which affects the quality of the interview and poses a threat to their mental health. Furthermore, large language models are insufficient to replace professional counselors in providing services.
This paper adopts a digital structured interview method that combines multimodal fusion and large language model analysis. By integrating multimodal data collection, feature extraction and large language model analysis, it generates personalized and dynamically adaptive interview strategies, including multimodal data feature extraction, singular value decomposition, text preprocessing and large language model fine-tuning, and generates new questioning paradigms.
It enables in-depth, personalized, and dynamically adaptive interview support, improves the accuracy of identifying the emotions and personality traits of interviewees, and enhances the authenticity and security of the interviews.
Smart Images

Figure CN118430816B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a digital structured interview processing method based on multimodal fusion and a large language model. Background Art
[0002] The statements in this section merely provide background art related to the present invention and do not necessarily constitute prior art.
[0003] Structured interview is an interview method widely used in psychological assessment, clinical diagnosis and market research. It collects information from the interviewee through a series of preset questions and standardized processes to ensure the efficiency of the interview and the comparability of information. Structured interviews can provide reliable and consistent data, which helps researchers or psychological counselors make accurate assessments and diagnoses. However, there are currently many challenges in traditional structured interviews. First, there is a shortage of qualified psychological counselors, making it difficult to meet a wide range of diverse needs; second, the interview process sometimes faces the risk of emotional instability of the interviewee, which not only affects the quality of the interview, but also poses a potential threat to the interviewee's mental health; in addition, long-term psychological interviews may also have an adverse effect on the counselor's own psychology, such as peer fatigue or emotional exhaustion.
[0004] The big language model is a major breakthrough in the field of artificial intelligence in recent years. It uses large-scale text data for training and can understand and generate natural language. Through complex algorithms and a huge knowledge base, the big language model can perform advanced tasks such as language understanding, sentiment analysis, and logical reasoning, showing impressive performance. These capabilities make the big language model a powerful tool in many application areas.
[0005] However, the large language model itself cannot directly obtain the emotional state of the interviewee, and it is difficult to fine-tune it to the level of a professional psychological counselor, which makes the large language model itself unable to directly replace the services of psychological counselors. Summary of the invention
[0006] In order to address the shortcomings of the prior art, the present invention provides a digital structured interview processing method based on multimodal fusion and a large language model. By integrating multimodal data acquisition, feature extraction and large language model analysis, an in-depth, personalized and dynamically adaptive interview strategy is proposed, thereby achieving more effective and safer support for psychological assessment.
[0007] In order to achieve the above object, the present invention adopts the following technical solution:
[0008] In a first aspect, the present invention provides a digital structured interview processing method based on multimodal fusion and a large language model.
[0009] A digital structured interview processing method based on multimodal fusion and large language model includes the following processes:
[0010] Under the preset questioning paradigm, obtain multimodal data from the interviewees;
[0011] Perform feature extraction on each modality of the multimodal data, convert the extracted features into a low-rank feature matrix, concatenate the low-rank feature matrices into a high-dimensional feature matrix, perform singular value decomposition on the high-dimensional feature matrix, and obtain a fused feature matrix;
[0012] Convert the interviewee's answer audio into text and pre-process the converted text;
[0013] The obtained fusion feature matrix is spatially embedded with the preprocessed text description, and prompt words are generated;
[0014] Based on the prompt words and the large language model fine-tuned with the interview corpus, a new questioning paradigm is generated. Questions are asked to the interviewee based on the new questioning paradigm and new prompt words are generated. This cycle is repeated until the end of the interview.
[0015] As a further limitation of the first aspect of the present invention, the multimodal data includes: two or more of: body movements, facial expressions, facial blood flow, eye movements, heart rate and brain waves.
[0016] As a further limitation of the first aspect of the present invention, feature extraction is performed on each modal data in the multimodal data, the extracted features are converted into a low-rank feature matrix, each low-rank feature matrix is spliced into a high-dimensional feature matrix, and the high-dimensional feature matrix is subjected to singular value decomposition to obtain a fused feature matrix, including:
[0017] Perform feature calculation on multimodal data to form multiple feature matrices on a time scale;
[0018] Each feature matrix is reduced in dimension through singular value decomposition to obtain a low-rank feature matrix of different features changing over time under the stimulation of the preset questioning paradigm;
[0019] The low-rank feature matrices after dimensionality reduction are concatenated into a high-dimensional feature matrix, and then the high-dimensional feature matrix is subjected to singular value decomposition to obtain a fused feature matrix, which is used as the state representation tensor.
[0020] As a further limitation of the first aspect of the present invention, converting the answer audio of the interviewee into text and preprocessing the converted text includes:
[0021] Useful features are extracted from the answer audio using Mel-frequency cepstral coefficients, the speech units in the features are identified through the acoustic model, the language structure and lexical constraints in the speech units are understood through the language model, the text output is determined based on the language structure and lexical constraints using a decoding algorithm, the output text is denoised, semantically segmented, normalized and formatted to obtain the preprocessed text.
[0022] As a further limitation of the first aspect of the present invention, the obtained fusion feature matrix and the preprocessed text description are spatially embedded, and prompt words are generated, including:
[0023] The fused feature matrix is used as the state description tensor of the interviewee, and the state description tensor is concatenated with the preprocessed text description to form the prompt word.
[0024] As a further limitation of the first aspect of the present invention, the state description tensor of the interviewee and the preprocessed text description are sent to a contrastive learning model to obtain prompt words; wherein the training of the contrastive learning model includes:
[0025] The preprocessed text description is encoded using the text transformer model, and the state description tensor is encoded using the Multimodal transformer model. The cosine similarity operation is performed on the encoded outputs of the text transformer and the Multimodal transformer. Through back propagation, the cosine similarity between the text description and the representation tensor corresponding to the text description is maximized, and the cosine similarity between the text description and the representation tensor not corresponding to the text description is minimized to obtain the trained contrastive learning model.
[0026] As a further limitation of the first aspect of the present invention, a new questioning paradigm is generated based on the prompt words and the large language model fine-tuned by the interview corpus, including:
[0027] According to the psychological counseling interview map, we obtain the actual psychological interview corpus, and use parameter fine-tuning to fine-tune the large language model through the interview corpus;
[0028] The prompt words are input into the fine-tuned large language model to generate a further new questioning paradigm. The new questioning paradigm is used as the preset questioning paradigm to ask questions and generate new prompt words. This cycle is repeated until the entire interview is completed.
[0029] In a second aspect, the present invention provides a digital structured interview processing system based on multimodal fusion and a large language model.
[0030] A digital structured interview processing system based on multimodal fusion and large language model, comprising:
[0031] The data acquisition module is configured to: acquire multimodal data of the interviewee under a preset questioning paradigm;
[0032] The feature fusion module is configured to: extract features from each modal data in the multimodal data, convert the extracted features into a low-rank feature matrix, concatenate the low-rank feature matrices into a high-dimensional feature matrix, perform singular value decomposition on the high-dimensional feature matrix, and obtain a fused feature matrix;
[0033] The speech processing module is configured to: convert the answer audio of the interviewee into text and pre-process the converted text;
[0034] The prompt word generation module is configured to: spatially embed the obtained fusion feature matrix and the preprocessed text description, and generate prompt words;
[0035] The questioning paradigm updating module is configured to generate a new questioning paradigm based on the prompt words and the large language model fine-tuned by the interview corpus, ask questions to the interviewee based on the new questioning paradigm and generate new prompt words, and repeat the cycle until the end of the interview.
[0036] In a third aspect, the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the digital structured interview processing method based on multimodal fusion and a large language model as described in the first aspect of the present invention.
[0037] In a fourth aspect, the present invention provides an electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the digital structured interview processing method based on multimodal fusion and a large language model as described in the first aspect of the present invention are implemented.
[0038] Compared with the prior art, the present invention has the following beneficial effects:
[0039] 1. The present invention innovatively proposes a digital structured interview processing method based on multimodal fusion and large language model. By integrating multimodal data acquisition, feature extraction and large language model analysis, it proposes an interview strategy that can be in-depth, personalized and dynamically adapted, thereby achieving more effective and safer support for psychological assessment.
[0040] 2. The present invention uses a multimodal fusion representation method to represent the relevant physiological characteristics of the interviewee and the interviewee information in a low-rank manner, and converts them into a language that can be understood by the large language model. Under the guidance of the interview graph, the large language model can continuously generate new personalized interview paradigms, effectively improving the recognition accuracy of the interviewee's emotions and personality traits, and improving the authenticity of the interview.
[0041] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0043] Figure 1 A flowchart of a digital structured interview processing method based on multimodal fusion and a large language model provided in Example 1 of the present invention;
[0044] Figure 2 A flow chart of state representation tensor calculation provided in Example 1 of the present invention;
[0045] Figure 3 This is a schematic diagram of generating prompt words provided in Example 1 of the present invention. DETAILED DESCRIPTION
[0046] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0047] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0048] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.
[0049] Embodiment 1:
[0050] like Figure 1 As shown, Embodiment 1 of the present invention provides a digital structured interview processing method based on multimodal fusion and a large language model, including the following processes:
[0051] S1: Obtain multimodal data from the interviewee under the preset questioning paradigm;
[0052] S2: Extract features from each modality of the multimodal data, convert the extracted features into a low-rank feature matrix, concatenate the low-rank feature matrices into a high-dimensional feature matrix, perform singular value decomposition on the high-dimensional feature matrix, and obtain a fused feature matrix;
[0053] S3: convert the interviewee's answer audio into text and pre-process the converted text;
[0054] S4: spatially embed the obtained fusion feature matrix and the preprocessed text description, and generate prompt words;
[0055] S5: Generate a new questioning paradigm based on the prompt words and the large language model fine-tuned by the interview corpus, ask questions to the interviewee based on the new questioning paradigm and generate new prompt words, and repeat this process until the end of the interview.
[0056] In step S1 of this implementation, specifically, it includes:
[0057] Multiple devices are used to obtain multimodal data under the questioning paradigm, including but not limited to using video cameras to capture the interviewee's facial expressions and body movements, using infrared camera technology to record facial temperature distribution, using eye trackers to measure the interviewee's gaze tracking and pupil response, using heart rate watches to continuously monitor the interviewee's heart rate variability, and using EEG belts to provide real-time data on brain electrical activity; at the same time, a microphone is used to record the audio expectations of the interviewee's answers.
[0058] In step S2 of this implementation, feature extraction is performed on the collected behavioral modalities and physiological modalities of the interviewees and fused into a unified tensor space, specifically, including:
[0059] S2.1: Feature extraction. The specific implementation is determined by the type of physiological modality. For example, the motion feature matrix of limb movements, the texture feature matrix of facial expressions, and the apnea feature matrix of facial photoplethysmogram can be obtained through video data. The blood flow distribution feature matrix of the interviewee can be obtained through an infrared camera. The line of sight feature matrix and the return feature matrix can be obtained through an eye tracker. Finally, N feature matrices can be obtained.
[0060] S2.2: Feature dimensionality reduction, using the singular value decomposition method to obtain the low-rank matrix of each feature matrix, while retaining the features and reducing the matrix dimension. The formula is as follows;
[0061] (1);
[0062] Among them, A is the original feature matrix, Σ is the eigenvalue matrix, and U and V are unitary matrices.
[0063] After singular value decomposition, the feature matrix achieves dimensionality reduction. Then, the N low-rank feature matrices after dimensionality reduction are concatenated into a high-dimensional feature matrix. The high-dimensional feature matrix is then subjected to singular value decomposition to obtain a fused feature matrix, i.e., the state representation tensor.
[0064] More specifically, Figure 2As shown, multimodal data may include data such as limb movements, facial expressions, eye movements, and breathing. The vital sign extraction module extracts features such as movement volume, limb contraction index, texture features, statistical features, line of sight features, back gaze features, respiratory rate, and apnea. The fusion module fuses each low-rank matrix (such as low-rank matrix 1, low-rank matrix 2, low-rank matrix 3, and low-rank matrix 4) into a high-order low-rank matrix, and obtains the state representation tensor after decomposition.
[0065] In step S3 of this implementation, the collected speech signal is converted into text to obtain a pre-processed text description, which specifically includes:
[0066] S3.1: Use Mel-frequency cepstral coefficients to convert speech signals into a series of feature vectors to represent the spectral properties of speech;
[0067] S3.2: Use a recurrent neural network to identify the mapping relationship from Mel-spectrogram features to speech units, and obtain the speech units in the audio;
[0068] S3.3: Use N-gram networks to capture linguistic structure and lexical constraints in speech units;
[0069] S3.4: Use the Viterbi algorithm to combine the output of the acoustic model and the knowledge of the language model to generate the final text output. Specifically, it searches for the most likely text description output given the speech signal.
[0070] In step S4 of this implementation, the state representation tensor and text description of the interviewee are sent to the contrastive learning model to obtain prompt words, specifically including:
[0071] S4.1: Pre-training of the contrastive learning network, taking the state representation tensor and the text description as data pairs for model training, where the text description is encoded using the text transformer model, and the state representation tensor is encoded using the Multimodal transformer model. The cosine similarity operation is performed on the encoded outputs of the text transformer and the Multimodal transformer. Through back propagation, the cosine similarity between the state representation tensor and the text description is maximized, and the cosine similarity between the state representation tensor and the text description that do not correspond is minimized.
[0072] S4.2: Use of contrastive learning networks, e.g. Figure 3As shown in FIG. 1 , the actual state representation tensor and the text description are sent to the trained contrastive learning network to obtain the prompt word. More specifically, when the subject (i.e., the interviewee) is in the state description, the answer is the subject's (i.e., the interviewee's) answer. Taking the answer "I'm ready" as an example, it is input into the text encoder to generate the encoding T1, T2, T3...T of the text description. N ; The state representation tensor is input into the tensor encoder for encoding R1, R1·T1, R1·T2, R1·T3, …, R1·T N , and finally generated the prompt word: the subject (i.e. the interviewee) was in a relaxed state and answered "I am ready".
[0073] In step S5 of this implementation, the prompt words are input into the large language model fine-tuned by the interview graph to generate further questioning paradigms, more specifically, including:
[0074] S5.1: Fine-tuning of the large language model. According to the psychological counseling interview map, we obtain the actual psychological interview corpus, and fine-tune the large language model through the interview corpus by fine-tuning some parameters;
[0075] S5.2: Input the prompt words into the large language model, which will automatically generate further question patterns, return and repeat the process of S1-S5 until the entire interview is completed (for example, the set interview time is reached or there is a reason for interrupting the interview).
[0076] Embodiment 2:
[0077] Embodiment 2 of the present invention provides a digital structured interview processing system based on multimodal fusion and a large language model, including:
[0078] The data acquisition module is configured to: acquire multimodal data of the interviewee under a preset questioning paradigm. The detailed working process is referred to the process of S1 in Example 1, which will not be described here;
[0079] The feature fusion module is configured to: extract features from each modal data in the multimodal data, convert the extracted features into a low-rank feature matrix, concatenate the low-rank feature matrices into a high-dimensional feature matrix, perform singular value decomposition on the high-dimensional feature matrix, and obtain a fused feature matrix. The detailed working process refers to the process of S2 in Example 1, which will not be repeated here;
[0080] The speech processing module is configured to: convert the answer audio of the interviewee into text and pre-process the converted text. The detailed working process is shown in the process of S3 in Example 1, which will not be repeated here;
[0081] The prompt word generation module is configured to: spatially embed the obtained fusion feature matrix and the preprocessed text description, and generate prompt words. The detailed working process is referred to the process of S4 in Example 1, which will not be repeated here;
[0082] The questioning paradigm updating module is configured to generate a new questioning paradigm based on the prompt words and the large language model fine-tuned by the interview corpus, ask questions to the interviewee based on the new questioning paradigm and generate new prompt words, and repeat the process until the end of the interview. For the detailed working process, please refer to the process S5 in Example 1, which will not be repeated here.
[0083] Embodiment 3:
[0084] Embodiment 3 of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the digital structured interview processing method based on multimodal fusion and a large language model as described in Embodiment 1 of the present invention.
[0085] Embodiment 4:
[0086] Embodiment 4 of the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the digital structured interview processing method based on multimodal fusion and a large language model as described in Embodiment 1 of the present invention are implemented.
[0087] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A digital structured interview processing method based on multimodal fusion and large language model, characterized in that: The process includes: Under the preset questioning paradigm, obtain multimodal data from the interviewees; Feature extraction is performed on each modal data in the multimodal data, and the extracted features are converted into a low-rank feature matrix. Each low-rank feature matrix is concatenated into a high-dimensional feature matrix, and the high-dimensional feature matrix is subjected to singular value decomposition to obtain a fused feature matrix, including: Perform feature calculation on multimodal data to form multiple feature matrices on a time scale; Each feature matrix is reduced in dimension through singular value decomposition to obtain a low-rank feature matrix of different features changing over time under the stimulation of the preset questioning paradigm; The low-rank feature matrices after dimensionality reduction are concatenated into a high-dimensional feature matrix, and then the high-dimensional feature matrix is subjected to singular value decomposition to obtain a fused feature matrix; Convert the interviewee's answer audio into text and pre-process the converted text; The obtained fusion feature matrix and the preprocessed text description are spatially embedded, and prompt words are generated, including: using the fusion feature matrix as a state representation tensor of the interviewee, and splicing the state representation tensor with the preprocessed text description to form prompt words; Specifically, the state representation tensor of the interviewee and the preprocessed text description are sent to the contrastive learning model to obtain the prompt words; wherein, the training of the contrastive learning model includes: encoding the preprocessed text description using the texttransformer model, encoding the state representation tensor using the Multimodaltransformer model, performing cosine similarity operation on the outputs encoded by the texttransformer and the Multimodal transformer, maximizing the cosine similarity between the state representation tensor and the text description, and minimizing the cosine similarity between the state representation tensor and the text description that do not correspond, through back propagation, to obtain the trained contrastive learning model; According to the prompt words and the large language model fine-tuned by the interview corpus, a new questioning paradigm is generated. According to the new questioning paradigm, questions are asked to the interviewee and new prompt words are generated. This cycle repeats until the end of the interview. More specifically, when the interviewee is in the state description, the answer is the interviewee's answer. The text is input into the text encoder to generate the encoding T1, T2, T3...TN of the text description; the state representation tensor is input into the tensor encoder for encoding R1, and after splicing, R1·T1, R1·T2, R1·T3,..., R1·TN are obtained, and finally the prompt words are generated; Generate a new questioning paradigm based on the prompt words and the large language model fine-tuned by the interview corpus, including: according to the psychological counseling interview map, obtain the actual psychological interview corpus, and fine-tune the large language model through the interview corpus by means of parameter fine-tuning; input the prompt words into the fine-tuned large language model to generate a further new questioning paradigm, use the new questioning paradigm as the preset questioning paradigm to ask questions and generate new prompt words, and repeat the cycle until all interviews are completed; The multimodal data includes: two or more of body movements, facial expressions, facial blood flow, eye movements, heart rate and brain waves; Useful features are extracted from the answer audio using Mel-frequency cepstral coefficients, the speech units in the features are identified through the acoustic model, the language structure and lexical constraints in the speech units are understood through the language model, the text output is determined based on the language structure and lexical constraints using a decoding algorithm, the output text is denoised, semantically segmented, normalized and formatted to obtain the preprocessed text.
2. A digital structured interview processing system based on multimodal fusion and large language model, characterized in that: include: The data acquisition module is configured to: acquire multimodal data of the interviewee under a preset questioning paradigm; The feature fusion module is configured to: extract features from each modal data in the multimodal data, convert the extracted features into a low-rank feature matrix, concatenate the low-rank feature matrices into a high-dimensional feature matrix, perform singular value decomposition on the high-dimensional feature matrix, and obtain a fused feature matrix, specifically, including: Perform feature calculation on multimodal data to form multiple feature matrices on a time scale; Each feature matrix is reduced in dimension through singular value decomposition to obtain a low-rank feature matrix of different features changing over time under the stimulation of the preset questioning paradigm; The low-rank feature matrices after dimensionality reduction are concatenated into a high-dimensional feature matrix, and then the high-dimensional feature matrix is subjected to singular value decomposition to obtain a fused feature matrix; The speech processing module is configured to: convert the answer audio of the interviewee into text and pre-process the converted text; The prompt word generation module is configured to: spatially embed the obtained fusion feature matrix and the preprocessed text description, and generate prompt words, including: taking the fusion feature matrix as the state representation tensor of the interviewee, splicing the state representation tensor and the preprocessed text description to form prompt words; Specifically, the state representation tensor of the interviewee and the preprocessed text description are sent to the contrastive learning model to obtain the prompt words; wherein, the training of the contrastive learning model includes: encoding the preprocessed text description using the texttransformer model, encoding the state representation tensor using the Multimodal transformer model, performing cosine similarity operation on the outputs encoded by the texttransformer and the Multimodal transformer, maximizing the cosine similarity between the state representation tensor and the text description, and minimizing the cosine similarity between the state representation tensor and the text description that do not correspond, through back propagation, to obtain the trained contrastive learning model; The questioning paradigm update module is configured to: generate a new questioning paradigm based on the prompt words and the large language model fine-tuned by the interview corpus, ask questions to the interviewee based on the new questioning paradigm and generate new prompt words, repeating the process until the end of the interview. More specifically, when the interviewee is in the state description, the answer is the interviewee's answer, and the text is input into the text encoder to generate the encoding T1, T2, T3...TN of the text description; the state representation tensor is input into the tensor encoder for encoding R1, and after splicing, R1·T1, R1·T2, R1·T3,..., R1·TN are obtained, and finally the prompt words are generated; Generate a new questioning paradigm based on the prompt words and the large language model fine-tuned by the interview corpus, including: according to the psychological counseling interview map, obtain the actual psychological interview corpus, and fine-tune the large language model through the interview corpus by means of parameter fine-tuning; input the prompt words into the fine-tuned large language model to generate a further new questioning paradigm, use the new questioning paradigm as the preset questioning paradigm to ask questions and generate new prompt words, and repeat the cycle until all interviews are completed; The multimodal data includes: two or more of body movements, facial expressions, facial blood flow, eye movements, heart rate and brain waves; Useful features are extracted from the answer audio using Mel-frequency cepstral coefficients, the speech units in the features are identified through the acoustic model, the language structure and lexical constraints in the speech units are understood through the language model, the text output is determined based on the language structure and lexical constraints using a decoding algorithm, the output text is denoised, semantically segmented, normalized and formatted to obtain the preprocessed text.
3. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the digital structured interview processing method based on multimodal fusion and large language model as described in claim 1 are implemented.
4. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the digital structured interview processing method based on multimodal fusion and large language model as described in claim 1 are implemented.
