Personalized remote court hearing system

Abstract

The invention relates to the technical field of personalized assistance of remote court hearing, in particular to a personalized remote court hearing system. Through an emotion analysis technology integrating facial expressions, voices and physiological signals, the emotional state of a party is monitored in real time, and the remote court hearing process is optimized. The emotional state of a party is accurately obtained through multi-mode emotional analysis, the emotional intensity of the party is analyzed, the real-time emotional state of the party is accurately controlled by combining the influence of keywords on the emotional of the party, the court trial rhythm and the questioning mode are automatically adjusted, a timely emotional guiding strategy is provided, and the court trial efficiency is improved. And continuously optimizing an emotion analysis algorithm and intervention measures. According to the system and the method, a court hearing person can timely observe the emotion change of the party, so that the court hearing person can effectively respond to the emotion feedback of the party, the emotion of the party is guided within a controllable range, the court hearing rhythm is better controlled, the court hearing efficiency is improved, and court hearing disputes are quickly solved.

Description

Technical Field

[0001] The present invention relates to the field of personalized auxiliary technology for remote court hearings, and in particular to a personalized remote court hearing system. Background Art

[0002] In recent years, the cloud-based shared court has integrated 5G network technology and a secure video exchange platform. Through cross-network data transmission, the official court appearance areas of courts at all levels are connected with law firms, mediation rooms and portable devices of the parties on the Internet, realizing local, cross-network and cross-domain mixed court sessions, and realizing the standardization and scientificization of court work, which has brought practical convenience to judicial practice, but at the same time, it has also faced many problems. In traditional trials, court staff can intuitively feel the subtle changes in the body movements, facial expressions and tone of voice of the parties, and the parties can also intuitively see, hear and feel the details of the other party's trial. Skilled trial organizers often have strong trial control capabilities, can effectively guide the defense between the parties in the trial, and promptly stop the parties' improper verbal confrontation. The existing cloud-based trial technology can conduct general communication and interaction, but it is not intuitive for the parties' emotional feedback on the trial. For example, when the parties leave early, it is not possible to accurately judge whether it is a device network problem or a subjective problem of the parties. They need to communicate passively by phone and wait, which affects the rhythm of the trial and also brings troubles to the characterization of leaving early during the trial. In addition, the trial organizers are not always able to effectively respond to the emotional feedback of the parties, which can easily lead to hidden dangers and even malicious incidents that should not have happened. Therefore, it is necessary to use technical means to obtain the emotional feedback of the parties in a timely manner during the cloud trial, so as to better control the trial rhythm, improve the trial efficiency, and quickly resolve trial disputes. Summary of the invention

[0003] The technical problem solved by the present invention is to provide a personalized remote trial system to assist trial personnel in effectively responding to the emotional state of the parties, so as to better control the trial rhythm and improve the efficiency and quality of the trial.

[0004] The basic solution provided by the present invention: a personalized remote trial system, including a data acquisition module, a multimodal emotion analysis module, an emotion intensity calculation module, a keyword analysis module, a visualization module and a feedback strategy suggestion module;

[0005] The data collection module is used to collect audio data and video data of the parties;

[0006] The multimodal emotion analysis module is used to identify the emotions of the parties based on the audio data and video data and to fuse the emotional intensity of the parties under multiple modes to obtain the emotion intensity value ES. i ;

[0007] The emotion intensity calculation module is used to calculate the emotion intensity value ES of each emotion. i Calculate EI:

[0008] Assume that the emotional state vector ES = [ES1, ES2, ES3...ES n ], where ES i is the intensity value of the i-th emotion;

[0009]

[0010] In the formula, ω i is the weight of each emotion, ω i satisfy

[0011] The keyword analysis module is used to analyze the final emotional intensity FEI after the keywords in the conversation affect the emotions based on the audio data:

[0012] Let keyword set K = {K1, K2, K3...K m}, K j is the jth keyword in the set, then the keyword impact factor KIF is:

[0013]

[0014] In the formula, v j For each keyword K j The weight, f j Keyword K j The frequency of occurrence in the conversation; combine the keyword impact factor KIF and the emotional intensity E to get the final emotional intensity FEI:

[0015] FEI=EI×(1+KIF)

[0016] In the formula, (1+KIF) means that the keyword can enhance or weaken the emotion intensity E, and the effect depends on the positive or negative value of KIF. If KIF is positive, it means that the keyword enhances the emotion; if KIF is negative, it means that the keyword weakens the emotion;

[0017] The feedback strategy suggestion module is set with an out-of-control threshold OCT. When the final emotional intensity FEI of the party reaches the out-of-control threshold OCT, the feedback strategy suggestion module sends an emotional out-of-control danger warning to the judge through the visualization module, prompting the judge to regulate the emotions of the party.

[0018] Further, the multimodal emotion analysis module includes a facial expression analysis module, a speech analysis module and a data fusion module;

[0019] The facial emotion analysis module is used to identify the changes in the facial expressions of the parties and calculate the emotional intensity ES based on the video data through computer vision technology. exper，i :

[0020] ES exper，i =ω expr ×E i

[0021] In the formula, E i is the facial expression feature of the i-th emotion, ω expr is the weight of the expression feature;

[0022] The voice emotion analysis module is used to calculate the voice emotion intensity ES of the parties through acoustic analysis technology based on audio data. voice，i :

[0023] ES voice，i =ω voice ×V i

[0024] Where V i is the acoustic feature of the i-th emotion, ω voice is the weight of the acoustic feature;

[0025] The data fusion module is used to fuse data from different modes to obtain the emotion intensity value ES. i :

[0026] ES i =α×ES exper,i +β×ES voice,i

[0027] In the formula, α and β are the emotional intensity ES, respectively. exper，i and speech emotion intensity ES voice，i The weights of these two modes satisfy α+β=1.

[0028] Furthermore, the data module is also used to collect physiological data of the parties;

[0029] The multimodal emotion analysis module is also used to identify the emotions of the parties according to the audio data, video data and physiological data, and to fuse the emotional intensity of the parties under multiple modes to obtain an emotion intensity value ES. i ;

[0030] The multimodal emotion analysis module includes a facial expression analysis module, a speech analysis module, a physiological signal analysis module and a data fusion module. The facial emotion analysis module is used to identify the changes in the facial expressions of the parties and calculate the emotional intensity ES through computer vision technology based on the video data. exper,i :

[0031] ESexper,i =ω expr ×E i

[0032] In the formula, E i is the facial expression feature of the i-th emotion, ω expr is the weight of the expression feature;

[0033] The voice emotion analysis module is used to calculate the voice emotion intensity ES of the parties through acoustic analysis technology based on audio data. voice,i :

[0034] ES voice,i =ω voice ×V i

[0035] Where V i is the acoustic feature of the i-th emotion, ω voice is the weight of the acoustic feature;

[0036] The physiological signal analysis module is used to calculate the physiological emotion intensity ES of the parties based on physiological data physio，i :

[0037] ES physio,i =ω physio ×P i

[0038] Where P i is the acoustic feature of the i-th emotion, ω physio is the weight of the physiological characteristics;

[0039] The data fusion module is also used to fuse data of different modes to obtain an emotion intensity value ES i :

[0040] ES i =α×ES exper,i +β×ES voice,i +γ×ES physio,i

[0041] In the formula, α, β and γ are the emotional intensity ES respectively. exper,i , Voice emotion intensity ES voice，i and physiological emotional intensity ES physio，i The weights of these three modes satisfy α+β+γ=1.

[0042] Furthermore, the facial expression module is further provided with an expression feature monitoring module, which is used to extract facial feature points through a facial key point detection algorithm, and record the displacement change of the feature points in continuous frames based on the facial contour, calculate the displacement change and acceleration information of the feature points, take the feature point with the largest displacement change as the emotional focus, and evaluate the expression feature value E according to the acceleration information of the emotional focus. i :

[0043] E i =k×|a|+b

[0044] Where k and b are constants determined by the experimental data of the emotion, and a is the acceleration of the emotion focus.

[0045] Furthermore, each keyword K j The weight v j Calculated based on the sentiment color and text importance of keywords:

[0046] First, the keyword emotional color is divided into positive emotional keywords and negative emotional keywords. Suppose S j Keyword K j The sentiment score of K j is a positive emotion keyword, then S j ≥0; if K j is a negative emotion keyword, then S j <0;

[0047] Secondly, the text importance of each keyword is quantified by the term frequency-inverse document frequency (TF-IDF) method. j ;

[0048] Then the emotion score S j and text importance T j Combined to calculate the importance weight v of each keyword j :

[0049]

[0050] In the formula, m is the number of keywords.

[0051] Furthermore, it also includes an audio preprocessing module, which includes an audio normalization module, an emphasis module, a windowing module and a silence removal module;

[0052] The audio normalization module is used to adjust the amplitude of the audio signal and eliminate the amplitude difference between different audio signals;

[0053] The emphasis module is used to enhance the energy of the high frequency part through the pre-emphasis filter;

[0054] The windowing module is used to divide the continuous audio signal into short-term frames, and each frame usually overlaps a part to maintain continuity;

[0055] The silence removal module is used to identify and remove the silent parts in the audio to reduce the amount of calculation and avoid noise interference.

[0056] Furthermore, it also includes a video preprocessing module, which includes a de-occlusion module, an image normalization module and a face alignment module;

[0057] The deocclusion module is used to detect and remove objects that occlude facial expressions in video frames;

[0058] The image normalization module is used to adjust the size and color of the image to meet the model input requirements;

[0059] The face alignment module is used to locate the key areas of the face using the face key point detection algorithm and extract a number of feature points; the faces are aligned according to these feature points and all faces are adjusted to the same posture and angle to eliminate the impact of posture changes on emotion recognition.

[0060] Furthermore, it also includes a guidance module, which is used to generate corresponding guidance strategies according to the emotions of the parties and display them to the judges through the visualization module for prompting; the emotions include four states: calmness, anxiety, anger and extremism.

[0061] The principles and advantages of the present invention are as follows: the present invention monitors the emotional state of the parties in real time and optimizes the remote trial process by integrating emotion analysis technology of facial expressions, voice and physiological signals, accurately obtains the emotional state of the parties and analyzes the emotional intensity of the parties through multimodal emotion analysis, and then accurately controls the real-time emotional state of the parties in combination with the emotional impact of keywords on the parties, automatically adjusts the rhythm of the trial and the way of asking questions, provides timely emotion guidance strategies, and continuously optimizes emotion analysis algorithms and intervention measures. In summary, the present invention realizes real-time control of the emotions of the parties in remote trials, solves the technical problem that the current remote trial system lacks emotional management guidance for the parties, and helps the trial organizers to obtain emotional feedback from the parties in a timely manner through automated emotion analysis technology, so that the parties can conduct trials in a relatively stable mood, avoiding the psychological impact of excessive emotions on the parties and excessive departure behavior, while improving judicial efficiency and fairness, it also improves the efficiency of trials. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 The present invention is a logic block diagram of a personalized remote trial system embodiment. DETAILED DESCRIPTION

[0063] The following is further described in detail through specific implementation methods:

[0064] The specific implementation process is as follows:

[0065] Embodiment 1

[0066] This embodiment is a personalized remote trial system based on multimedia information analysis, which is suitable for trial terminals in the scenes of mid-level and grassroots courts, procuratorate appearance rooms, and detention centers / prison appearance rooms. It is equipped with a multimedia environment, and some scenes are equipped with bracelets to collect physiological data. Figure 1 As shown, it includes a data acquisition module, a preprocessing module, a multimodal emotion analysis module, an emotion intensity calculation module, a keyword analysis module, a visualization module, a feedback strategy suggestion module, and a guidance module. This solution integrates emotion analysis technology of facial expressions, voice, and physiological signals to monitor the emotional state of the parties in real time, optimize the remote trial process, accurately obtain the emotional state of the parties through multimodal emotion analysis and analyze the emotional intensity of the parties, and then combine the emotional impact of keywords on the parties to accurately control the real-time emotional state of the parties, automatically adjust the trial rhythm and questioning methods, provide timely emotion guidance strategies, and continuously optimize emotion analysis algorithms and intervention measures. It realizes real-time control of the emotions of the parties in remote trials, solves the technical problem that the current remote trial system lacks emotional management guidance for the parties, and allows the parties to conduct trials in a relatively stable mood.

[0067] In this solution, the data collection module is the basic part of the personalized remote trial system, responsible for collecting various data of the parties during the trial. In order to fully and accurately capture the emotional state of the parties, this module needs to collect multiple types of data, including but not limited to audio data, video data and physiological data.

[0068] Specifically, the data acquisition module includes a video acquisition module, an audio acquisition module, and a heart rate acquisition module. The video acquisition module uses a high-resolution camera, supports automatic focus and light adjustment, and ensures that the facial expressions of the parties can be clearly captured under different lighting conditions. The facial expressions and body language of the parties are captured and recorded in real time by a high-resolution camera. The audio acquisition module uses a high-quality microphone with noise suppression function, which can clearly record the sound in the presence of background noise. The voice of the parties is captured by a high-quality microphone, including acoustic characteristics such as intonation, speaking speed and volume. The heart rate acquisition module uses non-invasive heart rate monitoring equipment such as smart bracelets to ensure that the parties are comfortable and do not affect the trial process. In this embodiment, the scenes of wearing bracelets mainly include witness protection, procuratorate appearance rooms, detention centers, and prison rooms and other official appearance places. The heart rate changes of the parties are monitored by non-invasive heart rate monitoring equipment as an important physiological indicator for assessing their emotional state. The data acquisition module collects information about the emotional state of the parties from multiple dimensions, providing a solid data foundation for subsequent emotional analysis and emotional guidance. Through this multimodal data collection method, the system can more comprehensively understand the emotional changes of the parties, thereby implementing effective emotion management strategies in remote trials.

[0069] The preprocessing module includes an audio preprocessing module, a video preprocessing module and a heart rate data preprocessing module, and the audio preprocessing module includes an audio normalization module, an emphasis module, a windowing module and a silent removal module. Specifically, the audio normalization module uses a linear or nonlinear scaling method to map the amplitude of the audio signal to a fixed range (such as between 0 and 1), and adjusts the amplitude of the audio signal so that it is within a unified range to eliminate the amplitude difference between different audio signals. The emphasis module is used to enhance the energy of the high-frequency part through a pre-emphasis filter to compensate for the high-frequency information that may be lost during the recording process. The windowing module uses window functions such as Hamming windows and Hanning windows to smooth frame edges, reduce spectrum leakage, and divide continuous audio signals into short-term frames, and each frame usually overlaps a part to maintain continuity. The silent removal module is used to identify and remove silent parts (such as silent segments) in the audio to reduce the amount of calculation and avoid noise interference. In the present embodiment, it is achieved by setting an energy threshold. When the energy of the frame is lower than the threshold, the frame is considered to be a silent frame and is removed. The video preprocessing module includes a de-occlusion module, an image normalization module, and a face alignment module. The de-occlusion module is used to check whether there are objects (such as hands, hair, etc.) that block facial expressions in the video frame. If there are, try to remove or adjust the angle to ensure that the facial expressions are clearly visible. This can be achieved through image processing techniques (such as morphological operations, face detection, etc.). The image normalization module is used to adjust the size and color of the image to meet the model input requirements. In this embodiment, the image is scaled to a fixed size (such as 224x224 pixels) and the color space is converted to RGB format. This helps to reduce the amount of calculation and improve the generalization ability of the model. The face alignment module uses a face key point detection algorithm (such as the 68-point model in the Dlib library) to locate the key areas of the face and extract 68 feature points. Then align the faces according to these feature points, that is, adjust all faces to the same posture and angle to eliminate the impact of posture changes on emotion recognition. This is achieved in this solution by affine transformation or deep learning methods. The heart rate data preprocessing module is used to clean, denoise and enhance the collected original ECG signals to facilitate subsequent feature extraction and analysis.

[0070] The multimodal emotion analysis module is used to identify the emotions of the parties based on audio data, video data and physiological data, and to fuse the emotional intensity of the parties under the three modes to obtain the emotional intensity value ES. i .

[0071] Specifically, the multimodal emotion analysis module includes a facial expression analysis module, a speech analysis module, a physiological signal analysis module, and a data fusion module;

[0072] The facial emotion analysis module is used to identify the changes in the facial expressions of the parties and calculate the intensity of the facial expressions through computer vision technology based on the video data. The recognition of facial expressions is carried out in the following five steps:

[0073] Feature vector construction: Using these 68 feature points, construct the feature vector of facial expression, which includes geometric features such as distance, angle, area, etc. between feature points. The feature points in this embodiment include but are not limited to the position information of key parts such as eyebrows, eyes, nose, mouth, etc.

[0074] Feature normalization: Since the face sizes and postures in different images may be different, the feature vectors are normalized in this embodiment to eliminate the impact of these differences on expression recognition.

[0075] Feature selection and dimensionality reduction: PCA (principal component analysis) is used to remove redundant features in the feature vector and reduce the data dimension while retaining the most important information.

[0076] Training the expression classification model: Select a data set containing a large number of annotated facial expressions. In this embodiment, the FER2013 data set is used. The data set includes images of multiple basic emotion categories (such as calm, anxious, angry, and aggressive). The data set is divided into a training set and a test set. After preprocessing the images in the data set, 68 feature points are extracted from each image using a facial key point detection algorithm, and a feature vector is constructed based on these feature points. The support vector machine (SVM) is trained using the training set data. During the training process, the model will learn how to identify different facial expression categories from the input feature vector. The model performance is evaluated through the test set and the SVM parameters are optimized using random search to obtain the trained expression classification model.

[0077] Expression recognition: The processed feature vector is input into the expression classification model to obtain the recognition results of expression categories such as calm, anxious, angry and aggressive.

[0078] Expression emotion intensity ES exper，i The calculation of is as follows:

[0079] ES exper,i =ω expr ×E i

[0080] In the formula, E i is the facial expression feature of the i-th emotion (such as smile), ω expr is the weight of the expression feature;

[0081] The facial expression module also has an expression feature monitoring module, which is used to extract facial feature points through the face key point detection algorithm, and record the displacement changes of the feature points in continuous frames based on the facial contour, calculate the displacement change and acceleration information of the feature points, take the feature point with the largest displacement change as the emotional focus, and evaluate the expression feature value E according to the acceleration information of the emotional focus. i:

[0082] E i =k×|a|+b

[0083] Where k and b are constants determined by the experimental data of the emotion, and a is the acceleration of the emotion focus.

[0084] The voice emotion analysis module is used to calculate the voice emotion intensity ES of the parties through acoustic analysis technology based on audio data. voice，i :

[0085] ES voice，i =ω voice ×V i

[0086] Where V i is the acoustic feature of the ith emotion (such as volume or tone), ω voice is the weight of the acoustic feature;

[0087] The physiological emotion analysis module is used to calculate the physiological emotion intensity ES of the parties based on physiological data physio，i :

[0088] ES physio，i =ω physio ×P i

[0089] Where P i is the physiological characteristics of the ith emotion (such as heart rate changes), ω physio is the weight of the physiological characteristics;

[0090] The data fusion module is used to fuse data from different modes to obtain the emotion intensity value ES. i :

[0091] ES i =α×ES exper，i +β×ES voice，i +γ×ES physio，i

[0092] In the formula, α, β and γ are the emotional intensity ES respectively. exper，i , Voice emotion intensity ES voice，i and physiological emotional intensity ES physio，i The weights of these three modes satisfy α+β+γ=1.

[0093] The emotion intensity calculation module is used to calculate the emotion intensity value ES of each emotion. i Calculate EI:

[0094] Assume that the emotional state vector ES = [ES1, ES2, ES3...ES n ], where ESi is the intensity value of the i-th emotion

[0095]

[0096] In the formula, ω i is the weight of each emotion, ω i satisfy

[0097] The keyword analysis module is used to analyze the final emotional intensity FEI after the keywords in the conversation affect the emotions based on the audio data:

[0098] Let keyword set K = {K1, K2, K3...K m}, K j is the jth keyword in the set, then the keyword impact factor KIF is:

[0099]

[0100] In the formula, v j For each keyword K j The weight, f j Keyword K j The frequency of occurrence in the conversation; where v j Calculated based on the keyword's sentiment (positive or negative) and text importance:

[0101] First, the keyword emotional color is divided into positive emotional keywords and negative emotional keywords. Suppose S j Keyword K j The sentiment score of K j is a positive emotion keyword, then S j ≥0; if K j is a negative emotion keyword, then S j <0;

[0102] Secondly, the text importance of each keyword is quantified by the term frequency-inverse document frequency (TF-IDF) method. j ;

[0103] Then the emotion score S j and text importance T j Combined to calculate the importance weight v of each keyword j :

[0104]

[0105] In the formula, m is the number of keywords.

[0106] Finally, the keyword influencing factor KIF and the emotional intensity E are combined to obtain the final emotional intensity FEI:

[0107] FEI=EI×(1+KIF)

[0108] In the formula, (1+KIF) means that the keyword can enhance or weaken the emotion intensity E, and the effect depends on the positive or negative value of KIF. If KIF is positive, it means that the keyword enhances the emotion; if KIF is negative, it means that the keyword weakens the emotion.

[0109] The feedback strategy suggestion module is set with an out-of-control threshold OCT. When the final emotional intensity FEI of the party reaches the out-of-control threshold OCT, the feedback strategy suggestion module sends an emotional out-of-control danger warning to the judge through the visualization module, prompting the judge to regulate the emotions of the party. The specific regulation strategy is prompted through the guidance module.

[0110] Specifically, the guidance module is used to generate corresponding guidance strategies according to the emotions of the parties and display them to the judges through the visualization module for prompting; the emotions include four states: calm, anxious, angry and aggressive;

[0111] When the client is in a calm state, the guidance strategies generated by the guidance strategy module include maintaining the status quo and encouraging expression:

[0112] Maintain the status quo: Maintain the current trial rhythm and method and continue as planned.

[0113] Encourage expression: Give the parties the opportunity to fully express their views and feelings and ensure that their voices are heard.

[0114] When the client is in an anxious state, the guidance strategies generated by the guidance strategy module include providing comfort, simplifying the problem and giving time:

[0115] Provide comfort: Use a gentle tone of voice and body language to convey support and understanding, making the person feel at ease.

[0116] Simplify the problem: Break down complex problems into simple parts to help the parties understand and answer them step by step.

[0117] Give time: Allow the client enough time to think and answer questions, and avoid rushing them.

[0118] When the parties are in an angry state, the guidance strategies generated by the guidance strategy module include suspending the trial, diverting attention, and emphasizing the rules:

[0119] Suspension of the trial: If anger affects the order of the trial, the court can be temporarily adjourned to give the parties time to calm down.

[0120] Divert attention: guide the parties to focus on the facts and evidence of the case rather than personal emotions.

[0121] Emphasize the rules: Remind the parties to abide by the trial rules and maintain respect and courtesy.

[0122] When the parties are in an over-excited state, the guidance strategies generated by the guidance strategy module include adjusting the topic and providing a break:

[0123] Adjust the topic: Change the topic at the right time to avoid touching on sensitive points too much and reduce the emotional burden of the parties involved.

[0124] Provide breaks: If needed, give the person a short break to help them recover emotionally.

[0125] In summary, this solution monitors the emotional state of the parties in real time and optimizes the remote trial process by integrating emotion analysis technology of facial expressions, voice and physiological signals. It accurately obtains the emotional state of the parties and analyzes the emotional intensity of the parties through multimodal emotion analysis, and then accurately controls the real-time emotional state of the parties in combination with the emotional impact of keywords on the parties, automatically adjusts the trial rhythm and questioning methods, provides timely emotion guidance strategies, and continuously optimizes emotion analysis algorithms and intervention measures. In summary, the present invention realizes real-time control of the emotions of the parties in remote trials, solves the technical problem that the current remote trial system lacks emotional management guidance for the parties, allows the parties to conduct trials in a relatively stable mood, avoids the psychological impact of excessive emotions on the parties, improves judicial efficiency and fairness, and enhances the satisfaction of the parties.

[0126] Embodiment 2

[0127] The only difference between the second embodiment and the first embodiment is that the second embodiment is applicable to the court trial terminals for personal use such as the Internet-based law firms, mediation rooms, and portable devices of the parties, rather than the court trial terminals in the official court trial area that carry physiological data collection devices.

[0128] Specifically, in this embodiment, a personalized remote trial system, whose data acquisition module only includes a video acquisition module and an audio acquisition module, collects the audio data and video data of the parties through the video acquisition module and the audio acquisition module. The video acquisition module uses a high-resolution camera and supports automatic focus and light adjustment to ensure that the facial expressions of the parties can be clearly captured under different lighting conditions. The facial expressions and body language of the parties are captured and recorded in real time through a high-resolution camera. The audio acquisition module uses a high-quality microphone with noise suppression function, which can record sound clearly in the presence of background noise. The voice of the party is captured through a high-quality microphone, including acoustic characteristics such as intonation, speaking speed and volume.

[0129] The preprocessing module only includes an audio preprocessing module and a video preprocessing module. The audio preprocessing module includes an audio normalization module, an emphasis module, a windowing module and a silent removal module. Specifically, the audio normalization module uses a linear or nonlinear scaling method to map the amplitude of the audio signal to a fixed range (such as between 0 and 1), and adjusts the amplitude of the audio signal so that it is within a unified range to eliminate the amplitude difference between different audio signals. The emphasis module is used to enhance the energy of the high-frequency part through a pre-emphasis filter to compensate for the high-frequency information that may be lost during the recording process. The windowing module uses window functions such as Hamming windows and Hanning windows to smooth the frame edges, reduce spectrum leakage, and divide the continuous audio signal into short-term frames, each frame usually overlaps a part to maintain continuity. The silent removal module is used to identify and remove the silent part (such as a silent segment) in the audio to reduce the amount of calculation and avoid noise interference. In this embodiment, it is achieved by setting an energy threshold. When the energy of the frame is lower than the threshold, the frame is considered to be a silent frame and is removed. The video preprocessing module includes a de-occlusion module, an image normalization module and a face alignment module. The de-occlusion module is used to check whether there are objects (such as hands, hair, etc.) that block facial expressions in the video frame. If there are, try to remove or adjust the angle to ensure that the facial expressions are clearly visible. This can be achieved through image processing techniques (such as morphological operations, face detection, etc.). The image normalization module is used to adjust the size and color of the image to meet the model input requirements. In this embodiment, the image is scaled to a fixed size (such as 224x224 pixels) and the color space is converted to RGB format. This helps to reduce the amount of calculation and improve the generalization ability of the model. The face alignment module uses a face key point detection algorithm (such as the 68-point model in the Dlib library) to locate the key areas of the face and extract 68 feature points. Then align the faces according to these feature points, that is, adjust all faces to the same posture and angle to eliminate the impact of posture changes on emotion recognition. This is achieved in this solution by affine transformation or deep learning methods.

[0130] The multimodal emotion analysis module is only used to identify the emotions of the parties based on the audio data and video data respectively, and to fuse the emotional intensity of the parties under the two modes to obtain the emotional intensity value ES. i .

[0131] Specifically, the multimodal emotion analysis module includes a facial expression analysis module, a speech analysis module, and a data fusion module;

[0132] The facial emotion analysis module is used to identify the changes in the facial expressions of the parties and calculate the intensity of the facial expressions through computer vision technology based on the video data. The recognition of facial expressions is carried out in the following five steps:

[0133] Feature vector construction: Using these 68 feature points, construct the feature vector of facial expression, which includes geometric features such as distance, angle, area, etc. between feature points. The feature points in this embodiment include but are not limited to the position information of key parts such as eyebrows, eyes, nose, mouth, etc.

[0134] Feature normalization: Since the face sizes and postures in different images may be different, the feature vectors are normalized in this embodiment to eliminate the impact of these differences on expression recognition.

[0135] Feature selection and dimensionality reduction: PCA (principal component analysis) is used to remove redundant features in the feature vector and reduce the data dimension while retaining the most important information.

[0136] Training the expression classification model: Select a data set containing a large number of annotated facial expressions. In this embodiment, the FER2013 data set is used. The data set includes images of multiple basic emotion categories (such as calm, anxious, angry, and aggressive). The data set is divided into a training set and a test set. After preprocessing the images in the data set, 68 feature points are extracted from each image using a facial key point detection algorithm, and a feature vector is constructed based on these feature points. The support vector machine (SVM) is trained using the training set data. During the training process, the model will learn how to identify different facial expression categories from the input feature vector. The model performance is evaluated through the test set and the SVM parameters are optimized using random search to obtain the trained expression classification model.

[0137] Expression recognition: The processed feature vector is input into the expression classification model, and the recognition results of expression categories include calm, anxious, angry and aggressive.

[0138] Expression emotion intensity ES exper，i The calculation of is as follows:

[0139] ES exper，i =ω expr ×E i

[0140] In the formula, E i is the facial expression feature of the i-th emotion (such as smile), ω expr is the weight of the expression feature;

[0141] The facial expression module also has an expression feature monitoring module, which is used to extract facial feature points through the face key point detection algorithm, and record the displacement changes of the feature points in continuous frames based on the facial contour, calculate the displacement change and acceleration information of the feature points, take the feature point with the largest displacement change as the emotional focus, and evaluate the expression feature value E according to the acceleration information of the emotional focus. i :

[0142] E i =k×|a|+b

[0143] Where k and b are constants determined by the experimental data of the emotion, and a is the acceleration of the emotion focus.

[0144] The voice emotion analysis module is used to calculate the voice emotion intensity ES of the parties through acoustic analysis technology based on audio data. voice，i :

[0145] ES voice，i =ω voice ×V i

[0146] Where V i is the acoustic feature of the ith emotion (such as volume or tone), ω voice is the weight of the acoustic feature;

[0147] The data fusion module is used to fuse data from different modes to obtain the emotion intensity value ES. i :

[0148] ES i =α×ES exper，i +β×ES voice，i

[0149] In the formula, α and β are the emotional intensity ES, respectively. exper，i and speech emotion intensity ES voice，i The weights of these two modes satisfy α+β=1.

[0150] Therefore, although analyzing the emotional intensity of the parties only through facial expressions and voice emotions is not as accurate as the analysis of professional trial terminals equipped with physiological data collection equipment, it can meet the needs of remote trials by Internet-based law firms, mediation rooms, and the parties' portable devices and other self-use trial terminals. It can make remote trials more convenient for the parties, enable trials to be conducted quickly, and greatly improve trial efficiency.

[0151] The above are only embodiments of the present invention. Common knowledge such as the known specific structures and characteristics in the scheme is not described in detail here. Ordinary technicians in the relevant field know all the common technical knowledge in the technical field to which the invention belongs before the application date or priority date, can obtain all the existing technologies in the field, and have the ability to apply conventional experimental means before that date. Ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the scope of protection of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A personalized remote trial system, characterized by: It includes data collection module, multimodal emotion analysis module, emotion intensity calculation module, keyword analysis module, visualization module and feedback strategy suggestion module; The data collection module is used to collect audio data and video data of the parties; The multimodal emotion analysis module is used to identify the emotions of the parties based on the audio data and video data and to fuse the emotional intensity of the parties under multiple modes to obtain the emotion intensity value ES. i ; The emotion intensity calculation module is used to calculate the emotion intensity value ES of each emotion. i Calculate EI: Assume that the emotional state vector ES = [ES1, ES2, ES3…ES n ], where ES i is the intensity value of the i-th emotion; In the formula, ω i is the weight of each emotion, ω i satisfy The keyword analysis module is used to analyze the final emotional intensity FEI after the keywords in the conversation affect the emotions based on the audio data: Let the keyword set K = {K1, K2, K3…K m },K j is the jth keyword in the set, then the keyword impact factor KIF is: In the formula, v j For each keyword K j The weight, f j Keyword K j The frequency of occurrence in the conversation; combine the keyword impact factor KIF and the emotional intensity EI to get the final emotional intensity FEI: FEI=EI×(1+KIF) In the formula, (1+KIF) indicates that keywords can enhance or weaken the intensity of emotions. The effect of EI depends on the positive or negative value of KIF. If KIF is positive, it means that keywords enhance emotions; if KIF is negative, it means that keywords weaken emotions. The feedback strategy suggestion module is set with an out-of-control threshold OCT. When the final emotional intensity FEI of the party reaches the out-of-control threshold OCT, the feedback strategy suggestion module sends an emotional out-of-control danger warning to the judge through the visualization module, prompting the judge to regulate the emotions of the party.

2. A personalized remote trial system according to claim 1, characterized in that: The multimodal emotion analysis module includes a facial expression analysis module, a speech analysis module and a data fusion module; The facial emotion analysis module is used to identify the changes in the facial expressions of the parties and calculate the emotional intensity ES based on the video data through computer vision technology. exper,i : IT exper,i =ω expr ×E i In the formula, E i is the facial expression feature of the i-th emotion, ω expr is the weight of the expression feature; The voice emotion analysis module is used to calculate the voice emotion intensity ES of the parties through acoustic analysis technology based on audio data. voice,i : IT voice,i =ω voice ×V i Where V i is the acoustic feature of the i-th emotion, ω voice is the weight of the acoustic feature; The data fusion module is used to fuse data from different modes to obtain the emotion intensity value ES. i : IS i =α×ES exper,i +β×ES voice,i In the formula, α and β are the emotional intensity ES, respectively. exper,i and speech emotion intensity ES voice,i The weights of these two modes satisfy α+β=1.

3. A personalized remote trial system according to claim 1, characterized in that: The data module is also used to collect physiological data of the parties; The multimodal emotion analysis module is also used to identify the emotions of the parties according to the audio data, video data and physiological data, and to fuse the emotional intensity of the parties under multiple modes to obtain an emotion intensity value ES. i ; The multimodal emotion analysis module includes a facial expression analysis module, a speech analysis module, a physiological signal analysis module and a data fusion module. The facial emotion analysis module is used to identify the changes in the facial expressions of the parties and calculate the emotional intensity ES through computer vision technology based on the video data. exper,i : IT exper,i =ω expr ×E i In the formula, E i is the facial expression feature of the i-th emotion, ω expr is the weight of the expression feature; The voice emotion analysis module is used to calculate the voice emotion intensity ES of the parties through acoustic analysis technology based on audio data. voice,i : IT voice,i =ω voice ×V i Where V i is the acoustic feature of the i-th emotion, ω voice is the weight of the acoustic feature; The physiological signal analysis module is used to calculate the physiological emotion intensity ES of the parties based on physiological data physio,i : IT physii,i =ω physio ×P i Where P i9 is the acoustic feature of the i-th emotion, ω physio is the weight of the physiological characteristics; The data fusion module is also used to fuse data of different modes to obtain an emotion intensity value ES i : IS i =α×ES exper,i +β×ES voice,i +γ×ES physio,i In the formula, α, β and γ are the emotional intensity ES respectively. exper,i , Voice emotion intensity ES voice,i and physiological emotional intensity ES physio,i The weights of these three modes satisfy α+β+γ=1.

4. A personalized remote trial system according to claim 2 or 3, characterized in that: The facial expression module is further provided with an expression feature monitoring module, which is used to extract facial feature points through a facial key point detection algorithm, and record the displacement changes of the feature points in continuous frames based on the facial contour, calculate the displacement change and acceleration information of the feature points, take the feature point with the largest displacement change as the emotional focus, and evaluate the expression feature value E according to the acceleration information of the emotional focus. ii : E i =k×|a|+b Where k and b are constants determined by the experimental data of the emotion, and a is the acceleration of the emotion focus.

5. A personalized remote trial system according to claim 4, characterized in that: Each keyword K j The weight v j Calculated based on the sentiment color and text importance of keywords: First, the keyword emotional color is divided into positive emotional keywords and negative emotional keywords. Suppose S j Keyword K j The sentiment score of K j is a positive emotion keyword, then S j ≥0; if K j is a negative emotion keyword, then S j <0; Secondly, the text importance of each keyword is quantified by the term frequency-inverse document frequency (TF-IDF) method. j ; Then the emotion score S j and text importance T j Combined to calculate the importance weight v of each keyword j : In the formula, m is the number of keywords.

6. A personalized remote trial system according to claim 5, characterized in that: It also includes an audio preprocessing module, which includes an audio normalization module, an emphasis module, a windowing module and a silence removal module; The audio normalization module is used to adjust the amplitude of the audio signal and eliminate the amplitude difference between different audio signals; The emphasis module is used to enhance the energy of the high frequency part through the pre-emphasis filter; The windowing module is used to divide the continuous audio signal into short-term frames, and each frame usually overlaps a part to maintain continuity; The silence removal module is used to identify and remove the silent parts in the audio to reduce the amount of calculation and avoid noise interference.

7. A personalized remote trial system according to claim 6, characterized in that: It also includes a video preprocessing module, which includes a de-occlusion module, an image normalization module and a face alignment module; The deocclusion module is used to detect and remove objects that occlude facial expressions in video frames; The image normalization module is used to adjust the size and color of the image to meet the model input requirements; The face alignment module is used to locate the key areas of the face using the face key point detection algorithm and extract a number of feature points; the faces are aligned according to these feature points and all faces are adjusted to the same posture and angle to eliminate the impact of posture changes on emotion recognition.

8. A personalized remote trial system according to claim 7, characterized in that: It also includes a guidance module, which is used to generate corresponding guidance strategies according to the emotions of the parties and display them to the judges through a visualization module for prompting; the emotions include four states: calmness, anxiety, anger and extremism.

Images