Method and system for multi-modal comprehensive evaluation of actor skills based on large language model

Abstract

The invention relates to the technical field of emotional expression evaluation methods, and discloses a method and system for multi-modal comprehensive evaluation of actor skills based on a large language model, and the method comprises the following steps: S1, data collection and preprocessing; s2, multi-modal feature extraction is carried out; s3, performing data alignment and script analysis; and S4, reasoning the large language model and outputting a result, inputting the large language model subjected to fine adjustment of the performance case, and generating quantitative scores, total scores and improvement suggestions on a time axis of each dimension. According to the method, a technology of combining computer vision, voice signal processing and natural language processing is adopted, through deep fusion of limb, micro-expression and voice multi-modal features, subtle differences of actor emotion expression are comprehensively captured, and a scene-based evaluation benchmark is generated in combination with deep analysis of a script text; and quantitative and traceable evaluation results are output by means of semantic reasoning capability of a large language model.

Description

Technical Field

[0001] This invention relates to the field of methods for evaluating emotional expression, and more specifically, to a method and system for evaluating actors' acting skills based on a large language model and multimodal comprehensive evaluation. Background Technology

[0002] In the performing arts, such as film and theater, the accuracy and emotional impact of an actor's expression are core indicators for evaluating acting skills. Traditional evaluation methods mainly rely on the subjective judgment of professionals such as directors and screenwriters, which has the following significant limitations:

[0003] The evaluation criteria are subjective and lack uniformity: human evaluation relies on personal experience and artistic perception, and there may be significant differences in the evaluation of the same performance segment, making it difficult to reach a consensus on the evaluation results.

[0004] Single-modal analysis is one-sided and ignores multi-dimensional correlations: Existing technologies often focus on a single modality, such as evaluating emotions solely through micro-expressions or voice, failing to capture the multi-dimensional synergy of emotional expression, resulting in discrepancies between evaluation results and actual emotions.

[0005] Detached from the script context, evaluations are out of touch with the needs: Actors' emotional expressions need to fit the character's background and scene logic, but traditional techniques fail to establish a connection with the script, leading to evaluation results that deviate from the performance requirements and draw incorrect conclusions.

[0006] The output results are vague and lack operability: traditional methods often produce vague descriptions such as "lack of emotional depth" or "insufficient expressiveness," without specifying concrete directions for improvement. Actors cannot know which characteristic at which point in time needs to be adjusted, making the evaluation difficult to implement.

[0007] In recent years, multimodal emotion computing technology has made progress: in the field of micro-expression recognition, tools such as OpenFace have achieved automated detection of AUs, but have not been associated with the emotional requirements of the script; in body movement analysis, tools such as MediaPipe can extract joint coordinates, but lack mapping rules with emotions; in speech emotion analysis, the combination of Whisper and Liborosa has achieved simultaneous extraction of textual and acoustic features, but has not been combined with body and micro-expression analysis. Meanwhile, the advantages of large language models (such as LLaMA and GPT series) in semantic understanding and reasoning have made multimodal fusion possible, but existing research has not optimized for the correlation of emotions and multimodal features in performance scenarios.

[0008] To address the aforementioned issues, this invention integrates multimodal feature extraction technology with the reasoning capabilities of large language models to construct a complete process encompassing feature extraction, script association, fusion evaluation, and feedback implementation, thus filling a gap in existing technologies. Summary of the Invention

[0009] To solve the above technical problems, the application provides an innovative system architecture and methodology, which integrates multi-modal feature extraction technology, including body movement, micro-expression, voice features, with the deep semantic understanding ability of large language model text, to build an efficient, accurate and scalable actor emotion expression evaluation platform, providing strong technical support for actor training, film and television creation, drama staging and other fields.

[0010] To achieve the above purpose, the technical scheme of the application is as follows:

[0011] The method for evaluating actor performance based on a large language model multi-modal comprehensive evaluation includes the following steps:

[0012] S1, data acquisition and preprocessing, collecting video data of actor performance, audio data separated from the video data and corresponding script text data, performing frame format conversion processing on the video data, and performing noise removal and sampling rate adjustment processing on the audio data;

[0013] S2, multi-modal feature extraction, after preprocessing, respectively extracting body movement features, micro-expression features and voice features, the body movement features include key node coordinates and action amplitude, speed and persistence features, the micro-expression features include facial action unit intensity, emotion type and emotion intensity, and the voice features include time-stamped dialogue text and pitch, speech rate, volume and pause features;

[0014] S3, data alignment and script analysis, performing time axis distribution and necessary standard extraction on the script text data to obtain script requirements and scoring reference table with time interval; matching the body movement features, micro-expression features and voice features with the time interval of the script requirements according to the time stamp to generate a three-dimensional comparison table;

[0015] S4, large language model reasoning and result output, integrating the three-dimensional comparison table and the scoring reference table into a structured Prompt, inputting the large language model fine-tuned by the performance case to generate quantitative scores in each dimension, total score and time axis improvement suggestions.

[0016] As another preferred embodiment of the application, the data acquisition and preprocessing in S1 specifically includes: S11, collecting performance video data with resolution ≥1920×1080 and frame rate ≥30fps, reading video frames through the cv2.VideoCapture function of OpenCV, and converting BGR format frames to RGB format frames using the cv2.cvtColor function; S12, separating audio data with a sampling rate of 44.1kHz from the video data, removing background noise using noise spectrum subtraction, normalizing audio amplitude, resampling to 16kHz single channel, and obtaining PCM format audio array.

[0017] As another preferred scheme of the present application, the extracting limb action features in S2 specifically comprises: S211: loading a MediaPipePose model, processing the RGB format video frame obtained in step S11, detecting and outputting a 2D normalized coordinate array of 33 whole body key nodes, a visibility confidence, and a 3D relative coordinate trajectory of 16 consecutive frames; S212: inputting the 2D normalized coordinate array, the 3D relative coordinate trajectory, and the time stamp into a FeaturizableMachine model trained based on a Kinetics400 dataset, calculating action amplitude, motion speed, and action persistence features, and outputting an 8-dimensional action feature vector and an action category label and confidence;

[0018] The MediaPipePose model is used for limb action feature extraction, adopts a two-step detector, tracker machine learning pipeline, and the process is as follows:

[0019] The detector is based on a light convolutional neural network (CNN) and locates a human body, a posture region of interest (ROI) in an RGB video frame, and outputs a human body bounding box;

[0020] The tracker takes the ROI cropped frame as input, and outputs a two-dimensional coordinate of each key point through a regression method and visibility;

[0021] For the 33 whole body key nodes, a 2D normalized coordinate array (dimension: and a visibility confidence array (dimension: ) are outputted;

[0022] The 3D relative coordinate trajectory combines multiple frame information, geometric constraint relationship, and calibrated camera parameters, converts the 2D coordinate into a 3D relative coordinate through a triangulation method, and the output dimension of 16 consecutive frames is wherein, 16=consecutive frame number, 33=key node number, and 3=X / Y / Z three-dimensional coordinate.

[0023] As another preferred scheme of the present application, the calculating action amplitude, motion speed, and action persistence features in S212 specifically comprises: action amplitude calculation: calculating a right arm swing angle based on the coordinate vector point product of the shoulder node 12, the elbow node 14, and the wrist node 16, calculating a finger bending degree based on the ratio of the actual distance of the knuckle nodes 19 and 18 to the straight state distance, and calculating a body forward angle based on the included angle of the line connecting the shoulder midpoint and the hip midpoint and the vertical direction; motion speed calculation: calculating the 3D Euclidean distance of the wrist node 16 adjacent frames, dividing by the frame interval time (1 / 30 seconds) to obtain the motion speed; action persistence calculation: counting the duration of the feature action, and converting into actual time.

[0024] As another preferred scheme of the present application, the step S2 of extracting micro-expression features specifically comprises: S221: using a MediaPipeFaceDetection model to perform face detection on the RGB format video frame obtained in step S11, obtaining a face bounding box and cropping a face region, and scaling to 224x224 pixels; S222: loading an OpenFace toolkit to process the cropped face region, detecting 68 face key points, and outputting the activation intensity (0-5 levels) of 30+ facial action units (AUs); S223: performing emotion association on the AU activation intensity based on a preset AU-emotion mapping rule, outputting a probability vector of 7 basic emotions and a dominant emotion label; and S224: accumulating the core AUs and auxiliary AUs of the dominant emotion according to weights, wherein the core AU weight is 0.6 and the auxiliary AU weight is 0.4, and the emotion duration is combined for calibration, and an emotion intensity score of 0-10 points is output.

[0025] As another preferred scheme of the present application, the step S2 of extracting speech features specifically comprises: S231: loading an OpenAIWhisper model, performing speech-to-text processing on the PCM format audio array obtained in step S12, and outputting a list of dialogue texts with start and end time stamps; and S232: using a librosa library to extract acoustic features from the audio array, including calculating the mean of the fundamental frequency (f0), the number of sound units per second (speech rate), the root mean square energy (volume), and the number of pauses with a silence duration of >0.5 seconds, and outputting acoustic feature data with time stamps.

[0026] As another preferred scheme of the present application, the script analysis in step S3 specifically comprises: S311: loading a trained LLaMA model, inputting the script text data, assigning a time interval to a scene based on a pre-trained "plot-duration mapping rule", and generating a script requirement with a time axis; and S312: extracting quantized standards corresponding to body movements, micro-expressions, and speech from the scene description of the script text data, determining the scoring dimensions and weights of "emotion fit degree, action specification, and speech adaptation degree" (emotion fit degree 40%, action specification 30%, and speech adaptation degree 30%), and generating a scoring benchmark table.

[0027] As another preferred scheme of the present application, the data alignment in step S3 specifically comprises: matching the time stamps of the body movement features, micro-expression features, and speech features obtained in step S2 with the script requirement time intervals obtained in step S311 with a time precision of 100 ms, binding the actual feature data at the corresponding time points with the script requirements, and generating a three-dimensional comparison table containing time stamps, actual features, and script requirements, wherein the actual feature data includes fist angle, AU intensity, and fundamental frequency value.

[0028] As another preferred embodiment of the present application, the large language model reasoning and result output in S4 specifically comprises: S41: integrating the three-dimensional correspondence table and the scoring benchmark table into a structured Prompt according to the "time point + actual performance + script requirements + scoring rules" template; S42: loading the LLaMA model fine-tuned by 500+ actor performance cases, inputting the structured Prompt, comparing the actual features with the should-be standards, calculating the quantitative scores of each scoring dimension 0-10 points, and converting the total score according to the weight; S43: outputting the scoring results in JSON format and the evaluation report in text format.

[0029] The system for evaluating actor performance based on large language model multi-modal comprehensive evaluation comprises:

[0030] The body recognition module is used for extracting actor body action features, and comprises a key node detection unit and a feature quantization unit; the key node detection unit adopts a MediaPipePose model (model_complexity=2) to process an RGB video frame, and outputs 2D coordinates and time sequence 3D trajectories of 33 whole body key nodes; the feature quantization unit adopts a FeaturizableMachine model to calculate action amplitude, motion speed and action persistence, and outputs a body action feature vector;

[0031] The micro-expression recognition module is used for extracting actor micro-expression features, and comprises an AU detection unit, an emotion association unit and an intensity calibration unit; the AU detection unit adopts an OpenFace toolkit to detect facial AU intensity; the emotion association unit determines a dominant emotion based on an AU-emotion mapping rule trained based on a BP4D dataset; and the intensity calibration unit outputs an emotion intensity score through AU weighted accumulation and duration calibration;

[0032] The language recognition module is used for extracting actor speech features, and comprises an audio preprocessing unit, a speech-to-text unit and an acoustic feature extraction unit; the audio preprocessing unit performs noise removal and volume normalization; the speech-to-text unit adopts a Whisper model to generate a script with a timestamp; and the acoustic feature extraction unit adopts a librosa library to extract a fundamental frequency, a speech rate, a volume and a number of pauses;

[0033] The large language fusion evaluation module is used for realizing multi-modal feature and script fusion evaluation, and comprises a script benchmark analysis unit, a data alignment unit, a Prompt construction unit and a large language model reasoning unit; the script benchmark analysis unit assigns a scene time axis, extracts should-be standards and scoring weights; the data alignment unit generates a three-dimensional correspondence table with a precision of 100 ms; the Prompt construction unit generates a structured instruction; and the large language model reasoning unit loads the LLaMA model fine-tuned by 500+ performance cases, and outputs quantitative scores and improvement suggestions.

[0034] The beneficial technical effects of the present application are:

[0035] The present application can effectively improve the efficiency, accuracy and operability of actor emotional expression evaluation. The present application adopts the technology of combining computer vision, speech signal processing and natural language processing, comprehensively captures the subtle differences of actor emotional expression through deep fusion of multi-modal features such as body, micro-expression and speech, generates scene-based evaluation criteria combined with deep analysis of script text, and outputs quantified and traceable evaluation results with the help of semantic reasoning ability of large language model.

[0036] The present application can help actors accurately locate the deficiencies in their performance and provide targeted improvement direction for training; at the same time, it provides scientific evaluation basis for directors, screenwriters and other creative personnel, assists in selecting the optimal performance segment, and optimizes the emotional expression effect of film and television, drama works.

[0037] The actor emotional expression evaluation system based on multi-modal fusion has the characteristics of standardization and standardization, can dynamically adjust the evaluation criteria according to different performance types and role settings, and the output of multi-dimensional quantitative scoring and time axis suggestions can unify the evaluation standard, reduce the subjectivity and cost of manual evaluation, and promote the transformation of performance evaluation from experience-driven to data-driven. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The scoring system framework for multi-modal comprehensive evaluation of actor performance based on large language model;

[0039] Figure 2 The scoring system flowchart for multi-modal comprehensive evaluation of actor performance based on large language model. DETAILED DESCRIPTION

[0040] In order to more clearly understand the technical means of the present application and can be implemented according to the content of the specification, the specific embodiments of the present application are further described in detail below, the following examples are used to illustrate the present application, but not to limit the scope of the present application.

[0041] In combination Figure 1 - Figure 2 The present application provides the following examples:

[0042] The method for multi-modal comprehensive evaluation of actor performance based on large language model comprises the following steps:

[0043] S1, data acquisition and preprocessing, collecting video data of actor performance, audio data separated from the video data and corresponding script text data, performing frame format conversion processing on the video data, and performing noise removal and sampling rate adjustment processing on the audio data;

[0044] S2, multi-modal feature extraction, after preprocessing, respectively extract limb action features, micro-expression features and speech features, the limb action features include key node coordinates and action amplitude, speed, persistence features, the micro-expression features include facial action unit intensity, emotion type and emotion intensity, the speech features include timestamped dialogue text and pitch, speech rate, volume, pause features;

[0045] S3, data alignment and script analysis, assign time axis and extract necessary standards to the script text data to obtain script requirements and scoring benchmark table with time interval; match the limb action features, micro-expression features and speech features with the time interval of the script requirements according to the timestamp to generate a three-dimensional comparison table;

[0046] S4, large language model reasoning and result output, integrate the three-dimensional comparison table and the scoring benchmark table into a structured Prompt, input the large language model fine-tuned by the performance case to generate quantitative scores in each dimension, total score and improvement suggestions on the time axis.

[0047] Further, the data collection and preprocessing in S1 specifically includes: S11, collect performance video data with resolution ≥1920×1080 and frame rate ≥30fps, read video frames through the cv2.VideoCapture function of OpenCV, and convert BGR format frames to RGB format frames using the cv2.cvtColor function; S12, separate audio data with a sampling rate of 44.1kHz from the video data, remove background noise using noise spectrum subtraction, normalize the audio amplitude, resample to 16kHz single channel, and obtain a PCM format audio array.

[0048] Further, the extraction of limb action features in S2 specifically includes: S211: load the MediaPipePose model, process the RGB format video frames obtained in step S11, detect and output 2D normalized coordinate arrays of 33 whole body key nodes, visibility confidence and 3D relative coordinate trajectories of 16 consecutive frames; S212: input the 2D normalized coordinate array, 3D relative coordinate trajectory and timestamp into the FeaturizableMachine model trained based on the Kinetics400 dataset, the FeaturizableMachine model is a pre-trained model for action feature quantization, calculate the action amplitude, motion speed and action persistence features, and output an 8-dimensional action feature vector and action category label and confidence.

[0049] The FeaturizableMachine model is used for limb action feature extraction, which adopts a two-step machine learning pipeline of detector and tracker, and the process is as follows:

[0050] Detector: Based on a lightweight convolutional neural network (CNN), it locates the human body and the region of interest (ROI) in RGB video frames, outputs the human body bounding box, and determines the presence and approximate location of the human body;

[0051] Tracker: Taking ROI cropped frames as input, it outputs the two-dimensional coordinates of each keypoint through a regression method. And visibility (0-1 confidence level, representing the reliability of the detection).

[0052] For 33 key nodes across the entire body (head, shoulder, elbow, wrist, hip, knee, ankle, etc.), output a 2D normalized coordinate array (dimension: and visibility confidence array (dimension: );

[0053] 3D Relative Coordinate Trajectory: Combining multi-frame information, geometric constraints (such as the limits of human joint movement), and calibrated camera parameters (assuming known camera intrinsic and extrinsic parameters), 2D coordinates are converted to 3D relative coordinates using triangulation. The output dimension for 16 consecutive frames is... (16 = number of consecutive frames, 33 = number of key nodes, 3 = X / Y / Z 3D coordinates).

[0054] Furthermore, the calculation of motion amplitude, motion speed, and motion duration features described in S212 specifically includes: motion amplitude calculation: calculating the right arm swing angle based on the dot product of the coordinate vectors of shoulder node 12, elbow node 14, and wrist node 16; calculating the finger flexion based on the ratio of the actual distance between finger nodes 19 and 18 to the distance in the straightened state; and calculating the body forward lean angle based on the angle between the line connecting the midpoint of the shoulder and the midpoint of the hip and the vertical direction; motion speed calculation: calculating the 3D Euclidean distance between adjacent frames of wrist node 16 and dividing it by the frame interval time (1 / 30 second) to obtain the motion speed; and motion duration calculation: counting the number of frames of the characteristic motion and converting it into actual time.

[0055] The FeaturizableMachine model is used for quantifying limb movement features. It calculates the amplitude, speed, and duration of movements based on the coordinates of key nodes. The core logic is as follows:

[0056] (1) Angle of right arm punch

[0057] Let the coordinate vector of shoulder node 12 be... The coordinate vector of elbow node 14 is The coordinate vector of wrist node 16 is = First, calculate the vector from shoulder to elbow. and .

[0058] Punch angle The calculation formula is: ;

[0059] in The dot product of two vectors is calculated as follows:

[0060] Let be the magnitudes of the two vectors, calculated as follows: and ; Pi, with a value of approximately 3.1416.

[0061] (2) Finger flexion

[0062] Let the actual distance between the distal node 19 of the proximal phalanx and the distal node 18 of the middle phalanx be... The distance between the two nodes in their fully extended state is calculated using their 3D coordinates. This is a preset standard value based on ergonomic data. (Curvature) , The value ranges from 0 to 1, where 0 represents fully straight and 1 represents fully bent. ;

[0063] (3) Angle of body leaning forward

[0064] Let the coordinates of the midpoint of the shoulder be... It is obtained by averaging the coordinates of left shoulder node 11 and right shoulder node 12. The coordinates of the midpoint of the hip are It is obtained by averaging the coordinates of left hip node 23 and right hip node 24. .

[0065] First calculate the vector of the line connecting the shoulder and hip. The vertical direction vector is Assuming the Y-axis is vertical. Forward tilt angle. The calculation formula is: ;

[0066] (4) Wrist movement speed

[0067] Let the 3D coordinates of wrist node 16 in the previous frame be... The 3D coordinates of the current frame are The video frame rate is 30fps, and the frame interval time is... Second.

[0068] Speed ​​of movement The calculation formula is:

[0069] (5) Continuity of movement

[0070] Assume: for characteristic actions (such as clenching a fist), the finger flexion must be satisfied. The number of consecutive frames is The video frame rate is Actual duration The calculation formula is: .

[0071] Finally, the features such as the right arm punching angle, finger bending angle, body forward lean angle, wrist speed, elbow speed, and motion duration are combined into an 8-dimensional motion feature vector, with an example of its JSON format as: {"punch_angle":55,"finger_bend":0.7,"lean_angle":15,"wrist_speed":1.2,...}), and the motion category label and confidence score are output, such as punching: 0.92.

[0072] Furthermore, the extraction of micro-expression features in S2 specifically includes: S221: using the MediaPipeFaceDetection model to perform face detection on the RGB format video frames obtained in S11, obtaining facial bounding boxes and cropping the facial region, scaling it to 224×224 pixels; S222: loading the OpenFace toolkit to process the cropped facial region, detecting 68 facial key points, and outputting the activation intensity (0-5 levels) of 30+ facial action units (AUs); S223: performing emotion association on the AU activation intensity based on a preset AU-emotion mapping rule, outputting probability vectors of 7 basic emotions and dominant emotion labels; S224: accumulating the core AU and auxiliary AU of the dominant emotion according to weights, where the core AU has a weight of 0.6 and the auxiliary AU has a weight of 0.4, calibrating in conjunction with the emotion duration, and outputting an emotion intensity score of 0-10.

[0073] The OpenFace toolkit is used for micro-expression feature extraction, realizing the mapping from facial key points to action unit (AU) intensity, and then to emotion. The core process is as follows:

[0074] (1) Facial pretreatment

[0075] Input: Preprocessed RGB video frames;

[0076] Processing: Call the MediaPipeFaceDetection model (model_selection=1, adapted for near-field detection) to locate the face bounding box (output normalized coordinates). After converting to pixel coordinates, the face area is cropped and scaled to 224×224 pixels (the standard input size of OpenFace).

[0077] Output: 224×224 RGB image of the face.

[0078] (2) Facial key point detection and AU intensity calculation

[0079] Input: 224×224 facial image;

[0080] Processing: Load the OpenFace pre-trained model, detect 68 facial key points (such as the corners of the eyes, corners of the mouth, and nostrils), and calculate the activation intensity (0-5 levels, 0=no activation, 1=slight activation, 5=strong activation) of 30+ facial action units (AUs) based on the relative displacement and angle changes of the key points.

[0081] Output: A 30-dimensional AU intensity vector (JSON format, example: {"AU1":0,"AU4":4,"AU5":3,"AU6":0,"AU12":0,...}), with timestamps (accurate to milliseconds).

[0082] (3) Emotional association

[0083] The "AU-emotion" mapping rules, trained on the BP4D dataset, are used to match seven basic emotions (joy, anger, sadness, fear, surprise, disgust, and neutrality). The rules are as follows:

[0084] Joy: AU6 (eyelid tightening) + AU12 (corner of mouth turning up) + AU25 (lips parting), and the total core AU intensity is ≥8;

[0085] Rage: AU4 (eyebrows pull down) + AU5 (upper eyelid lift) + AU23 (lower lip pull down), and the total core AU intensity is ≥7;

[0086] Ai: AU1 (inner eyebrow raised) + AU4 (eyebrow pulled down) + AU15 (corner of mouth pulled down), and the total core AU strength is ≥6;

[0087] Fear: AU1 (inner eyebrow lift) + AU2 (outer eyebrow lift) + AU5 (upper eyelid lift) + AU20 (lip stretch), and the total core AU intensity is ≥9;

[0088] Shock: AU1 (inner eyebrow raised) + AU2 (outer eyebrow raised) + AU5 (upper eyelid raised) + AU26 (chin pulled down), and the total core AU intensity is ≥8;

[0089] Dislike: AU9 (nose wrinkles) + AU10 (upper lip lifts) + AU15 (corner of mouth pulls down), and the total core AU intensity is ≥7;

[0090] Neutral: All core AU strengths ≤ 1, and no core combinations of the above emotions are activated.

[0091] Output: A 7-dimensional emotion probability vector (example: [0.02,0.85,0.03,0.01,0.02,0.04,0.03]) and the dominant emotion label (e.g., “anger”).

[0092] (4) Emotional intensity calibration

[0093] Input: Core AU intensity vector of the dominant emotion, emotion duration (total duration from emotion activation to disappearance);

[0094] Processing: The intensity of core AUs is accumulated with a weight of 0.6, and the intensity of auxiliary AUs (non-core AUs but related to the emotion) is accumulated with a weight of 0.4 to obtain the raw score; if the duration of the emotion is <0.53 seconds (the lower limit of the standard duration of micro-expressions), the raw score is multiplied by 0.8 for calibration; if it is >3 seconds (too long is considered as stiff expression), the raw score is multiplied by 0.9 for calibration; finally, the calibrated score is mapped to 0-10 points (10 points is extremely strong);

[0095] Output: Anger intensity score (JSON format, example: {"anger_score":6}), with timestamp.

[0096] Furthermore, the extraction of speech features in step S2 specifically includes: S231: Loading the OpenAIWhisper model, performing speech-to-text processing on the PCM format audio array obtained in step S12, and outputting a list of dialogue texts with start and end timestamps; S232: Using the librosa library to extract acoustic features from the audio array, including calculating the mean fundamental frequency (f0), syllables per second (speech rate), root mean square energy (volume), and the number of pauses with silence duration > 0.5 seconds, and outputting acoustic feature data with timestamps.

[0097] The OpenAIWhisper model is used for speech-to-text conversion and acoustic feature extraction. It implements end-to-end speech processing based on the Transformer architecture, and the process is as follows:

[0098] (1) Audio preprocessing input connection

[0099] Input: A 16kHz mono PCM format audio array obtained from S12;

[0100] Processing: Convert the PCM array to floating-point data supported by the Whisper model, and divide it into 30-second segments;

[0101] Output: Normalized audio clip.

[0102] (2) Speech to text

[0103] Input: Normalized audio clip;

[0104] Processing: Load the OpenAI Whisper model (base version, balancing speed and accuracy), set the language to "Chinese", and enable "word_timestamps=True" to obtain word-level timestamps; the model extracts audio spectral features through the encoder, and the decoder combines with the language model to generate text, outputting lines with start / end timestamps;

[0105] Output: A list of line texts with timestamps.

[0106] (3)Acoustic feature extraction

[0107] Based on the librosa library, extract 4 core acoustic features by segmenting according to line timestamps:

[0108] Mean fundamental frequency: Extract the fundamental frequency of each frame through the librosa.pyin function, which is the basic frequency of the human voice. For males, it is about 85 - 180 Hz, and for females, it is about 165 - 255 Hz. Calculate the mean within the segment;

[0109] Speech rate: Count the number of syllables in the lines within the segment (split based on the jieba tokenization tool, e.g., "I've had enough" is split into 3 syllables), and divide by the segment duration (in seconds) to obtain "syllables / second";

[0110] Volume (RMS energy): Calculate the root mean square energy of the audio within the segment through librosa.feature.rms (reflecting the volume size), and convert it to decibel (dB) value;

[0111] Number of pauses: Detect silent segments within the segment through librosa.effects.split (set the threshold to -40 dB, and a silent duration > 0.5 seconds is regarded as a valid pause), and count the number of pauses.

[0112] Output: Acoustic features with timestamps.

[0113] Furthermore, the script parsing described in S3 specifically includes: S311: Load the trained LLaMA model, input the script text data, and allocate time intervals for scenes based on the pre-trained "plot-duration mapping rule" to generate script requirements with a time axis; S312: Extract the quantitative ideal standards corresponding to body movements, micro-expressions, and voices from the scene descriptions of the script text data, determine the scoring dimensions and weights of "emotional fit, action normality, voice adaptability" (emotional fit 40%, action normality 30%, voice adaptability 30%), and generate a scoring benchmark table.

[0114] Furthermore, the data alignment described in S3 specifically includes: matching the timestamps of the body movement features, micro-expression features, and voice features obtained in S2 with a time precision of 100ms, and binding the actual feature data of the corresponding time point with the script requirements to generate a three-dimensional comparison table containing timestamps, actual features, and script requirements. The actual feature data includes punch angle, AU intensity, and fundamental frequency value.

[0115] Furthermore, the large language model reasoning and result output described in S4 specifically includes: S41: Integrating the three-dimensional comparison table and the scoring benchmark table into a structured Prompt according to the template of "time point + actual performance + script requirements + scoring rules"; S42: Loading the LLaMA model finely tuned with 500+ actor performance cases, inputting the structured Prompt, comparing actual features with expected standards, calculating the quantitative score of 0-10 points for each scoring dimension, and converting it into a total score according to weight; S43: Outputting the scoring results in JSON format and the evaluation report in text format.

[0116] The following is one application of the present invention:

[0117] Before making an evaluation, the shooting footage and script footage were prepared. The shooting footage included: an actor's performance clip, performance.mp4, 1920×1080 resolution, 30fps, the content of which was "Act 3, the protagonist is angry because he is betrayed, he punches his fist and roars: 'I've had enough!'"

[0118] Script material: plain text file script.txt, content is the same as above.

[0119] Data Acquisition and Preprocessing

[0120] 1. Data Acquisition:

[0121] Video: The performance video was recorded using a professional camera and saved as performance.mp4 (1920×1080, 30fps).

[0122] Audio: Separate the audio track from the video and save it as performance.wav (sampling rate 44.1kHz).

[0123] Script: Enter script.txt, which contains scene descriptions.

[0124] 2. Pretreatment:

[0125] Video frame processing:

[0126] Using OpenCV to read video: cv2.VideoCapture("performance.mp4") reads BGR frames and converts them to RGB format using cv2.cvtColor(frame,cv2.COLOR_BGR2RGB);

[0127] Cropping the face region: For each frame of RGB image, the MediaPipeFaceDetection model is used to perform detection on each frame of RGB image, obtain the normalized bounding box coordinates (xmin, ymin, width, height), convert them into pixel coordinates, crop the face region according to the bounding box, and scale it to 224×224 pixels.

[0128] Audio processing:

[0129] Load performance.wav using librosa, resample it to 16kHz mono, and generate a PCM format array audio_array.

[0130] Multimodal feature extraction

[0131] 1. Extraction of limb movement features

[0132] Key node detection:

[0133] Input: RGB video frames (1080×1920×3);

[0134] Processing: Load the MediaPipePose model (model_complexity=2), detect 33 key nodes across the entire body (head, shoulders, elbows, wrists, etc.), and output:

[0135] Single-frame 2D coordinates: (33,2) normalized array (e.g., shoulder node 12 coordinates [0.45,0.22]), and visibility confidence (33,1) (e.g., wrist node 16 confidence 0.98);

[0136] Temporal 3D trajectory: 3D relative coordinates (16,33,3) of 16 consecutive frames (e.g., the 3D coordinates of wrist node 16 in frame 10 are [0.12,0.34,0.05]).

[0137] Feature quantization:

[0138] Input: 2D coordinates, 3D trajectory and time stamp (e.g., 00:00:10.200);

[0139] Processing: Call the FeatureizableMachine model to calculate core action features:

[0140] Range of motion: right arm punching angle (calculated from three points: shoulder 12, elbow 14, wrist 16, resulting in 55°), body forward lean angle 15°, etc.

[0141] Movement speed: 3D distance between adjacent frames of wrist node 16 ÷ 1 / 30 seconds, equals 1.2m / s;

[0142] Action duration: The clenched fist state (bending angle ≥ 0.8) lasts for 3 frames, which is equivalent to 0.1 seconds;

[0143] Output: 8-dimensional action feature vector (JSON format, such as {"punch_angle":55,"speed":1.2,...}), and action label "punch: 0.92" (with timestamp).

[0144] Micro-expression feature extraction

[0145] AU detection:

[0146] Input: Cropped RGB image of the face (224×224);

[0147] Processing: OpenFace is invoked to detect 68 key points and calculate 30+ AU intensities (e.g., AU4 (eyebrow pull-down) intensity 4, AU5 (upper eyelid lift) intensity 3).

[0148] Output: 30-dimensional AU intensity vector (JSON, e.g., {"AU4":4,"AU5":3,...}, with timestamp 00:00:10.200).

[0149] Emotional association:

[0150] Input: AU intensity vector;

[0151] Processing: Match the core AU combination of "anger" (AU4+AU5+AU23), calculate the probability of 7 emotions, and the dominant emotion is "anger";

[0152] Output: An emotion probability vector (e.g., [0, 0.85, 0, ...]) and the label "anger".

[0153] Strength calibration:

[0154] Input: Emotion probability vector, duration (e.g., the angry expression lasts for 1.2 seconds);

[0155] Processing: Weighted sum of core AU (AU4, AU5) (0.6 weight) + auxiliary AU (0.4 weight), combined with duration calibration, to obtain an emotional intensity score of 6;

[0156] Output: Anger intensity score (JSON, e.g., {"anger_score":6}, with timestamp).

[0157] Speech feature extraction (corresponding to the language recognition module)

[0158] Speech-to-text:

[0159] Input: The preprocessed audio array audio_array;

[0160] Processing: Call the Whisper model to transcribe the dialogue and output the timestamped text.

[0161] [{"start":"00:00:10.000","end":"00:00:12.500","text":"I've had enough!"}]

[0162] Acoustic feature extraction:

[0163] Input: audio array, transcription timestamp;

[0164] Processing: Extract features using librosa:

[0165] The fundamental frequency (f0) has an average value of 280 Hz;

[0166] Speech rate: 5 syllables / second (text length: 3 characters ÷ 0.6 seconds);

[0167] Volume (RMS) 18dB;

[0168] Number of pauses: 0 (no silence > 0.5 seconds);

[0169] Output: JSON with 4 categories of features (e.g., {"f0":280,"speed":5,...}, with timestamps).

[0170] Data alignment and script analysis

[0171] Script Analysis

[0172] Input: script.txt (Content: "Act 3, the protagonist, enraged by betrayal, raises his fist and roars: 'I've had enough!'")

[0173] deal with:

[0174] Timeline allocation: Based on the mapping pattern of "anger scene + single action + short sentence", the time interval is determined to be 00:00:10.000-00:00:15.000;

[0175] Extracting the normative standard:

[0176] Limbs: Punch angle 50°-60°;

[0177] Micro-expressions: AU4+AU5 intensity ≥ 7 points;

[0178] Voice: Base frequency around 290Hz;

[0179] Output: {"Deserved Standard":{"punch_angle":"50°-60°","au4+au5":"≥7 points","f0":"around 290Hz"},"Scoring Dimensions":["Emotional Fit","Movement Standardization","Voice Adaptability"]}

[0180] Data alignment (corresponding to the data alignment unit of the large language fusion module)

[0181] enter:

[0182] Body language, micro-expression features, and voice features (all with timestamps, such as body language feature timestamp 00:00:10.200).

[0183] The script's time frame is 00:00:10.00000:00:15.000;

[0184] Processing: Matching with 100ms precision, binding actual features to script requirements, and generating a 3D lookup table:

[0185] json

[0186] [{"timestamp":"00:00:10.200","actual":{"punch_angle":55°,"au4+au5_score":6,"f0":280Hz},"script_req":"Angry scene, requires punching (50°-60°), AU4+AU5 intensity ≥ 7 points, fundamental frequency around 290Hz"}]

[0187] Large Language Model Inference and Output

[0188] Prompt build

[0189] Input: Three-dimensional comparison table, scoring criterion table;

[0190] Processing: Generate structured instructions based on the template;

[0191] Output: Structured instructions.

[0192] Large Language Model Reasoning (corresponding to the Large Language Fusion Module Large Language Model Reasoning Unit)

[0193] Input: Structured Prompt;

[0194] Processing: Load the LLaMA model finely tuned with 500+ performance examples and compare it with the standard:

[0195] Movement standardization: The angle of the punch meets the standard, scoring 9.0 points;

[0196] Emotional fit: AU intensity did not meet the standard, scoring 8.0 points;

[0197] Voice compatibility: Fundamental frequency deviation, score 7.5;

[0198] Total score: 24.5 points;

[0199] Improvement suggestions: "The emotional intensity needs to be increased (AU4+AU5 intensity is recommended to be ≥7 points), and the pitch can be adjusted to around 290Hz."

[0200] Output Results

[0201] JSON scoring results:

[0202] {"timestamp":"00:00:10.200",

[0203] "Emotional fit": 8.0

[0204] "Motion Standardization": 9.0

[0205] Voice compatibility: 7.5

[0206] Total Score: 24

[0207] Visualized Report:

[0208] 00:00:10.200 Performance Evaluation:

[0209] Movement standardization: 9.0 points (punch angle 55°, conforming to the 50°60° standard);

[0210] Emotional fit: 8.0 (AU4+AU5 intensity: 6, slightly below the requirement of ≥7).

[0211] Voice compatibility score: 7.5 (base frequency 280Hz, it is recommended to adjust to around 290Hz);

[0212] Total score: 24 points

[0213] Suggested improvements: Strengthen facial muscle control to enhance the intensity of anger, and adjust the tone of voice to around 290Hz.

[0214] A system for evaluating actors' acting skills based on a large language model and multimodal comprehensive evaluation includes:

[0215] The limb recognition module is used to extract the limb movement features of the actor, including a key node detection unit and a feature quantization unit. The key node detection unit uses the MediaPipePose model (model_complexity=2) to process RGB video frames and outputs the 2D coordinates and time-series 3D trajectory of 33 key nodes of the whole body. The feature quantization unit uses the FeaturizableMachine model to calculate the amplitude of movement, movement speed and movement duration, and outputs the limb movement feature vector.

[0216] The micro-expression recognition module is used to extract micro-expression features of actors, including an AU detection unit, an emotion association unit, and an intensity calibration unit. The AU detection unit uses the OpenFace toolkit to detect the facial AU intensity. The emotion association unit determines the dominant emotion based on the AU-emotion mapping rule trained on the BP4D dataset. The intensity calibration unit outputs an emotion intensity score by calibrating the AU weighted accumulation and duration.

[0217] The speech recognition module is used to extract the voice features of actors, including an audio preprocessing unit, a speech-to-text unit, and an acoustic feature extraction unit; the audio preprocessing unit performs noise removal and volume normalization; the speech-to-text unit uses the Whisper model to generate timestamped lines; the acoustic feature extraction unit uses the librosa library to extract fundamental frequency, speech rate, volume, and number of pauses.

[0218] The Large Language Fusion Evaluation Module is used to achieve the fusion evaluation of multimodal features and scripts. It includes a script benchmark analysis unit, a data alignment unit, a prompt construction unit, and a large language model inference unit. The script benchmark analysis unit allocates scene timelines, extracts normative standards, and assigns scoring weights. The data alignment unit generates a three-dimensional comparison table with 100ms precision. The prompt construction unit generates structured instructions. The large language model inference unit loads an LLaMA model finely tuned with over 500 performance cases and outputs quantitative scores and improvement suggestions.

[0219] The final acting score and suggestions are generated using an LLaMA model (Large Language Model MetaAI, developed by MetaAI) fine-tuned with over 500 actor performance examples. The fine-tuning process employs the Low-Rank Adaptation (LoRA) algorithm, whose core idea is to achieve efficient fine-tuning by introducing two trainable low-rank matrices to simulate parameter changes without altering the parameters of the original large model.

[0220] Assume the parameter matrix of the original large model is (m and n represent the high dimension, indicating model complexity), the LoRA algorithm superimposes two low-rank matrices. and (where k is the rank of the matrix), the fine-tuned parameter matrix for

[0221] Only optimize during the training phase The parameters are set, and the loss function used is the cross-entropy loss function. For the training samples...

[0222] in This is the probability that the model predicts sample i as class j. The loss function is calculated using the backpropagation algorithm. The gradient is used to update the parameters using stochastic gradient descent (SGD) or its variants (Adagrad, Adadelta, RMSProp, Adam) to minimize the loss and improve the model's performance on the acting evaluation task.

[0223] like Figure 1 As shown, the system includes a body recognition module, a micro-expression recognition module, a speech recognition module, and a large-scale language fusion evaluation module. Each module uses a unified timestamp and project number to link data, ensuring traceability from raw data to evaluation results.

[0224] Body Recognition Module: This module includes a key node detection unit and a feature quantization unit, used to extract quantitative features of the actor's body movements. The specific implementation method is as follows:

[0225] Key node detection unit:

[0226] The MediaPipePose model (e.g., model_complexity=2, high-precision mode) is used to process the input video frame sequence (resolution ≥1920×1080, frame rate ≥30fps) in real time. The video frame sequence is extracted from local video files (e.g., MP4, AVI format) or live camera streams using OpenCV, and generated after preprocessing: first, the original BGR frames are read using cv2.VideoCapture, then converted to RGB format using cv2.cvtColor, finally forming continuous RGB video frames.

[0227] Input: Continuous RGB video frames (single frame format: (1080 (height), 1920 (width), 3 (color channels)), pixel value (0255));

[0228] Processing logic: The BlazePose backbone network is used to detect human body regions and regress the coordinates and visibility confidence of 33 key nodes in the whole body (including head, shoulder, elbow, wrist, hip, knee, ankle, etc.).

[0229] Output: Single-frame 2D coordinates: a normalized array of coordinates (x, y ∈ [0, 1]) of (33 (number of key nodes), 2 (2D coordinates)), with visibility confidence (0, 1) of (33, 1);

[0230] Temporal 3D trajectory: A sequence of 16 consecutive 3D coordinates (format (16,33,3), including X, Y, and Z three-dimensional relative coordinates), generated through inter-frame correlation processing.

[0231] Feature quantization unit:

[0232] Based on the coordinates (2D / 3D) of 33 key nodes output by the key node detection unit, and input into a publicly available pre-trained action feature classifier (such as the FeaturizableMachine model trained on the Kinetics400 dataset, which is open source and commercially available), integrated processing of feature quantization and action recognition is achieved.

[0233] Input: a single-frame 2D coordinate array (33,2), a temporal 3D trajectory sequence (16,33,3) and corresponding timestamps;

[0234] Processing logic:

[0235] The model automatically calculates core action features based on key node coordinates through convolutional layers and attention mechanisms, including:

[0236] Range of motion: right arm swing angle (e.g., calculated by the vector dot product of three points: shoulder node 12, elbow node 14, and wrist node 16, in °), finger flexion (e.g., actual distance between knuckle nodes 19 and 18 / distance in the straightened state, range 01), body forward lean angle (e.g., the angle between the line connecting the midpoint of the shoulder and the midpoint of the hip and the vertical direction, in °).

[0237] Motion speed: 3D Euclidean distance between adjacent frames of wrist node 16 / frame interval time (1 / 30 seconds), unit m / s;

[0238] Action duration: The actual time (in seconds) converted from the number of frames of a characteristic action (such as a clenched fist with a bend of ≥0.8).

[0239] The model's built-in classification head identifies action categories (such as waving, bending over, clenching a fist, etc.) based on temporal feature sequences, and outputs the label with the highest matching probability and confidence score.

[0240] Output:

[0241] (8,) action feature vector (containing 5 types of amplitude features, 2 types of velocity features, and 1 type of duration feature), in JSON string format;

[0242] Action category label and confidence level (e.g., "waving: 0.92 (confidence level)"), with timestamp.

[0243] Micro-expression recognition module

[0244] This module includes an AU detection unit, an emotion association unit, and an intensity calibration unit, realizing the mapping from facial movements to emotion intensity. The specific implementation method is as follows:

[0245] AU detection unit:

[0246] The OpenFace toolkit was used to process the cropped facial area (224×224 resolution) in the video:

[0247] Input: RGB image of a face ((224,224,3));

[0248] Processing logic: Detect 68 facial key points and calculate the activation intensity of 30+ facial action units (AUs) (0-5 level, 0 for inactive).

[0249] Output: AU intensity vector of (30,) in JSON string format with timestamp.

[0250] Emotional Connection Unit:

[0251] Emotion types are associated based on a pre-defined "AU emotion" mapping rule (trained using the BP4D dataset):

[0252] Input: The intensity vector output by the AU detection unit;

[0253] Processing logic: For the 7 basic emotions (joy, anger, sorrow, fear, surprise, disgust, and neutrality), a core AU combination is used.

[0254] Joy: AU6 (tighten eyelids) + AU12 (turn up corners of mouth) + AU25 (split lips);

[0255] Anger: AU4 (eyebrows pull down) + AU5 (upper eyelid lift) + AU23 (lower lip pull down);

[0256] Sorrow: AU1 (Raise the inner eyebrow) + AU4 (Pull down the eyebrow) + AU15 (Pull down the corner of the mouth);

[0257] Fear: AU1 (raise inner eyebrow) + AU2 (raise outer eyebrow) + AU5 (raise upper eyelid) + AU20 (stretch lips);

[0258] Surprise: AU1 (inner eyebrow raised) + AU2 (outer eyebrow raised) + AU5 (upper eyelid raised) + AU26 (chin lowered);

[0259] Dislike: AU9 (nose wrinkles) + AU10 (upper lip lifts) + AU15 (corners of mouth pull down);

[0260] Neutral: No core AU combination activation (all AU strength ≤ level 1).

[0261] Calculate matching degree

[0262] Output: The emotion probability vector (summing up to 1) of (7,) and the dominant emotion label (e.g., “anger”).

[0263] Intensity calibration unit:

[0264] Standardize the intensity of emotions:

[0265] Input: Emotion probability vector and duration;

[0266] Processing logic: The core AU intensity is weighted and accumulated (core AU weight 0.6, auxiliary AU weight 0.4), and then calibrated based on the duration (0.53 seconds is the effective range);

[0267] Output: Emotion intensity score from 0 to 10 (10 is extremely strong), in JSON string format.

[0268] Language recognition module

[0269] This module includes an audio preprocessing unit, a speech-to-text unit, and an acoustic feature extraction unit, enabling comprehensive analysis of speech information. The specific implementation method is as follows:

[0270] Audio preprocessing unit:

[0271] Preprocessing of the separated audio tracks (sampling rate 44.1kHz) in the video:

[0272] Input: Raw audio WAV file;

[0273] Output: 16kHz mono audio array (in PCM format of (N,)).

[0274] Speech-to-text unit:

[0275] Speech-to-text conversion using the OpenAI Whisper model (base version):

[0276] Input: Preprocessed audio array;

[0277] Output: Dialogue with timestamps (e.g., "00:00:10.20000:00:12.500 I've had enough!").

[0278] Output: A list of text fragments with timestamps (JSON format).

[0279] Acoustic feature extraction unit:

[0280] Speech features were extracted using the librosa library:

[0281] Pitch: Mean of fundamental frequency (f0) (unit: Hz);

[0282] Speech rate: syllables per second (text length ÷ duration);

[0283] Volume: Root mean square energy (RMS, unit: dB);

[0284] Number of pauses: Number of intervals with a silence duration greater than 0.5 seconds;

[0285] Output: JSON strings representing the four characteristics mentioned above, along with timestamps.

[0286] Large Language Fusion Evaluation Module

[0287] This module includes a script benchmark analysis unit, a data alignment module, a Prompt construction unit, and a large language model inference unit to achieve multimodal evaluation. The specific implementation method is as follows:

[0288] Script benchmark analysis unit:

[0289] Input: Original script text (without a timeline, such as "Act 3, the protagonist, enraged by betrayal, raises his fist and roars: 'I've had enough!'");

[0290] Processing logic: A finely tuned LLaMA model is used to parse the script and complete two core tasks:

[0291] Assign timelines to scenes: Generate script requirements with time intervals based on pre-trained plot duration mapping patterns;

[0292] Extract the normative standards: Infer quantitative requirements from the scene description (body: punching 50°-60°; micro-expression: AU4+AU5 intensity ≥7 points; voice: fundamental frequency around 290Hz), and determine the scoring dimensions (emotional fit, action standardization, voice suitability) and their weights (40%, 30%, 30%).

[0293] Output:

[0294] Script requirements with a timeline;

[0295] A JSON-formatted scoring benchmark table (e.g., {"Deserved Standard":{"punch_angle":"50°60°","au4+au5":"≥7 points","f0":"around 290Hz"},"scoring dimensions":["emotional fit","action standardization","voice fit"],"weight":[0.4,0.3,0.3]}).

[0296] Data alignment unit:

[0297] enter:

[0298] Body movement features (JSON with timestamps), micro-expression intensity (JSON with timestamps), and voice features (JSON with timestamps).

[0299] The script requirements with timeline output by the script benchmark analysis unit;

[0300] Processing logic: With a precision of 100ms, match the timestamp of the actual feature with the time interval of the script, and bind the actual feature at the corresponding time point (such as a punch at 55°, AU4 intensity of 5 points) with the script requirements (such as "anger scenes require punching").

[0301] Output: A 3D lookup table in JSON format.

[0302] The model is based on a fine-tuned LLaMA-7B model and implements the timeline allocation of the script, extraction of normative standards, and definition of scoring dimensions. The core process is as follows:

[0303] (1) Model input preprocessing

[0304] Input: Original script text (no timeline, example: "Act 3, the protagonist, enraged by betrayal, raises his fist and roars: 'I've had enough!'; Act 4, the protagonist calmly negotiates, speaking in a measured tone: 'Let's talk again'").

[0305] Processing: Clean the text (remove special characters, standardize line breaks), add instruction prefixes ("Please parse the following script and output the timeline, expected standards and scoring dimension weights for each scene:"), and convert it into a token sequence supported by the model;

[0306] Output: The sequence of tokens that the model inputs.

[0307] (2) Scene timeline allocation

[0308] Based on the pre-trained "plot-duration mapping pattern" (trained from 500+ script samples), a time interval is assigned to each scene:

[0309] Mapping rule example: "Intense emotional scene (anger / fear) + single action (punching / running) + short sentence (≤5 words)" → 5 seconds; "Calm emotional scene (calm / sad) + no action + long sentence (>10 words)" → 10 seconds;

[0310] Processing: The model performs semantic classification (emotion type, number of actions, line length) on each scene text and outputs the time interval by matching mapping rules;

[0311] Output: Script with timeline. Requirements:

[0312] (3) Extraction of normative standards

[0313] The model extracts quantification requirements for body language, micro-expressions, and speech from scene descriptions, based on a scene-standard mapping library.

[0314] Normal body language standards: For example, "anger + punching" → punching angle 50°-60°, body leaning forward angle 10°-20°; "calm + no movement" → body leaning forward angle ≤5°, no large hand movements;

[0315] Microexpression normative standards: For example, "anger" → AU4 + AU5 intensity ≥ 7 points, dominant emotion is "anger"; "calm" → all AU intensity ≤ 2 points, dominant emotion is "neutral";

[0316] Standard speech patterns: For example, "angry + roaring" → base frequency 280-300Hz, volume ≥ -15dB; "calm + gentle" → base frequency 200-220Hz, volume ≤ -20dB;

[0317] Output: Standard JSON.

[0318] (4) Determination of scoring dimensions and weights

[0319] The scoring dimensions are weighted based on the "emotional priority" of the scenario.

[0320] In emotion-driven scenarios (such as anger or sadness): Emotional relevance 40%, Action accuracy 30%, Voice compatibility 30%;

[0321] Action-driven scenarios (such as martial arts and dance): Action accuracy 40%, Emotional fit 30%, Voice compatibility 30%;

[0322] In voice-dominated scenarios (such as monologues and recitations): voice compatibility 40%, emotional fit 30%, and proper action 30%;

[0323] Output: Scoring benchmark table.

[0324] Prompt building blocks:

[0325] enter:

[0326] A three-dimensional lookup table output by the data alignment unit;

[0327] The scoring benchmark table output by the script benchmark analysis unit;

[0328] Processing logic: Integrate "actual characteristics", "ideal standards", and "scoring dimensions and weights" according to the template to generate structured instructions (clearly specifying time points, actual performance, script requirements and scoring rules);

[0329] Output: Structured text instructions (e.g., "Evaluate the performance at 00:00:10.200: Actual punch angle 55° (standard 50° 60°), AU4 intensity 5 points (standard ≥ 7 points), fundamental frequency 280Hz (standard around 290Hz). Please score according to emotional fit (40%), action standardization (30%), and voice suitability (30%) (010 points)").

[0330] Large Language Model Inference Unit:

[0331] Input: Structured text instructions output by the Prompt building block;

[0332] Processing logic: Load the LLaMA model that has been fine-tuned with 500+ performance cases, compare the actual characteristics with the expected standards, calculate the scores (0, 10 points) of each dimension based on case experience, convert the total score according to the weight, and generate improvement suggestions;

[0333] Output:

[0334] The rating results are in JSON format, such as "timestamp":"00:00:10.200","emotional fit":8.5,"action standardization":9.0,"voice fit":7.0,"total score":8.2;

[0335] Text-based evaluation, such as "00:00:10.200: Emotional fit 8.5 points, Action conformity 9.0 points, Voice fit 7.0 points, Total score 8.2 points. Suggestion: Adjusting the tone to 290Hz would be better."

[0336] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for evaluating actors' acting skills based on a large language model and multimodal comprehensive evaluation, characterized in that, Includes the following steps: S1. Data acquisition and preprocessing: Acquire video data of the actors' performance, audio data separated from the video data, and corresponding script text data; perform frame format conversion processing on the video data; and perform noise removal and sampling rate adjustment processing on the audio data. S2. Multimodal feature extraction: After preprocessing, body movement features, micro-expression features, and speech features are extracted respectively. The body movement features include key node coordinates and movement amplitude, speed, and duration features. The micro-expression features include facial action unit intensity, emotion type, and emotion intensity. The speech features include timestamped dialogue text and tone, speech rate, volume, and pause features. S3. Data alignment and script parsing: Time axis allocation and normative standard extraction are performed on the script text data to obtain script requirements and scoring benchmark table with time intervals; the body movement features, micro-expression features, and voice features are matched with the time intervals of the script requirements according to timestamps to generate a three-dimensional comparison table; S4. Large Language Model Reasoning and Result Output: Integrate the three-dimensional comparison table and the scoring benchmark table into a structured Prompt, input it into the large language model fine-tuned by the performance case, and generate quantitative scores for each dimension, total score, and improvement suggestions on the timeline.

2. The method for evaluating actor emotional expression based on large language model reasoning according to claim 1, characterized in that, The data acquisition and preprocessing described in S1 specifically includes: S11, acquiring performance video data with a resolution ≥1920×1080 and a frame rate ≥30fps, reading video frames using computer vision tools, and converting BGR format frames to RGB format frames; S12, separating audio data with a sampling rate of 44.1kHz from the video data, removing background noise using noise spectrum subtraction, unifying audio amplitude through volume normalization, and then resampling to 16kHz mono to obtain a PCM format audio array.

3. The method for evaluating actor emotional expression based on large language model reasoning according to claim 1, characterized in that, The extraction of limb movement features described in S2 specifically includes: S211: Loading the MediaPipePose model, processing the RGB format video frames obtained in step S11, detecting and outputting 2D normalized coordinate arrays of 33 key nodes of the whole body, visibility confidence, and 3D relative coordinate trajectories of 16 consecutive frames; S212: Inputting the 2D normalized coordinate arrays, 3D relative coordinate trajectories, and time stamps into the FeaturizableMachine model trained on the Kinetics400 dataset. The FeaturizableMachine model is a pre-trained model for motion feature quantization, calculating motion amplitude, motion speed, and motion duration features, and outputting an 8-dimensional motion feature vector and motion category labels and confidence. The MediaPipePose model is used for limb motion feature extraction and employs a two-step detector and tracker machine learning pipeline, as follows: The detector is based on a lightweight convolutional neural network (CNN) to locate the human body and the region of interest (ROI) in RGB video frames and output the human body bounding box; The tracker takes a cropped frame of the ROI as input and outputs the two-dimensional coordinates of each keypoint using a regression method. and visibility; For 33 key nodes in the whole body, output a 2D normalized coordinate array and a visibility confidence array; The 3D relative coordinate trajectory combines multi-frame information, geometric constraints, and calibrated camera parameters to convert 2D coordinates into 3D relative coordinates using triangulation.

4. The method for evaluating actor emotional expression based on large language model reasoning according to claim 3, characterized in that, The calculation of motion amplitude, motion speed, and motion duration features described in S212 specifically includes: Motion amplitude calculation: calculating the right arm swing angle based on the dot product of the coordinate vectors of shoulder node 12, elbow node 14, and wrist node 16; calculating the finger flexion based on the ratio of the actual distance between finger nodes 19 and 18 to the distance in the straightened state; and calculating the body forward lean angle based on the angle between the line connecting the midpoint of the shoulder and the midpoint of the hip and the vertical direction; Motion speed calculation: calculating the 3D Euclidean distance between adjacent frames of wrist node 16 and dividing it by the frame interval time to obtain the motion speed; Motion duration calculation: counting the number of frames of the characteristic motion and converting it into actual time.

5. The method for evaluating actor emotional expression based on large language model reasoning according to claim 1, characterized in that, The extraction of micro-expression features in S2 specifically includes: S221: Using the MediaPipeFaceDetection model to perform face detection on the RGB format video frames obtained in step S11, obtaining facial bounding boxes and cropping the facial region, scaling it to 224×224 pixels; S222: Loading the OpenFace toolkit to process the cropped facial region, detecting 68 facial key points, and outputting the activation intensity of 30+ facial action units (AUs); S223: Associating the AU activation intensity with emotions based on preset AU-emotion mapping rules, outputting probability vectors of 7 basic emotions and dominant emotion labels; S224: Accumulating the core AU and auxiliary AU of the dominant emotion according to weights, where the core AU has a weight of 0.6 and the auxiliary AU has a weight of 0.4, calibrating in conjunction with the emotion duration, and outputting an emotion intensity score of 0-10.

6. The method for evaluating actor emotional expression based on large language model reasoning according to claim 1, characterized in that, Step S2, specifically the extraction of speech features, includes: S231: Loading the OpenAI Whisper model, performing speech-to-text processing on the PCM format audio array obtained in step S12, and outputting a list of dialogue texts with start and end timestamps; S232: Using the librosa library to extract acoustic features from the audio array, including calculating the fundamental frequency. Output timestamped acoustic characteristic data, including mean, syllables per second, root mean square energy, and number of pauses with a silence duration greater than 0.5 seconds.

7. The method for evaluating actor emotional expression based on large language model reasoning according to claim 1, characterized in that, The script parsing described in S3 specifically includes: S311: Loading the trained LLaMA model, inputting the script text data, allocating time intervals for scenes based on the pre-trained plot and duration mapping rules, and generating script requirements with a timeline; S312: Extracting quantitative normative standards corresponding to body movements, micro-expressions, and voice from the scene descriptions in the script text data, determining the scoring dimensions and weights of emotional fit, action standardization, and voice suitability, and generating a scoring benchmark table.

8. The method for evaluating actor emotional expression based on large language model reasoning according to claim 1, characterized in that, The data alignment described in S3 specifically includes: matching the timestamps of the body movement features, micro-expression features, and voice features obtained in S2 with the time interval required by the script obtained in S311, using a time precision of 100ms, binding the actual feature data of the corresponding time point with the script requirements, and generating a three-dimensional comparison table containing timestamps, actual features, and script requirements. The actual feature data includes punch angle, AU intensity, and fundamental frequency value.

9. The method for evaluating actor emotional expression based on large language model reasoning according to claim 1, characterized in that, The large language model reasoning and output described in S4 specifically include: S41: Integrating the three-dimensional comparison table and the scoring benchmark table into a structured Prompt based on templates of time points, actual performance, script requirements, and scoring rules; S42: Loading the LLaMA model finely tuned with 500+ actor performance cases, inputting the structured Prompt, comparing actual features with expected standards, calculating the quantitative score of 0-10 points for each scoring dimension, and converting it into a total score according to weights; S43: Outputting the scoring results in JSON format and the evaluation report in text format.

10. A system for evaluating actors' acting skills based on a large language model and multimodal comprehensive evaluation, characterized in that: To implement the method according to any one of claims 1-9, comprising: The limb recognition module is used to extract the limb movement features of the actor, including a key node detection unit and a feature quantization unit. The key node detection unit uses the MediaPipePose model to process RGB video frames and outputs the 2D coordinates and time-series 3D trajectory of 33 key nodes of the whole body. The feature quantization unit uses the FeaturizableMachine model to calculate the amplitude of movement, movement speed and movement duration, and outputs the limb movement feature vector. The micro-expression recognition module is used to extract micro-expression features of actors, including an AU detection unit, an emotion association unit, and an intensity calibration unit. The AU detection unit uses the OpenFace toolkit to detect the facial AU intensity. The emotion association unit determines the dominant emotion based on the AU-emotion mapping rule trained on the BP4D dataset. The intensity calibration unit outputs an emotion intensity score by calibrating the AU weighted accumulation and duration. The speech recognition module is used to extract the voice features of actors, including an audio preprocessing unit, a speech-to-text unit, and an acoustic feature extraction unit; the audio preprocessing unit performs noise removal and volume normalization; the speech-to-text unit uses the Whisper model to generate timestamped lines; the acoustic feature extraction unit uses the librosa library to extract fundamental frequency, speech rate, volume, and number of pauses. The Large Language Fusion Evaluation Module is used to achieve the fusion evaluation of multimodal features and scripts. It includes a script benchmark analysis unit, a data alignment unit, a prompt construction unit, and a large language model inference unit. The script benchmark analysis unit allocates scene timelines, extracts normative standards, and assigns scoring weights. The data alignment unit generates a three-dimensional comparison table with 100ms precision. The prompt construction unit generates structured instructions. The large language model inference unit loads an LLaMA model finely tuned with over 500 performance cases and outputs quantitative scores and improvement suggestions.

Images