Chinese opera performance identification method and system based on Chinese opera video

Abstract

The invention provides a traditional opera performance identification method based on a traditional opera video, and the method comprises the following steps: obtaining the traditional opera video, decoding the traditional opera video, and obtaining a video frame sequence; screening out key frames from the video frame sequence by adopting a frame difference method in combination with traditional opera performance rhythm characteristics; carrying out feature extraction on the screened key frames, wherein the extracted features comprise action features, costume features and makeup features; constructing an identification model based on a deep learning network and performing training; fusing the extracted action features, costume features and makeup features to obtain fused features; and inputting the fusion features into a trained recognition model, and outputting the type of opera performance by the recognition model. The method can adapt to opera video recognition of different theaters and different performance styles.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of opera identification, and particularly relates to an opera performance identification method and system based on opera videos. BACKGROUND

[0002] As a brilliant gem of traditional Chinese culture, opera covers many types such as Peking opera, Yueju opera, and Huangmei opera, each of which has unique characteristics in performance style, costumes, makeup, and movements. In the current era of rapid development of digital technology, a large number of opera performances are recorded as videos and stored, and these videos carry rich cultural connotations.

[0003] Identifying performances in opera videos can provide key basis for opera research, teaching, inheritance, and related cultural product development. However, current opera performance identification mainly relies on manual video watching and judgment, which is extremely inefficient and greatly affected by subjective factors, and the accuracy and consistency of the identification results are difficult to guarantee.

[0004] Some existing video identification methods are mostly applied in the fields of general action identification and object identification. Due to the strong action standardization and complex costumes and makeup of opera performances, these general methods cannot be directly applied to opera performance identification, and the identification effect is not good. Therefore, it is necessary to develop an opera performance identification method specifically for opera videos. SUMMARY

[0005] The present application aims to solve the problems in the prior art that opera video performance identification relies on manual work, resulting in low efficiency and poor accuracy, and that general video identification methods are not applicable.

[0006] The technical solution of the present application is as follows:

[0007] An opera performance identification method based on opera videos, comprising the following steps:

[0008] Obtaining an opera video, decoding the opera video to obtain a video frame sequence, and screening key frames from the video frame sequence by using a frame difference method combined with opera performance rhythm characteristics;

[0009] Extracting features from the screened key frames, including action features, costume features, and makeup features;

[0010] Constructing and training an identification model based on a deep learning network, fusing the extracted action features, costume features, and makeup features to obtain fused features, and inputting the fused features into the trained identification model to output the type of opera performance.

[0011] Further, the specific process of screening key frames by using the frame difference method combined with opera performance rhythm characteristics is as follows:

[0012] Calculate the pixel difference of two adjacent frames in a sequence of video frames, and mark the frame as a candidate key frame when the pixel difference exceeds a set threshold; combine the rhythm characteristics of the singing, reciting, doing and fighting links in opera performance to screen the candidate key frames and retain the key frames.

[0013] Further, the process of action feature extraction is: using a skeleton key point detection algorithm to detect the skeleton key points of the character in the key frame to obtain the skeleton key point coordinates; calculating the skeleton vector and joint angle according to the skeleton key point coordinates, and taking the skeleton vector and joint angle as the action feature.

[0014] Further, the process of costume feature extraction is: using an image segmentation algorithm to segment the costume area of the character in the key frame; performing color feature extraction and texture feature extraction on the segmented costume area; and combining the color feature and the texture feature as the costume feature.

[0015] Further, the process of makeup feature extraction is: using a face feature point detection algorithm to locate the face feature points of the character in the key frame; determining the makeup area according to the face feature points, and performing feature extraction on the makeup area, including the shape feature and the color feature of the makeup, and taking the shape feature and the color feature of the makeup as the makeup feature.

[0016] Further, the process of constructing and training the recognition model based on the deep learning network is:

[0017] Constructing a recognition model combining convolutional neural network and recurrent neural network; collecting opera video samples with opera performance type labels;

[0018] Inputting the sample fusion features and corresponding labels into the initial model, adjusting the parameters through back propagation until the recognition accuracy of the model on the validation set reaches a preset threshold.

[0019] Further, the fusion process of the action feature, the costume feature and the makeup feature includes:

[0020] Performing feature vectorization processing on the action feature, the costume feature and the makeup feature, and using feature splicing or feature attention mechanism to perform multi-feature fusion to form a unified fusion feature vector.

[0021] An opera performance recognition system based on opera videos, the system comprising:

[0022] A video preprocessing module for obtaining an opera video, decoding the opera video to obtain a sequence of video frames, and using frame difference method combined with the rhythm characteristics of opera performance to screen out key frames from the sequence of video frames;

[0023] The performance feature extraction module is configured to extract features from the selected key frames, and the extracted features include action features, costume features and makeup features.

[0024] The performance recognition module is configured to construct and train an identification model based on a deep learning network, fuse the extracted action features, costume features and makeup features to obtain fused features, and input the fused features into the trained identification model, so that the identification model outputs a type of opera performance.

[0025] Compared with the prior art, the present application has the following advantages:

[0026] The present application is specially designed for opera videos, and the key frame selection is combined with the rhythm features of opera performance to avoid interference of redundant frames; the extracted action, costume and makeup features are consistent with the performance characteristics of opera, thereby laying a foundation for accurate identification.

[0027] The identification model is constructed based on a deep learning network, and the advantages of convolutional neural networks and recurrent neural networks are fused, so that the learning and generalization capabilities are strong, and the opera video recognition can be adapted to different types of plays and different performance styles. BRIEF DESCRIPTION OF DRAWINGS

[0028] The accompanying drawings generally illustrate various embodiments of the present application and, together with the description given, serve to explain the principles of the present application. Wherever possible, the same reference numbers are used in the various drawings to refer to the same or similar parts. Such embodiments are illustrative rather than restrictive and are intended to provide examples of the present device or method. In the various drawings, like reference numbers refer to like parts throughout the several views. Such embodiments are illustrative rather than restrictive and are intended to provide examples of the present device or method.

[0029] Figure 1 The method flowchart of the present application is shown. DETAILED DESCRIPTION

[0030] It should be noted that the embodiments and features in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0031] As shown in Figure 1 The present application provides an opera performance recognition method based on opera videos, which includes the following steps:

[0032] Obtain an opera video, decode the opera video to obtain a video frame sequence, and select key frames from the video frame sequence by using a frame difference method combined with the rhythm features of opera performance.

[0033] Extract features from the selected key frames, and the extracted features include action features, costume features and makeup features.

[0034] An identification model is constructed based on a deep learning network and is trained; the extracted action features, costume features and makeup features are fused to obtain fused features; the fused features are input into the trained identification model, and the identification model outputs a type of opera performance

[0035] Embodiment 1

[0036] The video of the selected section of “The Drunken Beauty” is taken as a video to be processed, and the video is decoded by using an FFmpeg decoding tool to obtain a video frame sequence containing 12,000 frames.

[0037] The threshold value of the pixel difference between adjacent frames is set to 30, and the specific calculation method is as follows:

[0038] The difference values of the RGB three-channel pixel values of the adjacent two frames are calculated, and the average value of the three-channel difference values is taken as the single-frame pixel difference (formula: pixel difference = (|R1-R2|+|G1-G2|+|B1-B2|) / 3). When the pixel difference is greater than 30, the frame is marked as a candidate key frame, and 150 candidate key frames are obtained through preliminary screening.

[0039] Further screening is performed in combination with the rhythm characteristics of Peking opera “singing, reciting, acting and fighting”: the time interval threshold value of the candidate key frame is set to 1 second (that is, 25 frames), and the 150 candidate key frames are subjected to de-duplication processing: if the time interval of the continuous candidate key frames is less than 1 second, the frame with the largest pixel difference value is retained. Finally, 20 key frames are screened out.

[0040] Opera performance feature extraction

[0041] Action feature extraction

[0042] The 20 key frames are processed by using an OpenPose bone key point detection algorithm. The algorithm locates 18 bone key points (including the head vertex, neck, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles) through a pre-trained convolutional neural network, and outputs pixel coordinates (unit: pixel) with the top left corner of the image as the origin.

[0043] Taking the key frame of the “raising a cup to drink” action as an example, the right wrist key point coordinates, right elbow key point coordinates and right shoulder key point coordinates are extracted, and the following are calculated:

[0044] Bone vector: right shoulder-right elbow vector, right elbow-right wrist vector;

[0045] Joint angle: the right elbow joint angle is calculated by using the vector dot product formula.

[0046] Finally, the bone coordinate matrix, the bone vector and the joint angle matrix calculated based on the coordinates are extracted from the 20 key frames as the action feature set.

[0047] Clothing feature extraction

[0048] The key frame is processed by using Mask R-CNN image segmentation algorithm, and the pre-trained model is used to identify the "clothing" category, and the binary mask of the clothing area (the area with mask value 1 is clothing) is output. For the phoenix coronet and veil of Yang Guifei in "The Drunken Beauty", three sub-regions of veil main body, cloud shoulder and skirt are segmented.

[0049] Color feature extraction: the mean and variance of each sub-region RGB three channels are calculated respectively (3 channels x 2 parameters = 6 feature values). For example, the RGB mean of the main body of the veil is (250, 180, 50), and the variance is (15, 20, 10), which reflects the color feature of red as the main color with golden embroidery;

[0050] Texture feature extraction: the clothing area is converted into a gray scale image (gray value = 0.299R + 0.587G + 0.114B), and a 5x5 window is used to calculate the gray co-occurrence matrix (step length 1, angle 0°, 45°, 90°, 135°), and three core parameters (energy, contrast, entropy) are extracted (4 angles x 3 parameters = 12 feature values).

[0051] Feature combination: 6 color features and 12 texture features are spliced to form an 18-dimensional clothing feature vector.

[0052] Makeup feature extraction

[0053] The Dlib face feature point detection algorithm is used to locate 68 face feature points, among which the eyebrows, eyes and lips are the core makeup areas.

[0054] Shape feature extraction: Bezier curve fitting is performed on the 10 feature points of the eyebrow area to obtain parameters such as curvature and length of the eyebrow shape; the contour perimeter to area ratio is calculated for the 20 feature points of the lip area.

[0055] Color feature extraction: the mean value of the eye shadow range (with the eye center as the origin, extending outward by 10 pixels) of the eye area and the lip area in the HSV color space (H channel reflects hue, S channel reflects saturation) is extracted. For example, the H mean value of the eye shadow area is 30 (corresponding to orange red), and the S mean value of the lip area is 0.8 (high saturation reflects thick lip makeup);

[0056] Feature combination: 4 shape features and 4 color features are spliced to form an 8-dimensional makeup feature vector.

[0057] Model construction, training and performance recognition

[0058] Model construction

[0059] Constructing CNN-RNN hybrid model:

[0060] CNN part: 3 layers of convolutional layers (kernel size 3x3, stride 1) + 2 layers of pooling layers (2x2 max pooling) are used to extract spatial features from the input feature vector;

[0061] RNN part: 2 layers of LSTM networks (hidden layer dimension 64) are used to handle the temporal correlation of key frames (such as the action sequence of "raising a cup-drinking-dropping a cup");

[0062] Output layer: softmax activation function is used to output two classification results of "genre" (Peking Opera, Yueju Opera, etc.) and "role type" (qingyi, huadan, etc.).

[0063] Model training

[0064] Sample preparation: 1000 segments of Peking Opera videos (covering 30 classic plays such as "The Drunken Beauty of the Imperial Concubine" and "The King and the Concubine") are collected, and each video is labeled with "genre=Peking Opera" and role type (8 types such as qingyi and laosheng). The training set (700 segments), validation set (200 segments), and test set (100 segments) are divided according to the ratio of 7:2:1;

[0065] Feature preprocessing: standardize the action (36 dimensions), costume (18 dimensions), and makeup (8 dimensions) features of all samples (mean 0, variance 1) to eliminate dimensional differences;

[0066] Training parameters: Adam optimizer (learning rate 0.001) is used, batch size 32, and iteration 50 rounds. When the validation set accuracy does not improve for 5 consecutive rounds, use learning rate decay (decay factor 0.5);

[0067] Convergence criteria: when the validation set accuracy reaches 92.5% (exceeding the preset threshold of 90%) at the 32nd round of training, stop training.

[0068] Performance identification process

[0069] Feature fusion: after standardizing the action, costume, and makeup features of the key frames of "The Drunken Beauty of the Imperial Concubine", use feature splicing to form a 36+18+8=62-dimensional fusion feature vector, and form a 20x62 feature matrix according to the key frame order;

[0070] Model inference: input the feature matrix into the trained model, extract spatial features such as "phoenix crown and cloud veil" costume texture and "willow leaf eyebrow" makeup shape through CNN, and capture action temporal features such as "drunken step shaking" through LSTM;

[0071] The recognition result is that the model outputs "genre = Beijing opera" (confidence 98.2%) and "role type = qingyi" (confidence 96.7%), which is consistent with the artificial labeling result.

[0072] The above merely describes preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can make equivalent replacements or changes within the technical range disclosed by the present application according to the technical solution and inventive concept of the present application, which should be covered within the protection scope of the present application.

Claims

1.A method for identifying a performance of a traditional opera based on a video of the traditional opera, characterized in that, The method comprises the following steps: Obtaining a video of a traditional Chinese opera, decoding the video to obtain a sequence of video frames, and screening key frames from the sequence of video frames by using a frame difference method in combination with rhythm characteristics of performances of the traditional Chinese opera; Extracting features from the screened key frames, wherein the extracted features include action features, costume features, and makeup features; Constructing and training a recognition model based on a deep learning network, fusing the extracted action features, costume features, and makeup features to obtain fused features, and inputting the fused features into the trained recognition model to output a type of performance of the traditional Chinese opera. 2.The opera performance identification method based on opera video according to claim 1, characterized in that, The specific process of screening the key frames by using the frame difference method in combination with the rhythm characteristics of performances of the traditional Chinese opera comprises: Calculating pixel difference values of adjacent two frames in the sequence of video frames, marking a frame as a candidate key frame when the pixel difference value exceeds a set threshold, and screening the candidate key frames in combination with rhythm characteristics of singing, reciting, acting, and fighting in performances of the traditional Chinese opera to retain key frames. 3.The opera performance identification method based on opera video according to claim 1, characterized in that, The process of extracting the action features comprises: 4.The opera performance identification method based on opera video according to claim 1, characterized in that, detecting skeletal key points of a character in a key frame by using a skeletal key point detection algorithm to obtain skeletal key point coordinates, and calculating skeletal vectors and joint angles based on the skeletal key point coordinates, wherein the skeletal vectors and joint angles are taken as the action features. 5.The opera performance identification method based on opera video according to claim 1, characterized in that, The process of extracting the costume features comprises: 6.The opera performance identification method based on opera video according to claim 1, characterized in that, segmenting a costume region of the character in the key frame by using an image segmentation algorithm, extracting color features and texture features from the segmented costume region, and combining the color features and the texture features as the costume features. The process of extracting the makeup features comprises: positioning facial feature points of the character in the key frame by using a facial feature point detection algorithm, determining a makeup region based on the facial feature points, extracting features of the makeup region, including shape features and color features of the makeup, and taking the shape features and the color features of the makeup as the makeup features. 7.The opera performance recognition method based on opera video according to claim 1, characterized in that, The specific process of constructing and training the recognition model based on the deep learning network comprises: constructing a recognition model combining a convolutional neural network and a recurrent neural network, and collecting samples of the traditional Chinese opera with labels of types of performances of the traditional Chinese opera; 8.A system for identifying a performance of a traditional opera based on a video of the traditional opera, characterized in that, inputting sample fused features and corresponding labels into an initial model, adjusting parameters by back propagation until an identification accuracy of the model on a verification set reaches a preset threshold. The fusion process of the action features, the costume features, and the makeup features comprises: performing feature vectorization processing on the action features, the costume features, and the makeup features, and fusing the features by using a feature splicing or feature attention mechanism to form a unified fused feature vector. The system comprises: a video preprocessing module configured to obtain a video of a traditional Chinese opera, decode the video to obtain a sequence of video frames, and screen key frames from the sequence of video frames by using a frame difference method in combination with rhythm characteristics of performances of the traditional Chinese opera; a performance feature extraction module configured to extract features from the screened key frames, wherein the extracted features include action features, costume features, and makeup features; a performance identification module configured to construct and train a recognition model based on a deep learning network, fuse the extracted action features, costume features, and makeup features to obtain fused features, and input the fused features into the trained recognition model to output a type of performance of the traditional Chinese opera.

Images