A quatrains video annotation method based on a gene learning model

Abstract

The application discloses a kind of art video annotation methods based on gene learning model, the method includes the following steps: S1, the key frame of video is extracted, the subscript of key frame in all frames of art video is recorded, the time of current key frame is calculated, and key frame is input into feature extraction network and is extracted feature;S2, the audio of art video is separated, and audio feature is extracted;S3, the visual feature of each key frame and audio feature are synchronously aggregated, and the multi-modal fusion feature of each frame is obtained, the multi-modal fusion feature of all key frames is time-series aggregated, and video-level time-series aggregated feature is obtained;S4, according to the video-level time-series aggregated feature obtained, inference is carried out, and video-level feature is obtained, according to the key frame feature obtained, video frame-level feature is obtained;The application can realize the multi-granularity annotation of traditional art video, is helpful to subsequent management, retrieval of the video, is conducive to the spread of Chinese culture.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method for annotating traditional Chinese folk art videos based on a gene learning model. Background Technology

[0002] Traditional folk arts primarily rely on live performances or instrumental music by living entities (people). They transmit knowledge, values, and emotions through the spiritual and cultural space created by the performers and local customs and rituals, making them invaluable and non-renewable resources. Therefore, certain protective measures are necessary. Recording folk arts, labeling the videos, and categorizing and storing them are crucial means of protecting traditional folk arts. Digital technology can support the protection, inheritance, and dissemination of intangible cultural heritage such as folk arts. How to efficiently and standardizedly manage these culturally related video resources and their corresponding metadata has become an urgent issue.

[0003] The shortcomings of existing technology:

[0004] Currently, the annotation of traditional Chinese folk art videos faces many challenges. First, there is a lack of usable datasets for annotating traditional Chinese folk art videos, and the number of experts in the relevant field is small, resulting in low accuracy for annotation by ordinary people and low efficiency for manual annotation. Second, because traditional Chinese folk art videos contain a large amount of information with traditional cultural semantics and are highly specialized, the performers' forms, movements, clothing, and even the background music all have unique artistic characteristics, making it difficult for ordinary annotation algorithms and platforms to efficiently annotate traditional traditional Chinese folk art videos. Summary of the Invention

[0005] The purpose of this invention is to provide a method for annotating traditional Chinese folk art videos based on a gene learning model, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for annotating traditional Chinese folk art videos based on a gene learning model, the method comprising the following steps:

[0007] S1. Extract key frames from the video using techniques such as shot demarcation and key frame extraction, record the index of the key frame in all frames of the folk art video, calculate the time of the current key frame, and input the key frame into the feature extraction network for feature extraction.

[0008] S2. Extract corresponding audio features: Separate the audio of the folk art video, then extract 1 second of audio based on the time corresponding to the key frame, and input the audio segment into the feature extraction network for audio feature extraction;

[0009] S3. Multimodal temporal feature aggregation: The visual and audio features of each key frame are aggregated synchronously to obtain the multimodal fusion features of each frame. Then, the multimodal fusion features of all key frames are aggregated temporally to obtain video-level temporal aggregated features.

[0010] S4. Multi-granularity annotation: Infer video-level features based on the obtained video-level temporal aggregated features, and infer video frame-level features based on the obtained keyframe features.

[0011] Preferably, step S1 specifically includes the following steps:

[0012] a1. Input the video of the folk art to be labeled into the TransNetV2 shot delimitation algorithm, compress each frame of the video to a uniform small size of 48*27*3, and take every 100 frames as a segment. Extract the color histogram HIST and learnable similarity matrix of each frame, and return the probability of whether each frame of the video is a segment boundary frame. If the probability exceeds the threshold (the default is 0.5), it is judged as a shot boundary frame. Finally, the shot delimitation list of the video is obtained. Each item in the list represents a complete shot and includes the start frame index and end frame index of the shot.

[0013] a2. Video keyframe extraction: For each frame, the color, brightness, and motion information of the image's color saliency map are extracted. The image is measured from multiple different directions to comprehensively and accurately filter out information that precisely describes the rapid changes in the video frame. By calculating the mean square error (MSE) between two adjacent frames, the extreme points on the MSE curve can be used as candidate keyframes. By calculating the difference between the candidate keyframes and the preceding and following frames, the true keyframes are finally selected.

[0014] a3. Video keyframe feature extraction: Visual features of video frames are extracted using ViT / B16 pre-trained in CLIP.

[0015] Preferably, step S2 specifically includes the following steps:

[0016] b1. Separate the audio from the video. Determine the corresponding audio position based on the index of the recorded video keyframe and the video frame rate, and extract the corresponding 1-second audio segment.

[0017] b2. First, manually extract audio features from each audio segment, and then input them into the audio feature extraction network to obtain deep features.

[0018] Preferably, step S3 specifically includes the following steps:

[0019] c1. Combine the visual and audio features of video frames in the same dimension and perform dimensional transformation;

[0020] c2. After splicing, the features are subjected to attention fusion and then spliced ​​with the original visual and audio features to obtain multimodal synchronous fusion features;

[0021] c3. Input the multimodal synchronous fusion features of each frame into the temporal feature extraction algorithm to obtain video-level temporal aggregated features.

[0022] Preferably, the multi-granularity annotation includes a video-level annotation branch and a frame-level annotation branch, and step S4 specifically includes the following steps:

[0023] d1. Input the video-level multimodal temporal aggregation features obtained through the multimodal temporal aggregation feature step into the classification layer for prediction to obtain the coarse-grained video-level annotation results of the video;

[0024] d2. Input the keyframes of the folk art video obtained in the keyframe extraction step into the target detection framework for prediction to obtain fine-grained video frame-level annotation results of the video;

[0025] d3. Merge the coarse-grained video-level annotation results with the fine-grained video annotation results and remove duplicates to obtain the multi-grained multimodal annotation results.

[0026] Preferably, the keyframe extraction step a2 specifically includes the following steps:

[0027] e1. The dynamic characteristics of each video frame within a video shot are derived from the inter-frame difference between two adjacent frames:

[0028] F M (x,y)=I t (x,y)-I t-1 (x,y)

[0029] Where x and y represent the coordinates of the pixel, and t represents time t;

[0030] e2. The luminance characteristics of a video frame can be viewed as the average value of its R, G, and B channels at the corresponding position:

[0031]

[0032] Where x and y represent the coordinates of the pixel, and t represents time t;

[0033] e3. After obtaining the saliency map S of video frame I, multiply the saliency map S with the R, G, and B channels of the original video frame I respectively, and then merge the three channels to obtain the color saliency map R. The red / green color features F of the color saliency map in the CIELab color space are then obtained. Ra (x,y) and Yellow / Blue F Rb (x,y) color characteristics:

[0034]

[0035] Where 'a' represents the red / green intensity, the closer to -127 the greener it is, and the closer to 127 the redder it is; 'b' represents the yellow / blue intensity, the closer to -127 the bluer it is, and the closer to 127 the yellower it is; 'maxa' ('mina') represent the maximum (minimum) values ​​of the red and green channels in the current image, and 'maxb' ('mina') represent the maximum (minimum) values ​​of the yellow and blue channels in the current image, respectively.

[0036] e4. The static features of the original image, the color features of the color saliency image, and the motion features of the original image are fused to obtain the features L(x,y) of the video frame.

[0037] e5. Use the squared mean of the Euclidean distance as the similarity metric. The smaller the mean, the higher the similarity between two adjacent frames. For the input video frame sequence, use the following formula to measure similarity:

[0038]

[0039] Where i represents the index of the current frame, and m and n represent the width and height of the video frame, respectively.

[0040] The maximum value of the similarity curve indicates that the similarity difference between two consecutive video frames is too large. The two adjacent video frames corresponding to these maximum values ​​are added to the candidate frame sequence. Then, it is determined whether the difference between the current maximum value and the previous and next points is greater than the global mean.

[0041]

[0042] Where d′ and d″ represent the differences between the maximum point and the previous and next points, respectively. If they are greater than the global mean, the next frame of the two frames corresponding to the extreme point is taken as the final keyframe, and the index of the keyframe is recorded.

[0043] Preferably, the saliency map is obtained by the following steps:

[0044] f1. Using the CIELab color space to simulate the dual-color contrast system in the human cerebral cortex, modeling the human response to warm colors and insensitivity to cool colors:

[0045]

[0046]

[0047] Where x represents the coordinate vector of an image pixel, i.e., x = (x, y), where x and y are the horizontal and vertical coordinates of the image, respectively. a (x) and I b (x) represent the values ​​of the red-green channel a and the yellow-blue channel b, respectively. an and I bn These are the values ​​of the red-green and yellow-blue channels after regularization, where σC For variance;

[0048] f2. Use Gaussian mapping to model the part of the image that people tend to pay attention to:

[0049]

[0050] Where c is the coordinate vector of the image center point, σ D 2 It is variance;

[0051] f3. Using bandpass filtering to model salient objects in the human eye, the transfer function g(x) of the log-Gabor bandpass filter cannot be directly expressed mathematically at the origin. Therefore, it needs to be obtained through Fourier transform to obtain G(u). Hence, the transfer function g(x) (x=(x,y)∈R) 2 The converted mathematical formula is as follows:

[0052]

[0053] Where ω0 is the center frequency of the log-Gabor filter, u=(u,v) are frequency domain coordinates, and σ F The parameters for controlling bandwidth; the I obtained from the CIELab space L (x), I a (x) and I b Convolving (x) with the transfer function yields S. F (x):

[0054] S F (x)=((I L *g) 2 +(I a *g) 2 +(I b *g) 2 ) 12 (x)

[0055] f4, The saliency image of the video frame is determined by the S C (x), S D (x), S F Multiplying (x) together yields:

[0056] SDSP(x)=S D (x)·S C (x)·S F (x)

[0057] Preferably, the steps of manual audio feature extraction are as follows:

[0058] g1. Perform preprocessing operations on the audio segment, including: pre-emphasis, frame segmentation, and windowing;

[0059] g2. Discretize the audio frame using Fast Fourier Transform, transform it from the time domain to the frequency domain, obtain the corresponding energy distribution, and then take the modulus of the energy distribution and perform a quadratic operation to obtain the spectral line energy.

[0060] g3. Use the Mel filter to filter the audio signal, and multiply the Mel frequency by the amplitude spectrum and take the logarithm to obtain the Fbank manual audio features.

[0061] Preferably, the multi-granularity annotation includes coarse-grained annotation of the video and fine-grained annotation of the video from multiple dimensions to supplement the semantic information missing from the video-level annotation.

[0062] Preferably, the coarse-grained labeling includes labels such as water sleeve technique, hat wing technique, and single-person technique.

[0063] Compared with the prior art, the beneficial effects of the present invention are:

[0064] This paper presents a gene learning model-based annotation method for traditional Chinese performing arts videos. It mimics the behavior of human genes guiding humans to process multimodal information simultaneously, eliminating manual labor and ensuring accurate annotation. Simultaneously, it utilizes the audio and visual information of the videos to achieve synchronous fusion of multimodal features. The multimodal features are then subjected to temporal feature extraction to obtain coarse-grained video-level features. Keyframes are input into an improved target recognition network to obtain fine-grained frame-level features, thereby achieving automated annotation of traditional Chinese performing arts videos and improving annotation efficiency. Attached Figure Description

[0065] Figure 1 This is a flowchart of the present invention;

[0066] Figure 2 This is a schematic diagram of the video annotation process for traditional Chinese performing arts based on a gene learning model, as described in this invention.

[0067] Figure 3 This is a schematic diagram of the multimodal synchronous fusion module of the present invention;

[0068] Figure 4 This is a diagram of the improved frame-level annotation network architecture of the present invention;

[0069] Figure 5 This is a diagram illustrating the annotation effect of the traditional Chinese folk art video according to the present invention; Detailed Implementation

[0070] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] Example

[0072] Please see Figure 1-5 As shown, the present invention provides a technical solution for a method of annotating traditional Chinese folk art videos based on a gene learning model: This method includes the following steps:

[0073] S1. Extract key frames from the video using techniques such as shot demarcation and key frame extraction, record the index of the key frame in all frames of the folk art video, calculate the time of the current key frame, and input the key frame into the feature extraction network for feature extraction.

[0074] a1. Input the video of the folk art to be labeled into the TransNetV2 shot delimitation algorithm, compress each frame of the video to a uniform small size of 48*27*3, and take every 100 frames as a segment. Extract the color histogram HIST and learnable similarity matrix of each frame, and return the probability of whether each frame of the video is a segment boundary frame. If the probability exceeds the threshold (the default is 0.5), it is judged as a shot boundary frame. Finally, the shot delimitation list of the video is obtained. Each item in the list represents a complete shot and includes the start frame index and end frame index of the shot.

[0075] a2. Video keyframe extraction: For each frame, the color, brightness, and motion information of the image's color saliency map are extracted. The image is measured from multiple different directions to comprehensively and accurately filter out information that precisely describes the rapid changes in the video frame. By calculating the mean square error (MSE) between two adjacent frames, the extreme points on the MSE curve can be used as candidate keyframes. By calculating the difference between the candidate keyframes and the preceding and following frames, the true keyframes are finally selected.

[0076] a3. Video keyframe feature extraction: Visual features of video frames are extracted using ViT / B16 pre-trained in CLIP.

[0077] S2. Extract corresponding audio features: Separate the audio of the folk art video, then extract 1 second of audio based on the time corresponding to the key frame, and input the audio segment into the feature extraction network for audio feature extraction;

[0078] b1. Separate the audio from the video. Determine the corresponding audio position based on the index of the recorded video keyframe and the video frame rate, and extract the corresponding 1-second audio segment.

[0079] b2. First, manually extract audio features from each audio segment, and then input them into the audio feature extraction network to obtain deep features;

[0080] S3. Multimodal temporal feature aggregation: The visual and audio features of each key frame are aggregated synchronously to obtain the multimodal fusion features of each frame. Then, the multimodal fusion features of all key frames are aggregated temporally to obtain video-level temporal aggregated features.

[0081] c1. Combine the visual and audio features of video frames in the same dimension and perform dimensional transformation;

[0082] c2. After splicing, the features are subjected to attention fusion and then spliced ​​with the original visual and audio features to obtain multimodal synchronous fusion features;

[0083] c3. Input the multimodal synchronous fusion features of each frame into the temporal feature extraction algorithm to obtain video-level temporal aggregated features;

[0084] S4. Multi-granularity annotation: Infer video-level features based on the obtained video-level temporal aggregation features, and infer video frame-level features based on the obtained keyframe features.

[0085] c1. Combine the visual and audio features of video frames in the same dimension and perform dimensional transformation;

[0086] c2. After splicing, the features are subjected to attention fusion and then spliced ​​with the original visual and audio features to obtain multimodal synchronous fusion features;

[0087] c3. Input the multimodal synchronous fusion features of each frame into the temporal feature extraction algorithm to obtain video-level temporal aggregated features;

[0088] Furthermore, the keyframe extraction step in step a2 includes:

[0089] e1. The dynamic characteristics of each video frame within a video shot are derived from the inter-frame difference between two adjacent frames:

[0090] F M (x,y)=I t (x,y)-I t-1 (x,y)

[0091] Where x and y represent the coordinates of the pixel, and t represents time t;

[0092] e2. The luminance characteristics of a video frame can be viewed as the average value of its R, G, and B channels at the corresponding position:

[0093]

[0094] Where x and y represent the coordinates of the pixel, and t represents time t;

[0095] e3. After obtaining the saliency map S of video frame I, multiply the saliency map S with the R, G, and B channels of the original video frame I respectively, and then merge the three channels to obtain the color saliency map R. The red / green color features F of the color saliency map in the CIELab color space are then obtained. Ra (x,y) and Yellow / Blue F Rb (x,y) color characteristics:

[0096]

[0097] Where 'a' represents the red / green intensity, the closer to -127 the greener it is, and the closer to 127 the redder it is; 'b' represents the yellow / blue intensity, the closer to -127 the bluer it is, and the closer to 127 the yellower it is; 'maxa' ('mina') represent the maximum (minimum) values ​​of the red and green channels in the current image, and 'maxb' ('mina') represent the maximum (minimum) values ​​of the yellow and blue channels in the current image, respectively.

[0098] e4. The static features of the original image, the color features of the color saliency image, and the motion features of the original image are fused to obtain the features L(x,y) of the video frame.

[0099] e5. Use the squared mean of the Euclidean distance as the similarity metric. The smaller the mean, the higher the similarity between two adjacent frames. For the input video frame sequence, use the following formula to measure similarity:

[0100]

[0101] Where i represents the index of the current frame, and m and n represent the width and height of the video frame, respectively;

[0102] The maximum value of the similarity curve indicates that the similarity difference between two consecutive video frames is too large. The two adjacent video frames corresponding to these maximum values ​​are added to the candidate frame sequence. Then, it is determined whether the difference between the current maximum value and the previous and next points is greater than the global mean.

[0103]

[0104] Where d′ and d″ represent the differences between the maximum point and the previous and next points, respectively. If they are greater than the global mean, the next frame of the two frames corresponding to the extreme point is taken as the final keyframe, and the index of the keyframe is recorded.

[0105] Furthermore, the saliency map is obtained through the following steps:

[0106] f1. Using the CIELab color space to simulate the dual-color contrast system in the human cerebral cortex, modeling the human response to warm colors and insensitivity to cool colors:

[0107]

[0108]

[0109] Where x represents the coordinate vector of an image pixel, i.e., x = (x, y), where x and y are the horizontal and vertical coordinates of the image, respectively. a (x) and I b (x) represent the values ​​of the red-green channel a and the yellow-blue channel b, respectively. an and I bn These are the values ​​of the red-green and yellow-blue channels after regularization, where σ C For variance;

[0110] f2. Use Gaussian mapping to model the part of the image that people tend to pay attention to:

[0111]

[0112] Where c is the coordinate vector of the image center point, σ D 2 It is variance;

[0113] f3. Using bandpass filtering to model salient objects in the human eye, the transfer function g(x) of the log-Gabor bandpass filter cannot be directly expressed mathematically at the origin. Therefore, it needs to be obtained through Fourier transform to obtain G(u). Hence, the transfer function g(x) (x=(x,y)∈R) 2 The converted mathematical formula is as follows:

[0114]

[0115] Where ω0 is the center frequency of the log-Gabor filter, u=(u,v) are frequency domain coordinates, and σ F The parameters for controlling bandwidth; the I obtained from the CIELab space L (x), I a (x) and I b Convolving (x) with the transfer function yields S. F (x):

[0116] S F (x)=((I L *g) 2 +(I a *g) 2 +(I b *g) 2 )12 (x)

[0117] f4, The saliency image of the video frame is determined by the S C (x), S D (x), S F Multiplying (x) together yields:

[0118] SDSP(x)=S D (x)·S C (x)·S F (x)

[0119] Furthermore, the specific steps for manual audio feature extraction are as follows:

[0120] g1. Perform preprocessing operations on the audio segment, including: pre-emphasis, frame segmentation, and windowing;

[0121] g2. Discretize the audio frame using Fast Fourier Transform, transform it from the time domain to the frequency domain, obtain the corresponding energy distribution, and then take the modulus of the energy distribution and perform a quadratic operation to obtain the spectral line energy.

[0122] g3. Use the Mel filter to filter the audio signal, and multiply the Mel frequency by the amplitude spectrum and take the logarithm to obtain the Fbank manual audio features.

[0123] Furthermore, multi-granularity annotation includes coarse-grained annotation of the video and fine-grained annotation of the video from multiple dimensions to supplement the semantic information missing in the video-level annotation. Coarse-grained annotation includes tags such as water sleeve skills, hat wing skills, and single person fighting.

[0124] Furthermore, this invention provides a folk art video annotation algorithm based on a gene learning model. This algorithm mimics human gene-guided behavior of simultaneously utilizing multimodal information, simultaneously fusing audio and visual modal information to obtain coarse-grained annotations. The fine-grained branch uses video keyframes for target detection to obtain fine-grained annotations in three dimensions: role type, costume, and facial makeup. This mimics gene-guided human behavior of simultaneously utilizing audio and visual information, i.e., simultaneously aligning data from different modalities, eliminating manual intervention and ensuring accurate annotation. Simultaneously, it utilizes the audio and visual information of the folk art video for synchronous fusion to obtain multimodal fusion features. Temporal feature extraction of these multimodal features yields coarse-grained video-level features, and keyframes are input into an improved target recognition network to obtain fine-grained frame-level features. This achieves automated annotation of folk art videos, improving annotation efficiency.

[0125] The principle of the folk art video annotation method based on gene learning model of the present invention is as follows: The folk art video to be annotated is input into the TransNetV2 shot delimitation algorithm, each frame of the video is compressed to a uniform small size of 48*27*3, every 100 frames is a segment, and the color histogram HIST and learnable similarity matrix of each frame are extracted. The probability of whether each frame of the video is a segment boundary frame is returned. If it exceeds the threshold (the default is 0.5), it is judged as a shot boundary frame. Finally, the shot delimitation list of the video is obtained. Each item in the list represents a complete shot and includes the start frame index and end frame index of the shot.

[0126] Then, for each frame, color, brightness, and motion information are extracted from the image's color saliency map. The image is measured comprehensively from multiple directions to accurately filter out information that precisely describes the rapid changes in the video frame. By calculating the mean square error (MSE) between two adjacent frames, extreme points on the MSE curve are used as candidate keyframes. The differences between these candidate keyframes and the preceding and following frames are calculated to ultimately select the true keyframes. The motion information is the dynamic characteristics of each video frame within the video shot, derived from the inter-frame difference between two adjacent frames, as shown in the following formula:

[0127] F M (x,y)=I t (x,y)-I t-1 (x,y)

[0128] Where x and y represent the coordinates of the pixel, and t represents time t; the brightness feature of a video frame can be seen as the average value of its R, G, and B channels at the corresponding position:

[0129]

[0130] Where x and y represent the coordinates of the pixels, and t represents time t; the saliency map of the image consists of 3 parts S C (x), S D (x), S F (x) Results: Using the CIELab color space to simulate the dual-color contrast system in the human cerebral cortex, modeling the human response to warm colors and insensitivity to cool colors:

[0131]

[0132]

[0133] Where 'a' represents red / green intensity, the closer to -127 the greener, the closer to 127 the redder; 'b' represents yellow / blue intensity, the closer to -127 the bluer, the closer to 127 the yellower; 'maxa' ('mina') represent the maximum (minimum) values ​​of the red and green channels in the current image, and 'maxb' ('mina') represent the maximum (minimum) values ​​of the yellow and blue channels in the current image. 'x' represents the coordinate vector of the image pixels, i.e., x = (x, y), where x and y are the horizontal and vertical coordinates of the image, respectively. a (x) and I b (x) represent the values ​​of the red-green channel a and the yellow-blue channel b, respectively. an and I bn These are the values ​​of the red-green and yellow-blue channels after regularization, where σ C Let Variance be the variance.

[0134] Using Gaussian mapping to model the part of the image that people tend to pay attention to:

[0135]

[0136] Where c is the coordinate vector of the image center point, σ D 2 The variance is used to model salient objects in the human eye using a bandpass filter. The transfer function g(x) of the log-Gabor bandpass filter cannot be directly expressed mathematically at the origin, so it needs to be obtained through Fourier transform to G(u). Therefore, the transfer function g(x) (x=(x,y)∈R) 2 The converted mathematical formula is as follows:

[0137]

[0138] Where ω0 is the center frequency of the log-Gabor filter, u=(u,v) are frequency domain coordinates, and σ F The parameters for controlling bandwidth; the I obtained from the CIELab space L (x), I a (x) and I b Convolving (x) with the transfer function yields S. F (x):

[0139] S F (x)=((I L *g) 2 +(I a *g) 2 +(I b *g) 2 ) 12 (x)

[0140] The saliency image of the video frame is determined by the S C(x), S D (x), S F Multiplying (x) together yields:

[0141] SDSP(x)=S D (x)·S C (x)·S F (x)

[0142] After obtaining the saliency map S of video frame I, multiply the saliency map S with the R, G, and B channels of the original video frame I respectively, and then merge the three channels to obtain the color saliency map R. The red / green color features F of the color saliency map in the CIELab color space are then obtained. Ra (x,y) and Yellow / Blue F Rb (x,y) color characteristics.

[0143]

[0144] The static features of the original image, the color features of the color saliency image, and the motion features of the original image are fused. First, the features are normalized, and then the four features are concatenated column-wise to obtain the feature vector matrix L(x,y) of the video frame. Finally, the mean squared Euclidean distance is used as the similarity. The smaller the mean, the higher the similarity between adjacent frames. For the input video frame sequence, the following formula is used to measure the similarity and obtain the similarity curve:

[0145]

[0146] Where i represents the index of the current frame, and m and n represent the width and height of the video frame, respectively;

[0147] The maximum value of the similarity curve indicates that the similarity difference between two consecutive video frames is too large. The two adjacent video frames corresponding to these maximum values ​​are added to the candidate frame sequence. Then, it is determined whether the difference between the current maximum value and the previous and next points is greater than the global mean.

[0148]

[0149] Where d′ and d″ represent the differences between the maximum point and the previous and next points, respectively. If they are greater than the global mean, the next frame of the two frames corresponding to the maximum point is taken as the final keyframe, and the index of the keyframe is recorded. Finally, the visual features of the video frames are extracted using ViT / B16 pre-trained in CLIP.

[0150] The audio is then processed. First, the audio from the video is separated. Based on the index of the recorded video keyframes and the video frame rate, the corresponding audio positions are determined, and a 1-second audio segment is extracted. Each audio segment undergoes manual audio feature extraction (Fbank), including pre-emphasis, framing, and windowing. The audio frames are discretized using a Fast Fourier Transform (FFT) to transform them from the time domain to the frequency domain, obtaining the corresponding energy distribution. The modulus of the energy distribution is then taken and squared to obtain the spectral energy. Next, a Mel filter is used to filter the audio signal, and the logarithm of the Mel frequency multiplied by the amplitude spectrum is obtained to get the Fbank manual audio features. Finally, these features are input into the audio feature extraction network PANNs to obtain 512-dimensional audio depth features.

[0151] Video keyframes are input into the pre-trained ViT B / 16 model of CLIP to extract 512-dimensional visual features, which are then concatenated with audio depth features and subjected to dimensionality transformation. These features are then input into a residual attention fusion bottleneck block for initial fusion. Finally, the fused result is concatenated with both visual and audio depth features to obtain multimodal synchronous fusion features. These multimodal synchronous fusion features are then input into the temporal feature extraction algorithm TCN to obtain video-level temporal aggregated features, which are then filtered out for redundant information using an attention mechanism. The video-level multimodal temporal aggregated features obtained through this step are then input into a classification layer for prediction, resulting in coarse-grained video-level annotations. The keyframes of the folk art video obtained in the keyframe extraction step are input into the object detection framework for prediction, resulting in fine-grained video frame-level annotation results. As a further improvement of this invention, the fine-grained annotation method in this annotation method refers to the improved YOLOv8 object detection algorithm yolov8-AFPN-BAM. The original algorithm's Neck architecture is improved to a Progressive Feature Pyramid Network AFPN. Compared to FPN, AFPN avoids large semantic gaps between non-adjacent layers by fusing two adjacent low-level features and then progressively incorporating high-level features into the fusion process, thus enabling more efficient multi-scale feature fusion operations. Simultaneously, a BAM module is added to the SPPF module to enhance the filtering capability of effective information from the feature extraction network, transforming it into the SPPF-BAM module.

[0152] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for annotating traditional Chinese folk art videos based on a gene learning model, characterized in that: This method includes the following steps: S1. Extract keyframes: Extract keyframes from the video using shot demarcation and keyframe extraction techniques, record the index of the keyframe in all frames of the folk art video, calculate the time of the current keyframe, and input the keyframe into the feature extraction network for feature extraction. Step S1 specifically includes the following steps: a1. Input the video of the folk art to be labeled into the TransNetV2 shot delimitation algorithm, compress each frame of the video to a uniform small size of 48*27*3, take every 100 frames as a segment, and extract the color histogram HIST and learnable similarity matrix of each frame. Return the probability of whether each frame of the video is a segment boundary frame. If it exceeds the threshold, it is judged as a shot boundary frame. The video shot delimitation list is obtained. Each item in the list represents a complete shot, including the start frame index and end frame index of the shot. a2. Video keyframe extraction: For each frame, the color, brightness, and motion information of the image's color saliency map are extracted. The image is measured from multiple different directions to comprehensively and accurately filter out information that precisely describes the rapid changes in the video frame. By calculating the mean square error (MSE) between two adjacent frames, the extreme points on the MSE curve are taken as candidate keyframes. By calculating the difference between the candidate keyframes and the preceding and following frames, the true keyframes are selected. a3. Video keyframe feature extraction: Visual features of video frames are extracted using ViT / B16 pre-trained in CLIP. S2. Extract corresponding audio features: Separate the audio from the folk art video, extract the audio from 1 to 3 seconds according to the time corresponding to the key frame, and input the audio segment into the feature extraction network for audio feature extraction; S3. Multimodal temporal feature aggregation: The visual and audio features of each key frame are aggregated synchronously to obtain the multimodal fusion features of each frame. The multimodal fusion features of all key frames are then aggregated temporally to obtain video-level temporal aggregated features. Step S3 specifically includes the following steps: c1. Combine the visual and audio features of video frames in the same dimension and perform dimensional transformation; c2. After splicing, the features are subjected to attention fusion and then spliced ​​with the original visual and audio features to obtain multimodal synchronous fusion features; c3. Input the multimodal synchronous fusion features of each frame into the temporal feature extraction algorithm to obtain video-level temporal aggregated features. S4. Multi-granularity annotation: Infer video-level features based on the obtained video-level temporal aggregation features, and infer video frame-level features based on the obtained keyframe features. The multi-granularity annotation includes video-level annotation branches and frame-level annotation branches. Step S4 specifically includes the following steps: d1. Input the video-level multimodal temporal aggregation features obtained through step S3 into the classification layer for prediction to obtain the coarse-grained video-level annotation results of the video; d2. Input the keyframes of the folk art video obtained in the keyframe extraction step into the target detection framework for prediction to obtain fine-grained video frame-level annotation results of the video; d3. Merge the coarse-grained video-level annotation results with the fine-grained video frame-level annotation results and remove duplicates to obtain the multi-grained multimodal annotation results.

2. The method for annotating traditional Chinese folk art videos based on a gene learning model according to claim 1, characterized in that: Step S2 specifically includes the following steps: b1. Separate the audio from the video. Determine the corresponding audio position based on the index of the recorded video keyframe and the video frame rate, and extract the corresponding 1-second audio segment. b2. First, manually extract audio features from each audio segment, and then input them into the audio feature extraction network to obtain deep features.

3. The method for annotating traditional Chinese folk art videos based on a gene learning model according to claim 1, characterized in that: The keyframe extraction step a2 specifically includes the following steps: e1. The dynamic characteristics of each video frame within a video shot are derived from the inter-frame difference between two adjacent frames: Where x and y represent the coordinates of the pixel, and t represents time t; e2. The luminance characteristics of a video frame can be viewed as the average value of its R, G, and B channels at the corresponding position: Where x and y represent the coordinates of the pixel, and t represents time t; e3. After obtaining the saliency map S of video frame I, multiply the saliency map S with the R, G, and B channels of the original video frame I respectively, and then merge the three channels to obtain the color saliency map R. The red / green color features of the color saliency map in the CIELab color space are then obtained. Yellow / Blue Color characteristics: Where 'a' represents red / green intensity, the closer to -127 the greener, the closer to 127 the redder; 'b' represents yellow / blue intensity, the closer to -127 the bluer, the closer to 127 the yellower; 'maxa' ('mina') represent the maximum (minimum) values ​​of the red and green channels in the current image, and 'maxb' ('mina') represent the maximum (minimum) values ​​of the yellow and blue channels in the current image. e4. Fuse the static features of the original image, the color features of the color saliency image, and the motion features of the original image to obtain the features of the video frame. ; e5. Use the squared mean of the Euclidean distance as the similarity metric. The smaller the mean, the higher the similarity between two adjacent frames. For the input video frame sequence, use the following formula to measure similarity: in Represents the current frame index. and The width and height of the video frame are represented by the maximum values ​​of the similarity curve. The maximum values ​​of the similarity curves indicate a significant difference in similarity between consecutive video frames. The two adjacent video frames corresponding to these maximum values ​​are added to the candidate frame sequence. Then, it is determined whether the difference between the current maximum value and both the previous and next maximum values ​​is greater than the global mean. in and These represent the differences between the maximum point and the previous and next points, respectively. If they are both greater than the global mean, then the next frame of the two frames corresponding to the maximum point is taken as the final keyframe, and the index of the keyframe is recorded.

4. The method for annotating traditional Chinese folk art videos based on a gene learning model according to claim 3, characterized in that: The saliency map is obtained by the following steps: f1. Using the CIELab color space to simulate the dual-color contrast system in the human cerebral cortex, modeling the human response to warm colors and insensitivity to cool colors: in The coordinate vector representing the image pixels, i.e. , These are the horizontal and vertical coordinates of the image, respectively. and These represent the values ​​of the red-green channel (a) and the yellow-blue channel (b), respectively. and These are the values ​​of the red-green and yellow-blue channels after regularization, where... For variance; f2. Use Gaussian mapping to model the part of the image that people tend to pay attention to: in Let the coordinate vector be the center point of the image. It is variance; f3. Using bandpass filtering to model salient objects in the human eye, the transfer function of the log-Gabor bandpass filter. The expression at the origin cannot be directly represented using mathematical analytical formulas, so it needs to be obtained through Fourier transform. Therefore, the transfer function The converted mathematical formula is as follows: in It is the center frequency of the log-Gabor filter. For frequency domain coordinates, Parameters for controlling bandwidth; obtained from the CIELab color space. , and Convolution operation with transfer function to obtain : f4, The saliency image of the video frame is determined by the above. , , Multiplying them together gives: 。 5. The method for annotating traditional Chinese folk art videos based on a gene learning model according to claim 2, characterized in that: The specific steps for manual audio feature extraction are as follows: g1. Perform preprocessing operations on the audio segment, including: pre-emphasis, frame segmentation, and windowing; g2. Discretize the audio frame using Fast Fourier Transform, transform it from the time domain to the frequency domain, obtain the corresponding energy distribution, take the modulus of the energy distribution and perform a quadratic operation to obtain the spectral line energy. g3. Use the Mel filter to filter the audio signal, and multiply the Mel frequency by the amplitude spectrum and take the logarithm to obtain the Fbank manual audio features.

6. The method for annotating traditional Chinese folk art videos based on a gene learning model according to claim 1, characterized in that: The multi-granularity annotation includes coarse-grained annotation of the video and fine-grained annotation of the video from multiple dimensions to supplement the semantic information missing from the video-level annotation.

7. The method for annotating traditional Chinese folk art videos based on a gene learning model according to claim 6, characterized in that: The coarse-grained labeling includes water sleeve techniques, hat wing techniques, and single-person labeling.

Images