Characteristic value generation method and system based on video scene identification technology

Through the feature value generation method of video scene recognition technology, combined with machine learning algorithms, the problems of low efficiency of traditional video content recognition and poor accuracy of face recognition are solved, and efficient video scene recognition and content classification are achieved, which is suitable for a variety of applications Scene.

CN119027849APending Publication Date: 2024-11-26HANGZHOU ARCVIDEO TECHNOLOGY CO LTD
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202410890158.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-04
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Traditional video content recognition and classification methods are inefficient when facing a large amount of video data, and face recognition technology is greatly affected by factors such as lighting and angle, making it difficult to accurately identify video scenes and characters.

Method used

Using the feature value generation method based on video scene recognition technology, through frame extraction, face detection and tracking, the scene type is judged based on the proportion of faces in the picture, and the video is divided into time periods to extract feature values, using machine learning algorithms For example, the RNN model performs classification and recognition, and combines video preprocessing, scene type recognition, feature value extraction and classification recognition modules to improve the classification and retrieval efficiency of video content.

Benefits of technology

It realizes more accurate video scene recognition and content classification, avoids data tampering, improves recognition, is suitable for copyright protection and permission control, and enhances video surveillance and user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027849A_ABST
    Figure CN119027849A_ABST
Patent Text Reader

Abstract

The invention discloses a feature value generation method and system based on a video scene identification technology, and the method comprises the following steps: carrying out the frame extraction of an input video, so as to obtain all frames, carrying out the face detection of each frame, so as to obtain all faces in the frames, tracking each face to obtain position information of the face in the whole video; judging the scene type of the current picture according to the proportion of the human face in the picture; segmenting the video into a plurality of time periods, and extracting a feature value of each time period according to the scene type and the duration time of the picture in each time period; and taking the characteristic values of all the time periods as input, performing classification and identification by using a machine learning algorithm, and classifying the videos into different types.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of video processing, and in particular relates to a method and system for generating feature values ​​based on video scene recognition technology. Background Art

[0002] In the context of the current development of digital video technology, video content recognition and classification has become a hot research direction. This technology can be applied to many fields, such as video search, advertising positioning, and content filtering. However, with the continuous increase in video content, traditional recognition and classification methods are no longer applicable. Therefore, researchers continue to propose new methods and algorithms to improve the accuracy and efficiency of video content recognition and classification.

[0003] At the same time, face recognition technology is also developing. Face recognition has been widely used in security inspection, identity verification, and crime investigation. However, traditional face recognition methods have some limitations, such as lighting and angle issues. Summary of the invention

[0004] In view of the above problems, the present invention provides a method and system for generating feature values ​​based on video scene recognition technology, which are used to quickly and accurately identify the scenes of videos and improve the efficiency of video content classification and retrieval.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] On one hand, the present invention provides a method for generating a feature value based on a video scene recognition technology, comprising the following steps:

[0007] Perform frame extraction on the input video to obtain all frames, perform face detection on each frame to obtain all faces in the frame, and track each face to obtain the position information of the face in the entire video;

[0008] Determine the scene type of the current picture based on the proportion of the face in the picture;

[0009] The video is divided into several time periods, and the feature value of each time period is extracted according to the scene and duration of the picture in each time period;

[0010] The feature values ​​of all time periods are taken as input, and machine learning algorithms are used for classification and recognition to classify videos into different types.

[0011] In a possible implementation, the shot types include a close-up of a human body above the shoulders, a close-up of a human body above the chest, a medium shot of a human body above the knees, a panoramic view of the entire human body and part of the surrounding environment, and a distant view of the environment in which the subject is located.

[0012] A possible implementation further includes storing the extracted feature value information in a database.

[0013] In a possible implementation, the feature values ​​of all time periods are taken as input, and the classification and identification are performed using a machine learning algorithm to classify the videos into different types, specifically including:

[0014] The RNN model is constructed, and the input layer is the number of time steps and the input sequence of the feature vector of each step; the hidden layer is the LSTM or GRU layer used to capture the timing information in the sequence; the output layer is a fully connected layer, which is used to map the output of the LSTM or GRU layer to the classification label; the prepared sequence data and the corresponding video scene labels are divided into a training set and a test set; the RNN model is trained with the training set and verified on the test set. During the training process, the hyperparameters of the model need to be adjusted, including the number of time steps that determine the number of frames contained in each sequence, the number of LSTM / GRU units that determine the number of neurons in the hidden layer, the learning rate and the optimizer; frame extraction and feature extraction are performed on the new video, and then these features are prepared into sequence data in the previous way; the sequence data of the new video is input into the trained RNN model, and the model outputs the classification result of each sequence.

[0015] Another aspect of the present invention provides a feature value generation system based on video scene recognition technology, including a video preprocessing module, a scene recognition module, a feature value extraction module and a classification recognition module.

[0016] The video preprocessing module is used to extract frames from the input video to obtain all frames, perform face detection on each frame to obtain all faces in the frame, and track each face to obtain the position information of the face in the entire video;

[0017] The scene recognition module is used to determine the scene type of the current picture according to the proportion of the face in the picture;

[0018] The feature value extraction module is used to divide the video into several time periods, and extract the feature value of each time period according to the scene and duration of the picture in each time period;

[0019] The classification and recognition module is used to take the feature values ​​of all time periods as input, use a machine learning algorithm to perform classification and recognition, and classify the videos into different types.

[0020] In a possible implementation, a feature information storage module is further included, which is used to store the extracted feature value information in a database.

[0021] In a possible implementation, the shot types include a close-up of a human body above the shoulders, a close-up of a human body above the chest, a medium shot of a human body above the knees, a panoramic view of the entire human body and part of the surrounding environment, and a distant view of the environment in which the subject is located.

[0022] In a possible implementation manner, the working process of the classification and recognition module specifically includes:

[0023] The RNN model is constructed, and the input layer is the number of time steps and the input sequence of the feature vector of each step; the hidden layer is the LSTM or GRU layer used to capture the timing information in the sequence; the output layer is a fully connected layer, which is used to map the output of the LSTM or GRU layer to the classification label; the prepared sequence data and the corresponding video scene labels are divided into a training set and a test set; the RNN model is trained with the training set and verified on the test set. During the training process, the hyperparameters of the model need to be adjusted, including the number of time steps that determine the number of frames contained in each sequence, the number of LSTM / GRU units that determine the number of neurons in the hidden layer, the learning rate and the optimizer; frame extraction and feature extraction are performed on the new video, and then these features are prepared into sequence data in the previous way; the sequence data of the new video is input into the trained RNN model, and the model outputs the classification result of each sequence.

[0024] The use of the present invention has the following beneficial effects: it can better reflect the scene changes of the video. Since the recognition is based on the video content, the data level tampering caused by video re-encoding is avoided, the video recognition is improved, and it is more conducive to the inspection and proof of copyright protection. If a specific scene of a video is stolen or the copyright is infringed, the copyright owner can track and prevent piracy by identifying the scene. In addition, through the application of face recognition technology, the characters in the video can be more accurately identified and their permissions can be controlled. For example, the appearance time and frequency of certain characters can be limited to protect their privacy and copyright. The application range is very wide. For example, important scenes can be quickly identified in video surveillance to improve security. On video sharing websites, this technology can help users find videos of interest more quickly. In addition, this technology can also be applied to virtual reality, augmented reality and other fields to improve user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flowchart of the steps of a method for generating feature values ​​based on video scene recognition technology according to an embodiment of the present invention;

[0026] Figure 2 This is a principle block diagram of a feature value generation system based on video scene recognition technology according to another embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] See also Figure 1 , which is a flowchart of a method for generating feature values ​​based on video scene recognition technology according to an embodiment of the present invention, comprises the following steps:

[0029] S10, performing frame extraction on the input video to obtain all frames, performing face detection on each frame to obtain all faces in the frame, and tracking each face to obtain the position information of the face in the entire video; through the above steps, more accurate face information can be obtained, thereby providing a better data basis for subsequent processing.

[0030] S20, judging the scene type of the current picture according to the proportion of the face in the picture;

[0031] S30, dividing the video into several time periods, and extracting the feature value of each time period according to the scene and duration of the picture in each time period; for example, if the duration of a time period is 10 seconds, and the picture is a close-up of a person, the feature value of the time period can be expressed as [close-up of person, 10]. Through the above steps, the features of each scene of the video can be obtained, thereby providing more accurate data for classification and recognition.

[0032] S40, taking the feature values ​​of all time periods as input, using a machine learning algorithm for classification and recognition, and classifying the videos into different types.

[0033] In another embodiment of the present invention, in a feature value generation method based on video scene recognition technology, scene types include close-ups above the shoulders of a human body, close-ups above the chest of a human body, medium shots above the knees of a human body, panoramic views of the entire human body and part of the surrounding environment, and distant views of the environment in which the subject is located. In movies, directors and photographers are accustomed to using complex and changeable scene scheduling and lens scheduling, and alternately using various scene types to make the narration of the film's plot, the expression of the characters' thoughts and feelings, and the handling of character relationships more expressive, thereby enhancing the film's artistic appeal. Therefore, combined with the temporal changes in the scene type in the video, it can be used as one of the feature information of the video.

[0034] In a specific application example, judging the scene type according to the face ratio includes calculating the ratio of the face in the picture, that is, the ratio of the area of ​​the face frame to the area of ​​the entire frame, and judging the scene type of the current picture according to the ratio. Assume that the thresholds of different scene types are: when the face frame area accounts for more than 50% of the entire frame area, it is a close-up shot, which mainly presents the shots above the shoulders of the human body; when the face frame area accounts for 30%-50% of the entire frame area, it is a close-up shot, which mainly presents the shots above the chest of the human body; when the face frame area accounts for 10%-30% of the entire frame area, it is a medium shot, which mainly presents the shots above the knees of the human body; when the face frame area accounts for 1%-10% of the entire frame area, it is a panoramic shot, which mainly presents the entire human body; when the face frame area accounts for less than 1% of the entire frame area, it is a long shot, which mainly presents the shots of the environment in which the subject is located.

[0035] In a method for generating feature values ​​based on video scene recognition technology according to another embodiment of the present invention, continue to refer to Figure 1 , based on S10 - S40 , further comprising S50 , storing the extracted feature value information in a database for subsequent use or analysis.

[0036] In a method for generating feature values ​​based on video scene recognition technology in another embodiment of the present invention, S40 takes the feature values ​​of all time periods as input, uses a machine learning algorithm for classification and recognition, and classifies the video into different types, specifically including:

[0037] The RNN (Recurrent Neural Network) model is constructed, the input layer is the number of time steps, and the input sequence of the feature vector of each step; the hidden layer is the LSTM (Long Short Term Memory) or GRU (Gate Recurrent Unit) layer used to capture the temporal information in the sequence; the output layer is a fully connected layer, which is used to map the output of the LSTM or GRU layer to the classification label; the prepared sequence data and the corresponding video scene labels are divided into a training set and a test set; the RNN model is trained with the training set and verified on the test set. During the training process, the hyperparameters of the model need to be adjusted, and the hyperparameters include the number of time steps that determine the number of frames contained in each sequence, the number of LSTM / GRU units that determine the number of neurons in the hidden layer, the learning rate and the optimizer; the new video is frame extracted and feature extracted, and then these features are prepared into sequence data in the previous way; the sequence data of the new video is input into the trained RNN model, and the model outputs the classification result of each sequence. Through the above steps, the RNN can be effectively used to model the temporal features of the video, thereby improving the accuracy of video classification.

[0038] The above-set feature value generation method based on video scene recognition technology can more accurately identify the scene of the video and improve the efficiency of video content classification and retrieval by combining the size of the screen occupied by the face and the duration of the scene as the video feature value. Not only can different scenes be identified, but also more specific classification can be performed according to the changes in the size of the screen occupied by the face. For example, in a certain scene, the face occupies most of the screen, and it can be inferred that this is a scene of a party or celebration. In addition, by using the duration of the scene as a video feature value, the video can be analyzed more carefully. For example, for some important scenes, their importance or urgency can be evaluated by their duration in the video. The above settings improve the efficiency of video content classification and retrieval, and also enhance the recognition accuracy of video scenes, and have broad application prospects.

[0039] Corresponding to the method embodiment, Figure 2 The figure shows a principle block diagram of a feature value generation system based on video scene recognition technology according to another embodiment of the present invention, which includes a video preprocessing module 101, a scene recognition module 102, a feature value extraction module 103 and a classification recognition module 104, wherein the video preprocessing module 101 is used to extract frames from the input video to obtain all frames, perform face detection on each frame to obtain all faces in the frame, and track each face to obtain the position information of the face in the entire video. Through the above modules, more accurate face information can be obtained, thereby providing a better data basis for subsequent processing; the scene recognition module 102 is used to determine the scene type of the current picture according to the proportion of the face in the picture; the feature value extraction module 103 is used to divide the video into several time periods, and extract the feature value of each time period according to the scene and duration of the picture in each time period; for example, if the duration of a time period is 10 seconds, and the picture is a close-up of a person, the feature value of the time period can be expressed as [close-up of person, 10]. Through the above steps, the features of the video in each scene picture can be obtained, thereby providing more accurate data for classification recognition. The classification and recognition module 104 is used to take the feature values ​​of all time periods as input, use a machine learning algorithm to perform classification and recognition, and classify the videos into different types.

[0040] In a feature value generation system based on video scene recognition technology in another embodiment of the present invention, scene types include close-ups of the human body above the shoulders, close-ups of the human body above the chest, medium shots of the human body above the knees, panoramic views of the entire human body and part of the surrounding environment, and distant views of the environment in which the subject is located. In movies, directors and photographers are accustomed to using complex and changeable scene scheduling and lens scheduling, and alternately using various scene types to make the narration of the film's plot, the expression of the characters' thoughts and feelings, and the handling of the relationships between the characters more expressive, thereby enhancing the artistic appeal of the film. Therefore, combined with the temporal changes in the scene types in the video, it can be used as one of the feature information of the video.

[0041] In a specific application example, judging the scene type according to the face ratio includes calculating the ratio of the face in the picture, that is, the ratio of the area of ​​the face frame to the area of ​​the entire frame, and judging the scene type of the current picture according to the ratio. Assume that the thresholds of different scene types are: when the face frame area accounts for more than 50% of the entire frame area, it is a close-up shot, which mainly presents the shots above the shoulders of the human body; when the face frame area accounts for 30%-50% of the entire frame area, it is a close-up shot, which mainly presents the shots above the chest of the human body; when the face frame area accounts for 10%-30% of the entire frame area, it is a medium shot, which mainly presents the shots above the knees of the human body; when the face frame area accounts for 1%-10% of the entire frame area, it is a panoramic shot, which mainly presents the entire human body; when the face frame area accounts for less than 1% of the entire frame area, it is a long shot, which mainly presents the shots of the environment in which the subject is located.

[0042] In another embodiment of the present invention, in a feature value generation system based on video scene recognition technology, see Figure 2 Based on the video preprocessing module 101, the scene recognition module 102, the feature value extraction module 103 and the classification recognition module 104, it further includes a feature information storage module 105 for storing the extracted feature value information in a database for subsequent use or analysis.

[0043] In a feature value generation system based on video scene recognition technology in another embodiment of the present invention, the classification recognition module 104 takes the feature values ​​of all time periods as input, uses a machine learning algorithm to perform classification recognition, and classifies the video into different types, specifically including:

[0044] The RNN model is constructed, the input layer is the number of time steps, and the input sequence of the feature vector of each step; the hidden layer is the LSTM or GRU (Gate Recurrent Unit) layer for capturing the temporal information in the sequence; the output layer is a fully connected layer for mapping the output of the LSTM or GRU layer to the classification label; the prepared sequence data and the corresponding video scene labels are divided into a training set and a test set; the RNN model is trained with the training set and verified on the test set. During the training process, the hyperparameters of the model need to be adjusted, and the hyperparameters include the number of time steps that determine the number of frames contained in each sequence, the number of LSTM / GRU units that determine the number of neurons in the hidden layer, the learning rate and the optimizer; the new video is subjected to frame extraction and feature extraction, and then these features are prepared into sequence data in the previous manner; the sequence data of the new video is input into the trained RNN model, and the model outputs the classification result of each sequence. Through the above steps, the RNN can be effectively used to model the temporal features of the video, thereby improving the accuracy of video classification.

[0045] The feature value generation system based on the video scene recognition technology set up above can more accurately identify the scene of the video and improve the efficiency of video content classification and retrieval by combining the size of the screen occupied by the face and the duration of the scene as the video feature value. Not only can different scenes be identified, but also more specific classification can be performed according to the changes in the size of the screen occupied by the face. For example, in a certain scene, the face occupies most of the screen, and it can be inferred that this is a scene of a party or celebration. In addition, by using the duration of the scene as a video feature value, the video can be analyzed more carefully. For example, for some important scenes, their importance or urgency can be evaluated by their duration in the video. The above settings improve the efficiency of video content classification and retrieval, and also enhance the recognition accuracy of video scenes, and have broad application prospects.

[0046] It should be understood that the exemplary embodiments described herein are illustrative rather than restrictive. Although one or more embodiments of the present invention have been described in conjunction with the accompanying drawings, it should be understood by those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims

1. A method for generating feature values ​​based on video scene recognition technology, characterized in that: The following steps are involved: Perform frame extraction on the input video to obtain all frames, perform face detection on each frame to obtain all faces in the frame, and track each face to obtain the position information of the face in the entire video; Determine the scene type of the current picture based on the proportion of the face in the picture; The video is divided into several time periods, and the feature value of each time period is extracted according to the scene and duration of the picture in each time period; The feature values ​​of all time periods are taken as input, and machine learning algorithms are used for classification and recognition to classify videos into different types.

2. The feature value generation method based on video scene recognition technology according to claim 1, characterized in that: The types of shots include close-ups of the human body above the shoulders, close shots of the human body above the chest, medium shots of the human body above the knees, panoramic shots of the entire human body and part of the surrounding environment, and distant shots of the environment in which the subject is located.

3. The feature value generation method based on video scene recognition technology according to claim 1, characterized in that: The method further includes storing the extracted feature value information in a database.

4. The feature value generation method based on video scene recognition technology according to any one of claims 1 to 3, characterized in that: The feature values ​​of all time periods are taken as input, and the machine learning algorithm is used for classification and recognition, and the video is classified into different types, specifically including: The RNN model is constructed, and the input layer is the number of time steps and the input sequence of the feature vector of each step; the hidden layer is the LSTM or GRU layer used to capture the timing information in the sequence; the output layer is a fully connected layer, which is used to map the output of the LSTM or GRU layer to the classification label; the prepared sequence data and the corresponding video scene labels are divided into a training set and a test set; the RNN model is trained with the training set and verified on the test set. During the training process, the hyperparameters of the model need to be adjusted, including the number of time steps that determine the number of frames contained in each sequence, the number of LSTM / GRU units that determine the number of neurons in the hidden layer, the learning rate and the optimizer; frame extraction and feature extraction are performed on the new video, and then these features are prepared into sequence data in the previous way; the sequence data of the new video is input into the trained RNN model, and the model outputs the classification result of each sequence.

5. A feature value generation system based on video scene recognition technology, characterized in that: It includes video preprocessing module, scene recognition module, feature value extraction module and classification recognition module. The video preprocessing module is used to extract frames from the input video to obtain all frames, perform face detection on each frame to obtain all faces in the frame, and track each face to obtain the position information of the face in the entire video; The scene recognition module is used to determine the scene type of the current picture according to the proportion of the face in the picture; The feature value extraction module is used to divide the video into several time periods, and extract the feature value of each time period according to the scene and duration of the picture in each time period; The classification and recognition module is used to take the feature values ​​of all time periods as input, use a machine learning algorithm to perform classification and recognition, and classify the videos into different types.

6. The feature value generation system based on video scene recognition technology according to claim 5, characterized in that: It further includes a feature information storage module for storing the extracted feature value information in a database.

7. The feature value generation method based on video scene recognition technology according to claim 5, characterized in that: The types of shots include close-ups of the human body above the shoulders, close shots of the human body above the chest, medium shots of the human body above the knees, panoramic shots of the entire human body and part of the surrounding environment, and distant shots of the environment in which the subject is located.

8. The feature value generation method based on video scene recognition technology according to any one of claims 1 to 3, characterized in that: The working process of the classification and recognition module specifically includes: The RNN model is constructed, and the input layer is the number of time steps and the input sequence of the feature vector of each step; the hidden layer is the LSTM or GRU layer used to capture the timing information in the sequence; the output layer is a fully connected layer, which is used to map the output of the LSTM or GRU layer to the classification label; the prepared sequence data and the corresponding video scene labels are divided into a training set and a test set; the RNN model is trained with the training set and verified on the test set. During the training process, the hyperparameters of the model need to be adjusted, including the number of time steps that determine the number of frames contained in each sequence, the number of LSTM / GRU units that determine the number of neurons in the hidden layer, the learning rate and the optimizer; frame extraction and feature extraction are performed on the new video, and then these features are prepared into sequence data in the previous way; the sequence data of the new video is input into the trained RNN model, and the model outputs the classification result of each sequence.

Citation Information

Patent Citations

  • Lens classifying method, situation extracting method, abstract generating method and device

    CN101072305A

  • Method for classifying sports video based on key frame of main scene lens

    CN101604325A

  • Shot division method and system based on video analysis

    CN107437076A

  • Human behavior recognition method based on long-short-term memory network

    CN109101876A

  • Video scene analysis method and device based on human body

    CN111160134A