Method, device, electronic device and storage medium for identifying target object in video

Abstract

The present invention discloses a method, device, electronic device and storage medium for identifying a target object in a video, so as to solve the problem of high labor cost and time cost caused by manually reviewing videos in which a group of people of a specific age group appears alone in the prior art. The method comprises: determining the type of a video to be identified according to each video frame in an obtained frame sequence, wherein the frame sequence is a combination of video frames extracted from the video to be identified in time order, and the type of the video to be identified represents the generation method of the video to be identified; obtaining at least one video subunit from the frame sequence according to the type of the video to be identified, wherein the video subunit contains video frames of the same scene; for each video subunit, detecting a target object area in the video subunit, predicting the age information of the target object in each target object area, and identifying whether the target object in the video to be identified belongs to a group of a specified age group according to the corresponding age information of the target object.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, electronic device and storage medium for identifying a target object in a video. Background Art

[0002] With the popularity of online live broadcast platforms and video websites, a large number of videos uploaded by users are generated on various video storage platforms every day. Among the users who upload videos and the main video content in the uploaded videos, there are many types of videos in which groups of specific age groups appear alone, such as videos in which minors appear alone, videos in which the elderly appear alone, etc. In order to ensure the healthy use of the Internet by groups of specific age groups and prevent criminals from affecting and harming the physical and mental health of groups of specific age groups, it is necessary to screen and protect videos in which groups of specific age groups appear alone to reduce the exposure of these videos.

[0003] At present, the review of videos in which people of a specific age group appear alone mainly relies on reviewers to watch the entire video. Due to the high labor cost and the huge amount of videos produced and uploaded by users, manual review requires a lot of time and will greatly extend the time interval between users watching other types of videos except for videos in which people of a specific age group appear alone, affecting the user experience of the video.

[0004] Therefore, there is an urgent need for a method for reviewing videos to screen out videos in which groups of specific age groups appear alone, so as to save manpower and time costs. Summary of the invention

[0005] In order to solve the problem of high labor and time costs caused by manually reviewing videos in which a group of people in a specific age group appear alone in the prior art, an embodiment of the present invention provides a method, device, electronic device and storage medium for identifying a target object in a video.

[0006] In a first aspect, an embodiment of the present invention provides a method for identifying a target object in a video, comprising:

[0007] Determine the type of the video to be identified according to each video frame in the obtained frame sequence, wherein the frame sequence is a combination of video frames extracted from the video to be identified in time order, and the type of the video to be identified represents the generation method of the video to be identified;

[0008] Obtaining at least one video sub-unit from the frame sequence according to the type of the video to be identified, wherein the video sub-unit includes video frames of the same scene;

[0009] For each video sub-unit, the target object area in the video sub-unit is detected, and the age information of the target object in each target object area is predicted, and based on the corresponding age information of the target object, it is identified whether the target object in the video to be identified belongs to a group of a specified age group.

[0010] In a second aspect, an embodiment of the present invention provides a device for identifying a target object in a video, comprising:

[0011] a determination unit, configured to determine the type of the video to be identified according to each video frame in the obtained frame sequence, wherein the frame sequence is a combination of video frames extracted from the video to be identified in time sequence, and the type of the video to be identified represents a generation method of the video to be identified;

[0012] an obtaining unit, configured to obtain at least one video sub-unit from the frame sequence according to the type of the video to be identified, wherein the video sub-unit includes video frames of the same scene;

[0013] The identification unit is used to detect the target object area in each video sub-unit, predict the age information of the target object in each target object area, and identify whether the target object in the video to be identified belongs to a group of specified age group based on the age information of the corresponding target object.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for identifying target objects in a video as described in the present invention when executing the program.

[0015] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for identifying a target object in a video described in the present invention.

[0016] The beneficial effects of the embodiments of the present invention are as follows:

[0017] In an embodiment of the present invention, a video audit device determines the type of a video to be identified based on each video frame in an acquired frame sequence, wherein the frame sequence is a combination of video frames extracted by the video audit device from the video to be identified in chronological order, and the type of the video to be identified represents the generation method of the video to be identified. Furthermore, at least one video sub-unit is obtained from the frame sequence according to the type of the video frame to be identified, and the video sub-unit contains video frames of the same scene. The target object area in each video sub-unit is detected respectively, and the age information of the target object in each target object area is predicted. According to the age information of the corresponding target object in the target object area of ​​each video frame sub-unit, it is identified whether the target object in the video to be identified belongs to a group of specified age group. Compared with the prior art, the method for identifying target objects in the above-mentioned video provided by the embodiment of the present invention uses the video audit device to automatically identify whether the target object in the video to be identified belongs to a group of specified age group according to the combination of video frames extracted from the video to be identified in chronological order, thereby effectively saving the manpower cost and time cost consumed by manual review.

[0018] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0020] Figure 1 A schematic diagram of an application scenario of the method for identifying a target object in a video provided by an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of an implementation flow of a method for identifying a target object in a video provided by an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of an implementation flow of determining the type of a video to be identified provided by an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of an implementation flow of determining the similarity between two adjacent video frames provided by an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of an implementation flow of detecting a target object area in each video sub-unit provided by an embodiment of the present invention;

[0025] Figure 6 A schematic diagram of an implementation flow of predicting age information of a target object in each target object area provided by an embodiment of the present invention;

[0026] Figure 7 A schematic diagram of the structure of a device for identifying a target object in a video provided by an embodiment of the present invention;

[0027] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to solve the problem of high labor and time costs caused by manually reviewing videos in which a group of people in a specific age group appear alone in the prior art, an embodiment of the present invention provides a method, device, electronic device and storage medium for identifying a target object in a video.

[0029] The preferred embodiments of the present invention are described below in conjunction with the drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other if there is no conflict.

[0030] First reference Figure 1, which is a schematic diagram of an application scenario of the method for identifying a target object in a video provided by an embodiment of the present invention, and may include a video decoding device 110 and a video review device 120. The video decoding device 110 and the video review device 120 are connected via a network. When it is necessary to identify whether a video is a video in which a group of a specific age group appears alone, the video review device 120 may extract video frames generated after decoding the video to be identified from the video decoding cluster of the video decoding device 110 in a sequence according to a preset time interval. For example, a frame of the decoded video of the video to be identified may be extracted every 1 second to obtain a frame sequence. The video review device 120 determines the type of the video to be identified based on each video frame in the obtained frame sequence. The type of the video may include but is not limited to the following types: a co-produced type video, a picture set type video, and an independent complete type video. The co-produced type video is generally divided into two different scenes on the left and right with the central axis as the dividing line. The picture set type video is generally a video generated by combining multiple single pictures at a set time interval. The independent complete type video is an independent complete video shot in the same scene. The video review device 120 obtains at least one video sub-unit from the obtained frame sequence according to the type of the video to be identified, and then the video review device 120 detects the target object area in each video sub-unit respectively, wherein the target object is a face, and the target object area is a face position area, predicts the age information of the face in each face position area, and identifies whether the face in the video to be identified belongs to a group of specified age group according to the age information of the corresponding face in the face position area of ​​each video frame sub-unit, that is, identifies whether the video to be identified is a video in which a group of specified age group appears alone.

[0031] In another application scenario of the embodiment of the present invention, the video review device 120 may also decode the video to be identified, obtain decoded video frames, and extract the frame sequence in chronological order, which is not limited by the embodiment of the present invention.

[0032] The video decoding device 110 and the video review device 120 may be independent physical servers or cloud servers that provide basic cloud computing services such as cloud servers, cloud databases, and cloud storage, which is not limited in this embodiment of the present invention.

[0033] Based on the above application scenarios, the following will refer to the attached Figure 2 to Figure 4 The exemplary embodiments of the present invention are described in more detail. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present invention, and the embodiments of the present invention are not limited in any way. On the contrary, the embodiments of the present invention can be applied to any applicable scenario.

[0034] like Figure 2As shown, it is a schematic diagram of the implementation process of the method for identifying a target object in a video provided by an embodiment of the present invention. The method for identifying a target object in a video can be applied to the above-mentioned video review device 120, and specifically may include the following steps:

[0035] S21. Determine the type of the video to be identified according to each video frame in an obtained frame sequence, wherein the frame sequence is a combination of video frames extracted from the video to be identified in time sequence.

[0036] In specific implementation, the video audit device extracts the video frames generated after decoding the video to be identified from the video decoding cluster of the video decoding device in a preset time interval sequence to obtain a frame sequence arranged in time order. The frame sequence is a combination of video frames extracted from the video frames to be identified in time order. For example, a video frame after decoding the video to be identified can be extracted every 1 second. This is not limited in the embodiment of the present invention. Furthermore, the video audit device determines the type of the video to be identified based on each video frame in the obtained frame sequence. The type of the video to be identified represents the generation method of the video to be identified.

[0037] Specifically, according to Figure 3 The process shown in the figure determines the type of video to be identified, and may include the following steps:

[0038] S31, performing edge detection on each video frame respectively, and when it is determined that the number of pixel rows of the boundary detected in the preset rectangular area at the central axis of each video frame is greater than a first preset threshold, determining that the video to be identified is a co-shot type video.

[0039] During specific implementation, the video review device performs edge detection on each video frame in the obtained frame sequence.

[0040] Specifically, the Canny edge detection operator can be used to perform edge detection on each video frame respectively. Assuming that the resolution of the video frame is: w*h, that is, the number of pixels of the side length is w, and the number of pixels of the width is h, the number of pixel rows in which the boundary is detected in the preset rectangular area at the central axis of each video frame is counted, wherein the preset rectangular area at the central axis of the video frame can be set to a rectangular area with a size of 6*h with the central axis of the video frame as the center line, and the embodiment of the present invention is not limited to this. Then, the number of pixel rows in which the boundary is detected in the rectangular area with a size of 6*h at the central axis of each video frame is counted. If the number of pixel rows in which the boundary is detected in the rectangular area with a size of 6*h at the central axis of each video frame is greater than a first preset threshold, it is determined that the boundary detected in each video frame is the dividing line of the video frame (that is, the vertical boundary, generally the central axis), and it is determined that the video to be identified is a co-shot type video, wherein the first preset threshold can be set according to an empirical value, for example, it can be set to 0.6*h, and the embodiment of the present invention is not limited to this. During specific implementation, a suitable edge detection operator may be selected according to actual needs. In addition to the Canny operator, the edge detection operator may also be, but is not limited to, the Sobel operator, Prewitt operator, Roberts operator, or Marr-Hildreth operator, etc., which is not limited in the embodiment of the present invention.

[0041] S32: When it is determined that the number of pixel rows where a boundary is detected within the preset rectangular area at the central axis of any of the video frames is less than or equal to the first preset threshold, determine the similarity between every two adjacent video frames.

[0042] In a specific implementation, if the number of pixel rows detected from the boundary within the preset rectangular area at the central axis of any of the video frames is less than or equal to the first preset threshold, it is determined that the video to be identified is not a co-shot type video, and the similarity between each two adjacent video frames is further determined.

[0043] Specifically, according to Figure 4 The process shown determines the similarity between two adjacent video frames and may include the following steps:

[0044] S41. Calculate the DHash value of each of two adjacent video frames according to the difference hash algorithm.

[0045] In specific implementation, the video review device can calculate the DHash values ​​of any two adjacent video frames according to the Difference Hash (DHash) algorithm.

[0046] S42: Determine the Hamming distance between the DHash values ​​of the two adjacent video frames.

[0047] During specific implementation, the video review device calculates the Hamming distance between the DHash values ​​of the two adjacent video frames, where the Hamming distance between two equal-length character strings refers to the number of different characters in corresponding positions of the two character strings. For example, the Hamming distance between "1011101" and "1001001" is 2. The DHash values ​​of the two adjacent video frames calculated in step S41 are two equal-length character strings. The Hamming distance between the DHash values ​​of the two adjacent video frames is the number of different characters in corresponding positions of the DHash values ​​of the two adjacent video frames.

[0048] S43: Determine the similarity between the two adjacent video frames according to the Hamming distance.

[0049] In a specific implementation, the video review device determines the similarity between the two adjacent video frames based on the Hamming distance between the DHash values ​​of the two adjacent videos.

[0050] Specifically, the similarity between two adjacent video frames is calculated by the following formula:

[0051]

[0052] Where p represents the similarity between two adjacent video frames;

[0053] d represents the Hamming distance between the DHash values ​​of two adjacent video frames.

[0054] S33: When it is determined that the similarity between any two adjacent video frames is greater than a second preset threshold, determine that the video to be identified is a picture set type video.

[0055] In a specific implementation, if the similarity between any two adjacent video frames in the frame sequence is greater than a second preset threshold, the video to be identified is determined to be a picture set type video, wherein the second preset threshold can be set according to an empirical value, for example, it can be set to any value between 0.9 and 0.95, and the embodiment of the present invention is not limited to this.

[0056] S34: When it is determined that the similarities between each two adjacent video frames are less than or equal to the second preset threshold, determine that the video to be identified is an independent complete type video.

[0057] In a specific implementation, if the similarity between every two adjacent video frames in the frame sequence is less than or equal to the second preset threshold, it is determined that the video to be identified is an independent and complete type of video.

[0058] In the embodiment of the present invention, the video review device extracts video frames from the video frames generated after decoding the video to be identified in chronological order and combines them into a frame sequence. For videos in which a group of people of a specific age group appear alone, if the review is conducted after the video is exposed, there will be a window period for the video content, which increases the risk of damage to the rights and interests of the uploader of the video in which a group of people of a specific age group appear alone and the group of the specific age group in the video. Therefore, in the embodiment of the present invention, the video frame sequence sampling after video decoding is used, and automatic review and screening can be achieved during the video decoding process, thereby improving the timeliness of video review. And the type of the video to be identified is identified in the following way: first, edge detection is performed on each video frame in the frame sequence to determine whether the video to be identified is a co-production type video; if it is not a co-production type video, the similarity between each two adjacent video frames is further determined to determine whether it is a picture set type video or an independent complete type video; then, according to the type of the video to be identified, the video frames in the frame sequence are divided into at least one video sub-unit (i.e., the minimum video unit) to perform more fine-grained identification of the video to be identified and improve the accuracy of the identification result.

[0059] S22. Obtain at least one video sub-unit from the frame sequence according to the type of the video to be identified.

[0060] In a specific implementation, the video review device obtains at least one video sub-unit, ie, a minimum video unit, from the frame sequence according to the type of the video to be identified, wherein the video sub-unit (minimum video unit) contains video frames of the same scene.

[0061] Specifically, when the video to be identified is a co-produced video, the video review device may obtain at least one video sub-unit from the frame sequence in the following manner:

[0062] The column with the most boundary pixels detected within the preset rectangular area at the central axis of each video frame in the frame sequence is used as the dividing line of each video frame, and each video frame is divided into two left and right video sub-frames; and the left video sub-frame sequence and the right video sub-frame sequence are respectively determined as two video sub-units.

[0063] In specific implementation, since the video contents on both sides of the dividing line of the video of the co-produced type video generally belong to two different scenes, each video frame is divided into two left and right video sub-frames according to the dividing line of each video frame in the detected frame sequence. In the frame sequence, the column with the most pixels detected at the boundary in the preset rectangular area at the central axis of each video frame is the dividing line of each video frame. The dividing line of each video frame is used as the dividing line to divide it into two left and right video sub-frames, and the video sub-frame sequence on the left and the video sub-frame sequence on the right are respectively determined as two video sub-units, that is, the video sub-frame sequence on the left and the video sub-frame sequence on the right are determined as two minimum video units. In this way, it can be ensured that the two divided video sub-units each belong to the same scene, further improving the recognition result.

[0064] When the video to be identified is a picture set type video, the video review device can obtain at least one video sub-unit from the frame sequence in the following manner:

[0065] When it is determined that the similarity between adjacent video frames is less than a third preset threshold, the adjacent video frames are used as cutting points to segment the frame sequence; each video frame combination obtained after segmentation is determined as a video sub-unit, and the video frames in each video sub-unit are determined as video sub-frames.

[0066] In specific implementation, the similarity between every two adjacent video frames in the frame sequence is still determined according to steps S41 to S43. When it is determined that the similarity between adjacent video frames is less than the third preset threshold, the adjacent video frames are used as cutting points to segment the frame sequence, and each video frame combination after segmentation is determined as a video sub-unit (i.e., the minimum video unit), and the video frames in each video sub-unit are determined as video sub-frames, wherein the third preset threshold can be set according to actual conditions, for example, it can be set to 0.6, which is not limited in the embodiment of the present invention. Assuming that the frame sequence contains 10 video frames, wherein the similarity between the 4th video frame and the 5th video frame is less than 0.6, the similarity between the 7th video frame and the 8th video frame is less than 0.6, and the similarity between every other two adjacent video frames is greater than 0.6, then the frame sequence can be segmented into the following three video sub-units: the 1st to 4th video frames are one video sub-unit, the 5th to 7th video frames are one video sub-unit, and the 8th to 10th video frames are one video sub-unit. For picture set type videos, if the similarity between two adjacent frames is small, it is believed that the video content before and after the two frames has low coherence, and the probability that the video content before and after the two frames is the same scene is also low.

[0067] In the embodiment of the present invention, since the picture set type video is generated by combining multiple single pictures according to a set time interval, the coherence between the contents of two adjacent frames of pictures can be judged according to the similarity between the two frames of pictures. The stronger the coherence between the pictures, the higher the probability that these consecutive pictures belong to the same scene. The frame sequence is segmented according to the coherence between the picture contents to divide the video sub-units, thereby further improving the accuracy of the recognition results.

[0068] When it is determined that the video to be identified is an independent and complete type of video, that is, when the video to be identified is an independent and complete video in the same scene, the frame sequence is not segmented, and the frame sequence is directly determined as a video sub-unit (that is, the smallest video unit), and the video frame in the video sub-unit is determined as a video sub-frame.

[0069] S23. For each video sub-unit, the target object area in the video sub-unit is detected, and the age information of the target object in each target object area is predicted, and according to the age information of the corresponding target object, it is identified whether the target object in the to-be-identified video belongs to a group of a specified age group.

[0070] In specific implementation, Figure 5 The process shown detects the target object area in each video sub-unit, and may include the following steps:

[0071] S51. For each video sub-unit, perform face detection on each video sub-frame of the video sub-unit according to a face detection model to obtain face position information and face feature information in each video sub-frame, wherein the face position information includes a face rectangular bounding box for marking the face.

[0072] During specific implementation, the video review device performs face detection on each video subframe of each video subunit according to the face detection model, and obtains face position information and face feature information in each video subframe, wherein the face position information includes a face rectangular bounding box for marking the face, wherein the face detection model can use but is not limited to the following models: DBFace model, RetinaFace model, SRN (Selective Refinement Network) model, etc.

[0073] S52, respectively calculating the intersection-and-union ratio of each face rectangular bounding box in each video subframe and each face rectangular bounding box in its adjacent video subframe.

[0074] In specific implementation, the video review device calculates the intersection over union (IoU) of the area of ​​each face rectangular bounding box in each video subframe and the area of ​​each face rectangular bounding box in its adjacent video subframe. The intersection over union ratio of the areas of two face rectangular bounding boxes is: the ratio of the intersection of the areas of the two face rectangular bounding boxes to the union of the areas of the two face rectangular bounding boxes.

[0075] S53: Mark two human face rectangular bounding boxes whose intersection-over-union ratio is greater than a fourth preset threshold as a continuous human face detection pair.

[0076] In a specific implementation, two human face rectangular bounding boxes whose intersection-over-union ratio in every two adjacent video subframes is greater than a fourth preset threshold are marked as a continuous human face detection pair, wherein the fourth preset threshold can be set according to an empirical value, for example, can be set to 0.8, and the embodiment of the present invention is not limited to this.

[0077] S54. For each continuous face detection pair, when it is determined that the similarity of the facial feature information corresponding to the two human face rectangular bounding boxes of the face detection pair is greater than the fifth preset threshold, it is determined that the face detection pair detects the same face; otherwise, it is determined that the face detection pair detects different faces.

[0078] During specific implementation, the video review device further determines the similarity of facial feature information of the faces in the two corresponding human face rectangular bounding boxes for each continuous face detection pair detected. When it is determined that the similarity of facial feature information of the faces in the two human face rectangular bounding boxes of the face detection pair is greater than the fifth preset threshold, it is determined that the face detection pair has detected the same face. Otherwise, it is determined that the face detection pair has detected different faces. The similarity of two human face feature information can be determined by, but is not limited to, calculating the Euclidean distance or cosine similarity between the two human face feature information. The fifth preset threshold can be set according to an empirical value, for example, it can be set to 0.9, which is not limited in the embodiment of the present invention. The above method improves the accuracy of face detection by first calculating the intersection and union ratio of the areas of the human face rectangular bounding boxes detected in two adjacent video subframes, and combining the similarity of the detected facial feature information to further determine whether the detected face is the same face.

[0079] S55. Record the face-related information detected from the video subunit.

[0080] During specific implementation, the video review device uniquely marks the information of the same person detected from the video sub-unit at the person level. For example, a unique identifier can be set for each detected face, and relevant information of each face detected in the video sub-unit is recorded. The relevant information of the face includes at least face position information, face feature information and face identification information.

[0081] In an embodiment of the present invention, for each video sub-unit, a face detection model is used to perform face detection on each video sub-frame in the video sub-unit to obtain face position information and face feature information in each video sub-frame, wherein the face position information is the target area for detection, including a face rectangular bounding box for marking the face, and the target object is the face. Then, the intersection and union ratio of each face rectangular bounding box in each video sub-frame and each face rectangular bounding box in its adjacent video sub-frame is calculated respectively, and two face rectangular bounding boxes whose intersection and union ratio is greater than a fourth preset threshold are marked as a continuous face detection pair. For each continuous face detection pair, when it is determined that the similarity of the face feature information corresponding to the two face rectangular bounding boxes of the face detection pair is greater than a fifth preset threshold, the face detection pair is determined. The detected faces are the same person, otherwise, it is determined that the face detection pair has detected different faces, and the number of faces appearing in the video subunit is recorded. Compared with the prior art that uses a single picture as the input of the face detection model for face detection, since the person information of a single picture is affected by the model accuracy, the state of the person in the picture, the light intensity in the picture, the light angle, etc., the anti-interference ability for identifying the input image is poor, and it is easy to give fluctuating prediction results, resulting in low recognition accuracy. In the embodiment of the present invention, multiple video frames extracted from the video frames generated after decoding the video to be identified are used as the input of the face detection model to perform face detection, so that the detection process is less affected by the state of the person in the picture, the light intensity in the picture and the light angle, thereby improving the accuracy of the face detection result. Moreover, in the prior art, the entire video can be used as the input of the face detection model. Since the video file occupies a large space and contains redundant information other than non-video image information (such as audio information), this will increase the network transmission load and make the hardware resource calculation amount larger. In the embodiment of the present invention, the video frame after video decoding is used as the input of the face detection model. Since the video frame after video decoding only contains image information and does not contain other redundant information such as audio information, it can avoid using the video file as the input of the face detection model, reducing the network transmission load and reducing the hardware resource calculation amount. In addition, in the implementation of the present invention, by calculating the intersection and union ratio of the face rectangular boundary box in the adjacent video frames and the similarity of the corresponding two face feature information, it is detected whether the corresponding two faces in the two adjacent video frames are the same face. If the time interval between the two video frames is long, the false recognition rate of the same face detected in the two video frames is high. Therefore, by detecting whether the same face exists in each two adjacent video frames, the accuracy of face recognition is improved.

[0082] Furthermore, the video review device predicts the age information of the target object in each target object area for each video sub-unit.

[0083] In specific implementation, Figure 6 The process shown in the figure predicts the age information of the target object in each target object area, that is, predicts the age information of each face detected in each video subframe of the video subunit, and may include the following steps:

[0084] S61. For each video sub-unit, perform age detection on each human face detected in each video sub-frame of the video sub-unit according to an age detection model to obtain an age prediction value corresponding to each human face in each video sub-frame.

[0085] In specific implementation, the video review device performs age detection on each face detected in each video subframe of each video subunit according to the age detection model, and obtains the age prediction value corresponding to each face in each video subframe.

[0086] Specifically, for each video sub-unit, the face position information and face feature information obtained from each video sub-frame in the video sub-unit according to the face detection model are input into the age detection model to obtain the age prediction value corresponding to each face in each video sub-frame. The age detection model can be, but is not limited to, the following models: BridgeNet model, deep learning model based on convolutional neural network, etc., which is not limited in the embodiment of the present invention.

[0087] S62: For the same human face detected from different video subframes in the video sub-unit, use a preset clustering algorithm to remove outliers in each age prediction value of the same human face.

[0088] During specific implementation, the video review device counts the information of the same face detected from different video subframes in the video sub-unit, and uses a preset clustering algorithm to eliminate outliers in the age prediction values ​​of the same face, wherein the preset clustering algorithm can be but is not limited to using a K-means clustering algorithm, and the embodiment of the present invention does not limit this.

[0089] Specifically, for the same face detected from different video subframes in the video subunit, the K-means clustering algorithm is used to calculate the cluster center of each age prediction value of the same face, and outliers are determined as false detections and deleted, thereby improving the accuracy of age prediction.

[0090] In the embodiment of the present invention, for each video sub-unit, an age detection model is further used to perform age detection on the face detected in each video frame of the video sub-unit to predict the age corresponding to the face in each video frame. For the same face detected from different video sub-frames in the video sub-unit, a preset clustering algorithm is used to remove outliers in each age prediction value of the face to improve the accuracy of age detection. In addition, in the embodiment of the present invention, multiple video frames extracted from the video frames generated after decoding the video to be identified are used to perform age detection based on the age detection model, so that the detection process is less affected by the state of the person in the picture, the light intensity and the light angle in the picture, and the accuracy of the age detection result is improved. In addition, the use of video files as the input of the age detection model is avoided, the network transmission load is reduced, and the calculation amount of hardware resources is reduced.

[0091] Furthermore, the video review device identifies whether the target object in the video to be identified belongs to a group of specified age group based on the age information of the target object identified from each video sub-unit, that is: based on the age information of the face identified from each video sub-unit, it identifies whether the face in the video to be identified belongs to a group of specified age group, thereby determining whether the video to be identified is a video in which the group of specified age group appears alone.

[0092] In a specific implementation, for a video subunit in which the number of detected faces is 1, when it is determined that the minimum value of the age prediction value and the mode of the age prediction value for the face are both within the specified age group, then it is determined that the target object in the video subunit belongs to the group of the specified age group, and the mode of the age prediction value represents the age prediction value with the largest number of occurrences; for a video subunit in which the number of detected faces is greater than 1, when it is determined that the minimum value of the age prediction value and the mode of the age prediction value for each face are both within the specified age group, and the number of frames of the age prediction value of the face detected from the video subunit in the specified age group is less than a sixth preset threshold, then it is determined that the target object in the video subunit belongs to the group of the specified age group; when it is determined that the target object in each video subunit belongs to the group of the specified age group, it is determined that the target object in the video to be identified belongs to the group of the specified age group.

[0093] Specifically, for a video sub-unit in which the number of detected faces is 1, when it is determined that the minimum value of the age prediction value and the mode of the age prediction value for the face are both in a specified age group, the video review device determines that the face in the video sub-unit belongs to the group in the specified age group, and the mode of the age prediction value represents the age prediction value that appears most times.

[0094] For a video subunit in which the number of detected faces is greater than 1, when it is determined that the minimum value of the age prediction value for each face and the mode of the age prediction value are both within the specified age range, and the number of frames of the age prediction value of the face detected from the video subunit outside the specified age range is less than the sixth preset threshold, it is determined that the face in the video subunit belongs to the group of the specified age range. If the number of frames of the age prediction value of the face detected from the video subunit outside the specified age range is greater than or equal to the sixth preset threshold, it is determined that the face in the video subunit does not belong to the group of the specified age range. The sixth preset threshold can be set as needed, for example, it can be set to: 5, and the embodiment of the present invention does not limit this.

[0095] In an embodiment of the present invention, it is first determined whether the target object identified in each video sub-unit of the video to be identified belongs to a group of a specified age group. When the target objects identified in each video sub-unit all belong to the group of the specified age group, it is determined that the target object in the video to be identified belongs to the group of the specified age group. By detecting the divided minimum video units separately, the accuracy of target object identification is further improved.

[0096] Furthermore, when it is determined that the faces in each video sub-unit belong to a specified age group, it is determined that the faces in the video to be identified belong to the specified age group, that is, the video to be identified is a video in which the specified age group appears alone. For example, if the faces in each video sub-unit belong to minors, it is determined that the video to be identified is a video in which minors appear alone; if the faces in each video sub-unit belong to the elderly, it is determined that the video to be identified is a video in which the elderly appear alone.

[0097] If the number of subframes in which no human face is detected in any video subframe of any video subunit is greater than the seventh preset threshold, the video to be identified is determined to be a video of a group of people not in the specified age group appearing alone. The seventh preset threshold can be set according to the number of video frames in the frame sequence, for example, it can be set to half the number of video frames in the frame sequence, which is not limited in the embodiment of the present invention.

[0098] It should be noted that the method for identifying target objects in videos provided by the embodiments of the present invention can be performed before the video is released, or the specified age group can be identified in the released video, so as to retrieve the video in which the specified age group appears alone. In addition, in the automatic detection process of the video review device after the actual video is uploaded, since the video frames extracted after the same video is decoded need to be input as different detection models, but saved as files will lead to high storage space occupancy, therefore, in the embodiments of the present invention, it is considered to reuse the existing video frames in the decoding cluster. In the scenario of real-time video parallel review, the existing video frame sequence in the decoding cluster can be reused to make full use of the existing decoding resources. This can not only ensure the real-time performance of the decoded video frames, but also reduce the consumption of decoding resources caused by repeated decoding of the same video, thereby improving the utilization rate of decoding resources.

[0099] In a method for identifying a target object in a video provided by an embodiment of the present invention, a video review device determines the type of a video to be identified based on each video frame in an acquired frame sequence, wherein the frame sequence is a combination of video frames extracted by the video review device from the video to be identified in chronological order, and the type of the video to be identified represents the generation method of the video to be identified. Furthermore, at least one video subunit is obtained from the frame sequence according to the type of the video frame to be identified, and the video subunit contains video frames of the same scene. The target object area in each video subunit is detected respectively, and the age information of the target object in each target object area is predicted. According to the age information of the corresponding target object in the target object area of ​​each video frame subunit, it is identified whether the target object in the video to be identified belongs to a group of specified age group. Compared with the prior art, the method for identifying a target object in a video provided by an embodiment of the present invention automatically identifies whether the target object in the video to be identified belongs to a group of specified age group according to a combination of video frames extracted by the video review device from the video to be identified in chronological order, thereby effectively saving the manpower cost and time cost consumed by manual review.

[0100] Based on the same inventive concept, an embodiment of the present invention also provides a device for identifying a target object in a video. Since the principle of solving the problem by the above-mentioned device for identifying a target object in a video is similar to that of the method for identifying a target object in a video, the implementation of the above-mentioned device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0101] like Figure 7 As shown, it is a schematic diagram of the structure of a device for identifying a target object in a video provided by an embodiment of the present invention, which may include:

[0102] A determination unit 71 is used to determine the type of the video to be identified according to each video frame in the obtained frame sequence, wherein the frame sequence is a combination of video frames extracted from the video to be identified in time order, and the type of the video to be identified represents the generation method of the video to be identified;

[0103] An obtaining unit 72, configured to obtain at least one video sub-unit from the frame sequence according to the type of the video to be identified, wherein the video sub-unit includes video frames of the same scene;

[0104] The identification unit 73 is used to detect the target object area in each video sub-unit, predict the age information of the target object in each target object area, and identify whether the target object in the video to be identified belongs to a group of specified age group based on the age information of the corresponding target object.

[0105] Preferably, the video frame is generated after decoding the video to be identified;

[0106] The determination unit 71 is specifically used to perform edge detection on each of the video frames respectively, and when it is determined that the number of pixel rows of the boundaries detected from the preset rectangular area at the central axis of each of the video frames is greater than a first preset threshold, the video to be identified is determined to be a combined type video; when it is determined that the number of pixel rows of the boundaries detected from the preset rectangular area at the central axis of any video frame in the video frames is less than or equal to the first preset threshold, the similarity between every two adjacent video frames is determined; when it is determined that the similarity between any two adjacent video frames is greater than a second preset threshold, the video to be identified is determined to be a picture set type video; when it is determined that the similarity between every two adjacent video frames is less than or equal to the second preset threshold, the video to be identified is determined to be an independent and complete type video.

[0107] Preferably, the acquisition unit 72 is specifically used to, when it is determined that the video to be identified is a co-shot type video, use the column with the most pixels detected as the boundary in the preset rectangular area at the central axis of each video frame in the frame sequence as the dividing line of each video frame, and divide each video frame into two left and right video sub-frames; and respectively determine the left video sub-frame sequence and the right video sub-frame sequence as two video sub-units.

[0108] Preferably, the acquisition unit 72 is specifically used to, when it is determined that the video to be identified is a picture set type video, and when it is determined that the similarity between adjacent video frames is less than a third preset threshold, use the adjacent video frames as cutting points to segment the frame sequence; determine each video frame combination obtained after the segmentation as a video sub-unit, and determine the video frames in each video sub-unit as video sub-frames.

[0109] Preferably, the obtaining unit 72 is specifically configured to determine the frame sequence as a video sub-unit and determine the video frame in the video sub-unit as a video sub-frame when it is determined that the video to be identified is an independent complete type of video.

[0110] Preferably, the recognition unit 73 is specifically used to perform face detection on each video subframe of the video subframe according to the face detection model for each video subunit, and obtain face position information and face feature information in each video subframe, wherein the face position information includes a face rectangular bounding box that marks the face; calculate the intersection and union ratio of each face rectangular bounding box in each video subframe and each face rectangular bounding box in its adjacent video subframe; mark two face rectangular bounding boxes whose intersection and union ratio is greater than a fourth preset threshold as a continuous face detection pair; for each continuous face detection pair, when it is determined that the similarity of the face feature information corresponding to the two face rectangular bounding boxes of the face detection pair is greater than a fifth preset threshold, determine that the face detection pair detects the same face, otherwise, determine that the face detection pair detects different faces; record the face-related information detected from the video subunit.

[0111] Preferably, the recognition unit 73 is specifically used to perform age detection on each face detected in each video subframe of the video subunit according to the age detection model, and obtain age prediction values ​​corresponding to each face in each video subframe; for the same face detected from different video subframes in the video subunit, use a preset clustering algorithm to eliminate outliers in the age prediction values ​​of the same face.

[0112] Preferably, the recognition unit 73 is specifically used for, for a video sub-unit in which the number of detected faces is 1, when it is determined that the minimum value of the age prediction value and the mode of the age prediction value for the face are both within a specified age group, then it is determined that the target object in the video sub-unit belongs to the group of the specified age group, and the mode of the age prediction value represents the age prediction value with the largest number of occurrences; for a video sub-unit in which the number of detected faces is greater than 1, when it is determined that the minimum value of the age prediction value and the mode of the age prediction value for each face are both within the specified age group, and the number of frames in which the age prediction value of the face detected from the video sub-unit is outside the specified age group is less than a sixth preset threshold, then it is determined that the target object in the video sub-unit belongs to the group of the specified age group; when it is determined that the target object in each video sub-unit belongs to the group of the specified age group, it is determined that the target object in the video to be identified belongs to the group of the specified age group.

[0113] Preferably, the determination unit 71 is specifically used to determine the similarity between two adjacent video frames through the following steps: calculating the DHash value of each of the two adjacent video frames according to the differential hash DHash algorithm; determining the Hamming distance between the DHash values ​​of the two adjacent video frames; and determining the similarity between the two adjacent video frames according to the Hamming distance.

[0114] Based on the same technical concept, the embodiment of the present invention further provides an electronic device 800, referring to Figure 8 As shown, the electronic device 800 is used to implement the method for identifying a target object in a video described in the above method embodiment. The electronic device 800 of this embodiment may include: a memory 801, a processor 802, and a computer program stored in the memory and executable on the processor, such as a program for identifying a target object in a video. When the processor executes the computer program, the steps in the above methods for identifying a target object in a video are implemented, such as Figure 2 Alternatively, when the processor executes the computer program, the functions of the modules / units in the above-mentioned device embodiments are implemented, for example, 71.

[0115] The specific connection medium between the memory 801 and the processor 802 is not limited in the embodiment of the present invention. Figure 8 In the embodiment, the memory 801 and the processor 802 are connected via a bus 803. The bus 803 is Figure 8 The connection between other components is shown by bold lines, and is not intended to be limiting. The bus 803 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0116] The memory 801 may be a volatile memory, such as a random-access memory (RAM); the memory 801 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), or the memory 801 may be any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 801 may be a combination of the above memories.

[0117] Processor 802, configured to implement Figure 2A method for identifying a target object in a video is shown, comprising:

[0118] The processor 802 is used to call the computer program stored in the memory 801 to execute the following Figure 2 The step S21 shown in the figure determines the type of the video to be identified according to each video frame in the obtained frame sequence, wherein the frame sequence is a combination of video frames extracted from the video to be identified in time order, step S22 obtains at least one video sub-unit from the frame sequence according to the type of the video to be identified, and step S23 detects the target object area in the video sub-unit for each video sub-unit, predicts the age information of the target object in each target object area, and identifies whether the target object in the video to be identified belongs to a group of specified age group according to the age information of the corresponding target object.

[0119] An embodiment of the present application also provides a computer-readable storage medium that stores computer-executable instructions required to execute the above-mentioned processor, which includes a program required to execute the above-mentioned processor.

[0120] In some possible implementations, various aspects of the method for identifying a target object in a video provided by the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the method for identifying a target object in a video according to various exemplary embodiments of the present invention described above in this specification. For example, the electronic device may execute the following steps: Figure 2 The step S21 shown in the figure determines the type of the video to be identified according to each video frame in the obtained frame sequence, wherein the frame sequence is a combination of video frames extracted from the video to be identified in time order, step S22 obtains at least one video sub-unit from the frame sequence according to the type of the video to be identified, and step S23 detects the target object area in the video sub-unit for each video sub-unit, predicts the age information of the target object in each target object area, and identifies whether the target object in the video to be identified belongs to a group of specified age group according to the age information of the corresponding target object.

[0121] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0123] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0125] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0126] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for identifying a target object in a video, characterized in that: include: Determine the type of the video to be identified according to each video frame in the obtained frame sequence, wherein the frame sequence is a combination of video frames extracted from the video to be identified in time order, and the type of the video to be identified represents the generation method of the video to be identified; Obtaining at least one video subunit from the frame sequence according to the type of the video to be identified, wherein the video subunit includes video frames of the same scene; At least one video sub-unit is obtained from the frame sequence according to the type of the video to be identified, specifically including: when it is determined that the video to be identified is a co-production type video, the column with the most pixels detected from the boundary in the preset rectangular area at the central axis of each video frame in the frame sequence is used as the dividing line of each video frame, each video frame is divided into two left and right video sub-frames, and the left video sub-frame sequence and the right video sub-frame sequence are respectively determined as two video sub-units; when it is determined that the video to be identified is a picture set type video, when it is determined that the similarity between adjacent video frames is less than a third preset threshold, the adjacent video frames are used as cutting points to segment the frame sequence, each video frame combination obtained after segmentation is respectively determined as a video sub-unit, and the video frames in each video sub-unit are determined as video sub-frames; when it is determined that the video to be identified is an independent and complete type video, the frame sequence is determined as a video sub-unit, and the video frames in the video sub-unit are determined as video sub-frames; For each video sub-unit, the target object area in the video sub-unit is detected, and the age information of the target object in each target object area is predicted, and based on the corresponding age information of the target object, it is identified whether the target object in the video to be identified belongs to a group of a specified age group.

2. The method according to claim 1, characterized in that The video frame is generated after the video to be identified is decoded; Determining the type of the video to be identified according to each video frame in the obtained frame sequence specifically includes: Performing edge detection on each of the video frames, and when it is determined that the number of pixel rows of the boundaries detected in the preset rectangular area at the central axis of each of the video frames is greater than a first preset threshold, determining that the video to be identified is a co-shot type video; When it is determined that the number of pixel rows in which a boundary is detected within the preset rectangular area at the central axis of any video frame among the video frames is less than or equal to the first preset threshold, determining the similarity between every two adjacent video frames; When it is determined that the similarity between any two adjacent video frames is greater than a second preset threshold, determining that the video to be identified is a picture set type video; When it is determined that the similarities between each two adjacent video frames are less than or equal to the second preset threshold, it is determined that the video to be identified is an independent and complete type of video.

3. The method according to claim 1 or 2, characterized in that For each video sub-unit, detecting a target object area in the video sub-unit specifically includes: For each video sub-unit, perform face detection on each video sub-frame of the video sub-unit according to the face detection model to obtain face position information and face feature information in each video sub-frame, wherein the face position information includes a face rectangular bounding box for marking the face; Calculate the intersection-and-union ratio of each face rectangular bounding box in each video subframe and each face rectangular bounding box in its adjacent video subframe respectively; Marking two face rectangular bounding boxes whose intersection-over-union ratio is greater than a fourth preset threshold as a continuous face detection pair; For each continuous face detection pair, when it is determined that the similarity of the facial feature information corresponding to the two face rectangular bounding boxes of the face detection pair is greater than a fifth preset threshold, it is determined that the face detection pair detects the same face; otherwise, it is determined that the face detection pair detects different faces; Record the face-related information detected from the video subunit.

4. The method according to claim 3, characterized in that Predicting the age information of the target object in each target object area includes: For each video sub-unit, performing age detection on each face detected in each video sub-frame of the video sub-unit according to the age detection model, and obtaining an age prediction value corresponding to each face in each video sub-frame; For the same human face detected from different video subframes in the video subunit, a preset clustering algorithm is used to remove outliers in each age prediction value of the same human face.

5. The method according to claim 4, characterized in that According to the age information of the corresponding target object, identifying whether the target object in the to-be-identified video belongs to a group of a specified age group specifically includes: For a video subunit in which the number of detected human faces is 1, when it is determined that the minimum value of the age prediction value for the human face and the mode of the age prediction value are both within the specified age range, it is determined that the target object in the video subunit belongs to the group of the specified age range, and the mode of the age prediction value represents the age prediction value with the largest number of occurrences; For a video subunit in which the number of detected faces is greater than 1, when it is determined that the minimum value of the age prediction value for each face and the mode of the age prediction value are both within the specified age range, and the number of frames of the age prediction value of the face detected from the video subunit in the specified age range is less than a sixth preset threshold, it is determined that the target object in the video subunit belongs to the group of the specified age range; When it is determined that the target object in each video sub-unit belongs to the specified age group, it is determined that the target object in the to-be-identified video belongs to the specified age group.

6. The method according to claim 2, characterized in that The similarity between two adjacent video frames is determined by the following steps: Calculate the DHash values ​​of two adjacent video frames according to the difference hash DHash algorithm; Determining a Hamming distance between DHash values ​​of the two adjacent video frames; The similarity between the two adjacent video frames is determined according to the Hamming distance.

7. A device for identifying a target object in a video, characterized in that: include: a determination unit, configured to determine the type of the video to be identified according to each video frame in the obtained frame sequence, wherein the frame sequence is a combination of video frames extracted from the video to be identified in time sequence, and the type of the video to be identified represents a generation method of the video to be identified; an obtaining unit, configured to obtain at least one video sub-unit from the frame sequence according to the type of the video to be identified, wherein the video sub-unit includes video frames of the same scene; The obtaining unit is specifically used for, when it is determined that the video to be identified is a co-production type video, using the column with the most pixels detected as the boundary in the preset rectangular area at the central axis of each video frame in the frame sequence as the dividing line of each video frame, dividing each video frame into two left and right video sub-frames, and respectively determining the left video sub-frame sequence and the right video sub-frame sequence as two video sub-units; when it is determined that the video to be identified is a picture set type video, when it is determined that the similarity between adjacent video frames is less than a third preset threshold, using the space between the adjacent video frames as a cutting point to segment the frame sequence, determining each video frame combination obtained after segmentation as a video sub-unit, and determining the video frames in each video sub-unit as video sub-frames; when it is determined that the video to be identified is an independent and complete type video, determining the frame sequence as a video sub-unit, and determining the video frames in the video sub-unit as video sub-frames; The identification unit is used to detect the target object area in each video sub-unit, predict the age information of the target object in each target object area, and identify whether the target object in the video to be identified belongs to a group of specified age group based on the age information of the corresponding target object.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for identifying a target object in a video as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for identifying a target object in a video as described in any one of claims 1 to 6 are implemented.

Images