Video content area detection method and device, electronic equipment and storage medium

Abstract

The invention discloses a video content area detection method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the video frame sequence analysis of a to-be-detected video, and obtaining the content area positioning information of a plurality of video frames in the to-be-detected video; then carrying out content region positioning on the video frame sequence, and carrying out video classification screening on a to-be-detected video according to a video classification rule; and finally, carrying out aggregation analysis on the video frame sequence content region of the target video to obtain a video-level content region. According to the method, by analyzing the video frame sequence and introducing a classification screening and clustering aggregation mechanism, error content region detection generated by instantaneous interference is effectively filtered, and a stable and consistent video-level detection result in the time dimension is obtained. Meanwhile, the technical scheme provided by the invention is high in adaptability, can adapt to specific requirements of different application scenes, and has good generalization ability.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and more specifically to a method, apparatus, electronic device, and storage medium for detecting video content regions. Background Technology

[0002] In applications such as video processing, content moderation, intelligent editing, and adaptive playback for both portrait and landscape orientations, accurately defining the effective content area of ​​a video is a crucial preprocessing step. For example, when removing black borders, cropping irrelevant content, or performing enhancement analysis on specific areas, it is essential to first determine the effective content area of ​​the video.

[0003] In existing technologies, common methods include rule-based cropping (such as directly cropping fixed pixels at the edges of the video), color statistics-based detection (such as detecting large areas of black or solid-color borders), and single-frame object detection methods. However, these methods have significant shortcomings: rule-based cropping cannot adapt to videos with changing content positions; color-based detection is prone to misclassifying dark content in the frame as black borders; and single-frame detection methods ignore the continuity of the video in the temporal dimension, resulting in unstable judgment results that are easily affected by noise in individual frames (such as brief subtitles, logos, or image jitter), causing the final output content region bounding boxes to jump along the timeline, making it impossible to form a stable and consistent video-level detection result.

[0004] Therefore, existing technologies have failed to provide an adaptive method that can comprehensively consider the temporal coherence and spatial distribution characteristics of video, thereby robustly and accurately outputting video-level content region detection. Summary of the Invention

[0005] In view of the above-mentioned problems of existing methods, this invention proposes a video content region detection method, apparatus, electronic device and storage medium.

[0006] Specifically, the embodiments of the present invention provide the following technical solutions:

[0007] In a first aspect, embodiments of the present invention provide a video content region detection method, including:

[0008] Video frame sequence analysis involves sampling the video to be detected to obtain a video frame sequence, and then obtaining the content region positioning information of the video frame sequence.

[0009] Video classification and filtering involves selecting target videos from the videos to be detected based on the location features of the content regions in the video frame sequence and according to preset rules.

[0010] Video-level content region calculation: For the target video, aggregate and analyze the content region positioning information of its video frame sequence to obtain video-level content region positioning information.

[0011] Secondly, embodiments of the present invention provide a video content region detection device, comprising:

[0012] The video frame sequence analysis module is used to sample the video to be detected to obtain a video frame sequence and acquire the content region positioning information of the video frame sequence.

[0013] The video classification and filtering module is used to filter out target videos from the videos to be detected according to preset rules based on the location features of the content regions in the video frame sequence.

[0014] The video-level content region calculation module is used to aggregate and analyze the content region positioning information of the video frame sequence of the target video to obtain video-level content region positioning information.

[0015] Thirdly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the video content region detection method as described in the first aspect.

[0016] Fourthly, embodiments of the present invention also provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video content region detection method as described in the first aspect.

[0017] As can be seen from the above technical solution, the present invention has the following beneficial effects: The technical solution provided by the present invention analyzes video frame sequences and introduces classification filtering and clustering aggregation mechanisms to effectively filter out erroneous content region location information caused by transient interference, thereby obtaining stable and consistent video-level detection results in the time dimension. Furthermore, the technical solution provided by the present invention is highly adaptable and can meet the specific needs of different application scenarios, exhibiting good generalization ability. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a video content region detection method provided in an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of the structured correction of video frame sequence content region positioning information provided in an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram illustrating the alignment and correction of the content region positioning information of a video frame sequence with the video frame boundary according to an embodiment of the present invention.

[0022] Figure 4 This is a schematic diagram of video frame classification provided in an embodiment of the present invention.

[0023] Figure 4 (a) is a schematic diagram of video frame classification with only top and bottom black borders provided in an embodiment of the present invention.

[0024] Figure 4 (b) is a schematic diagram of video frame classification with only left and right black borders provided in an embodiment of the present invention.

[0025] Figure 4 (c) is a schematic diagram of video frame classification with black borders on all four sides provided in an embodiment of the present invention.

[0026] Figure 4 (d) is a schematic diagram of full-screen video frame classification provided in an embodiment of the present invention.

[0027] Figure 4 (e) in the figure is a schematic diagram of other types of video frame classification provided in an embodiment of the present invention.

[0028] Figure 5 This is a schematic diagram of the structure of a video content region detection device provided in an embodiment of the present invention.

[0029] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0030] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. The following examples are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0031] Figure 1 A flowchart of a video content region detection method according to an embodiment of the present invention is shown, as follows: Figure 1 As shown, the video content region detection method provided in this embodiment of the invention specifically includes the following:

[0032] Step 1: Video frame sequence analysis. The video to be detected is sampled to obtain a video frame sequence, and the content region positioning information of the video frame sequence is obtained.

[0033] In this step, it should be noted that the present invention does not limit the specific sampling method, and video frame sequences can be obtained by sampling according to actual needs.

[0034] For example, fixed-interval sampling can be performed, and the sampled frames are determined according to the following formula:

[0035]

[0036] Among them, f i Let be the index of the i-th sampled frame, T be the total number of frames in the video to be detected, and K be the preset number of samples.

[0037] For example, the total number of video frame sequences can be preset in advance, and random sampling, key frame sampling, or sampling after removing the first and last video frames can be performed on the video to be detected.

[0038] Furthermore, it should be noted in this step that the present invention does not limit the specific method for obtaining the content region positioning information of the video frame sequence. It can be a traditional detection method, such as color statistics-based detection; or a target detection algorithm based on neural networks, such as the YOLO (You Only Look Once) series of detection models.

[0039] In this embodiment, it can be understood that the positioning information can be a rectangular bounding box defined by the coordinates of the upper left and lower right corners, or a set of polygon vertex coordinates or other representations that can determine the content area. This invention does not limit this.

[0040] Furthermore, in order to determine the positioning information more accurately, after obtaining the content region positioning information of the video frame sequence, the method further includes optimizing and correcting the content region positioning information.

[0041] The optimization and correction methods include, but are not limited to, using one or more of the following methods:

[0042] The content region positioning information is corrected based on the pixel features of the video frame sequence.

[0043] Based on the spatial distribution consistency of content region positioning information in video frame sequences, the content region positioning information is structurally corrected.

[0044] The content region positioning information of the video frame sequence is aligned and corrected with the video frame boundaries.

[0045] The correction of content region positioning information based on pixel features of video frame sequences includes:

[0046] Extract the luminance channel of the video frame sequence in the YUV color space.

[0047] The analysis calculates the pixel brightness gradient within a preset range centered on the boundary pixel of the content area.

[0048] The system scans and finds the pixel with the most significant and continuous pixel brightness gradient, then adjusts the content region boundary to that pixel position, making the content region boundary more accurately match the actual content region boundary.

[0049] The spatial distribution consistency of content region positioning information based on video frame sequences is used to structurally correct the content region positioning information, including:

[0050] Using the geometric center of the video frame sequence as a reference, the overall distribution trend of the content area of ​​the video frame sequence in the vertical and / or horizontal directions is statistically analyzed.

[0051] When more than a preset proportion of content regions are concentrated in the upper or lower half of a video frame, it can be presumed that the content regions of the video frame should have a vertically symmetrical structure. The content regions of the video frame should then be symmetrically adjusted according to this vertical symmetry. Figure 2 As shown.

[0052] Similarly, when more than a preset proportion of content areas are concentrated in the left or right half of a video frame, it can be inferred that the content areas of the video frame should be symmetrical in the horizontal direction, and the content areas of the video frame should be corrected symmetrically according to the symmetrical structure.

[0053] The step involves aligning and correcting the content region positioning information of the video frame sequence with the video frame boundaries, such as... Figure 3 As shown, it includes:

[0054] If the distance difference between one or more edges of the content region boundary of the video frame sequence and the corresponding edge of the video frame is less than a first preset threshold, then the positioning information of that edge is attached to the positioning information of the corresponding edge of the video frame, that is, the positioning information of that edge is corrected to match the positioning information of the corresponding edge of the video frame.

[0055] If the distance difference between any side of the content region boundary of the video frame sequence and the corresponding side of the video frame is less than the first preset threshold, then the positioning information of that side remains unchanged.

[0056] Step 2, video classification and filtering: based on the positional features of the content regions in the video frame sequence, target videos are filtered out from the videos to be detected according to preset rules.

[0057] Step 21: Classify and label the video frame sequence according to the relative positional relationship between the content area and the invalid area of ​​the video frame sequence.

[0058] The invalid area can be customized according to the actual situation, such as black borders, solid color areas, blurred areas, fixed logo areas, etc. in video frames.

[0059] Furthermore, during the classification and labeling process, to more clearly determine whether invalid regions exist, horizontal and vertical boundary distance thresholds can be set. These boundary distance thresholds can be fixed values ​​or adaptive ratios set according to different video frame resolutions and sizes. The specific values ​​are not limited in this invention and can be set according to actual conditions. For example, the horizontal boundary distance threshold can be set to 100, and the vertical boundary distance threshold to 50; alternatively, the horizontal boundary distance threshold can be set to 10% of the vertical boundary length of the video frame, and the vertical boundary distance threshold can be set to 10% of the horizontal boundary length of the video frame.

[0060] When the distance difference between one or more edges of the current video frame content region boundary and the corresponding edge of the video frame is greater than a preset boundary distance threshold, it can be determined that there is an invalid region outside the one or more edges of the current video frame content region boundary; when the distance difference between any edge of the current video frame content region boundary and the corresponding edge of the video frame is greater than a preset boundary distance threshold, it can be determined that there is no invalid region in the current video frame content region.

[0061] In this step, the relative positional relationship between the content area and the invalid area of ​​the video frame sequence represents the overall layout tendency of the content area in the video frame picture. For example: the content area is centered and displayed in full screen, the content area has a black border at the top, the content area is centered but has black borders at the top and bottom, the content is centered but has blurred areas on the top, bottom, left and right sides, etc.

[0062] For example, to filter out target videos with a layout where the content area is centered but has black borders on all four sides (top, bottom, left, and right), the video frames can be divided into the following five categories, such as... Figure 4 As shown:

[0063] (1) Video frames with only top and bottom black borders: Invalid areas exist at the top and bottom boundaries of the video frame, and the video frame appears as pairs of horizontal black borders at the top and bottom, such as Figure 4 As shown in (a) in the figure.

[0064] (2) Video frames with only left and right black borders: There are invalid areas on the left and right edges of the video frame, and the video frame appears as pairs of vertical black borders on the left and right, such as Figure 4 As shown in (b) of the diagram.

[0065] (3) Video frames with black borders on all four sides: Invalid areas exist on the top, bottom, left, and right boundaries of the video frame. The video frame appears as if the content area is "sandwiched" by the four sides of the structural black borders, forming a typical sandwich structure, such as Figure 4 As shown in (c) in the figure.

[0066] (4) Full-screen video frame: There are no invalid areas on the top, bottom, left, and right boundaries of the video frame, and the video frame is displayed in full screen, such as... Figure 4 As shown in (d) in the figure.

[0067] (5) Other types of video frames: Invalid regions exist only outside a single boundary, or, no invalid regions exist outside a single boundary, or, black borders exist only outside one boundary in both the vertical and horizontal directions, and the video frame displays right-angled (L-shaped) black borders, such as... Figure 4 As shown in (e) in the diagram.

[0068] Step 22: Based on the classification results, assign weight scores to video frames of each category.

[0069] In this step, it is understood that, in order to filter out target videos, higher scores can be set for target-classified video frames, and correspondingly, lower scores can be set for non-target video frames. The specific scores are not limited in this invention and can be set according to actual needs.

[0070] For example, to filter out target videos with a layout where the content area is centered but there are black bars on all four sides (top, bottom, left, and right), the following weight scores are assigned to video frames with different category labels: Video frames with black bars on all four sides are assigned a higher weight score to represent the target category video frames; video frames with only black bars on the top and bottom or only black bars on the left and right sides are assigned a medium weight score to represent near-target category video frames; and full-screen video frames or other types of video frames are assigned the lowest weight score to represent non-target category video frames. The weight scores can be set as follows: video frames with black bars on all four sides (1 point) > video frames with only black bars on the top and bottom (0.5 points) = video frames with only black bars on the left and right sides (0.5 points) > full-screen video frames (0 points) = other video frames (0 points).

[0071] Step 23: Calculate the comprehensive score of the video to be detected by averaging the scores of all video frames in the video frame sequence.

[0072] In this step, it should be noted that the method for calculating the mean based on the scores of all video frames in the video frame sequence can be selected according to the actual situation, and this invention does not impose any restrictions. Exemplary mean calculation methods include, but are not limited to: arithmetic mean, weighted average, and median.

[0073] Step 24: Compare the comprehensive score with the second preset threshold to determine whether the video to be detected belongs to the target video.

[0074] If the overall score is greater than the second preset threshold, the video to be detected is determined to belong to the target video; otherwise, it does not belong to the target video.

[0075] Step 3: Video-level content region calculation. For the target video, aggregate and analyze the content region positioning information of its video frame sequence to obtain video-level content region positioning information.

[0076] Specifically, for the target video, the aggregation and analysis of its video frame sequence content region location information to obtain video-level content region location information includes:

[0077] From the content region positioning information of the video frame sequence, any video frame is selected as the reference video frame, and its content region positioning information is used as the reference positioning information.

[0078] Using the reference positioning information as a reference, video frames in the video frame sequence whose video frame content region positioning information matches the reference positioning information are clustered into the same cluster set.

[0079] The "match" means that the difference between the positioning information of any side of the content area of ​​the current video frame and the positioning information of the corresponding side in the reference video frame is within a preset difference range.

[0080] From all cluster sets, select the cluster set with the most video frames as the target cluster set.

[0081] The target cluster set represents the content region location information that is most stable in time and has the highest frequency of occurrence.

[0082] For all content regions in the target cluster set, the mean values ​​of the positioning information on the top, bottom, left, and right sides are calculated. The statistical results are combined to generate new content region positioning information, which is the video-level content region positioning information of the video to be detected.

[0083] It should be noted that the method used for calculating the mean is not limited in this invention, and the method can be selected according to the actual situation. For example, the method for calculating the mean includes, but is not limited to: arithmetic mean, weighted average, and median.

[0084] Figure 5 A schematic diagram of a video content region detection device according to an embodiment of the present invention is shown, as follows: Figure 5 As shown, the video content region detection device provided in this embodiment of the invention specifically includes the following components:

[0085] The video frame sequence analysis module is used to sample the video to be detected to obtain a video frame sequence and acquire the content region positioning information of the video frame sequence.

[0086] In this module, it should be noted that the present invention does not limit the specific sampling method, and video frame sequences can be obtained by sampling according to actual needs.

[0087] For example, fixed-interval sampling can be performed, and the sampled frames are determined according to the following formula:

[0088]

[0089] Among them, f i Let be the index of the i-th sampled frame, T be the total number of frames in the video to be detected, and K be the preset number of samples.

[0090] For example, the total number of video frame sequences can be preset in advance, and random sampling, key frame sampling, or sampling after removing the first and last video frames can be performed on the video to be detected.

[0091] In this unit, it should be noted that the present invention does not limit the specific method for obtaining the content region positioning information of the video frame sequence. It can be a traditional detection method, such as color statistics-based detection; or a target detection algorithm based on neural networks, such as the YOLO (You Only Look Once) series of detection models.

[0092] In this embodiment, it can be understood that the positioning information can be a rectangular bounding box defined by the coordinates of the upper left and lower right corners, or a set of polygon vertex coordinates or other representations that can determine the content area. This invention does not limit this.

[0093] Furthermore, in order to determine the positioning information more accurately, the video frame sequence analysis module also includes a positioning information correction unit.

[0094] The positioning information correction unit is used to correct the positioning information using one or more of the following methods after obtaining the content region positioning information of the video frame sequence:

[0095] The content region positioning information is corrected based on the pixel features of the video frame sequence.

[0096] Based on the spatial distribution consistency of content region positioning information in video frame sequences, the content region positioning information is structurally corrected.

[0097] The content region positioning information of the video frame sequence is aligned and corrected with the video frame boundaries.

[0098] The correction of content region positioning information based on pixel features of video frame sequences includes:

[0099] Extract the luminance channel of the video frame sequence in the YUV color space.

[0100] The analysis calculates the pixel brightness gradient within a preset range centered on the boundary pixel of the content area.

[0101] The system scans and finds the pixel with the most significant and continuous pixel brightness gradient, then adjusts the content region boundary to that pixel position, making the content region boundary more accurately match the actual content region boundary.

[0102] The spatial distribution consistency of content region positioning information based on video frame sequences is used to structurally correct the content region positioning information, including:

[0103] Using the geometric center of the video frame sequence as a reference, the overall distribution trend of the content area of ​​the video frame sequence in the vertical and / or horizontal directions is statistically analyzed.

[0104] When more than a preset proportion of content areas are concentrated in the upper or lower half of a video frame, it can be presumed that the content areas of the video frame should have a vertically symmetrical structure, and the content areas of the video frame should be corrected symmetrically according to the vertically symmetrical structure.

[0105] Similarly, when more than a preset proportion of content areas are concentrated in the left or right half of a video frame, it can be inferred that the content areas of the video frame should be symmetrical in the horizontal direction, and the content areas of the video frame should be corrected symmetrically according to the symmetrical structure.

[0106] The step of aligning and correcting the content region positioning information of the video frame sequence with the video frame boundaries includes: If the distance difference between one or more edges of the content region boundary of the video frame sequence and the corresponding edge of the video frame is less than a first preset threshold, then the positioning information of that edge is attached to the positioning information of the corresponding edge of the video frame, that is, the positioning information of that edge is corrected to match the positioning information of the corresponding edge of the video frame.

[0107] If the distance difference between any side of the content region boundary of the video frame sequence and the corresponding side of the video frame is less than the first preset threshold, then the positioning information of that side remains unchanged.

[0108] The video classification and filtering module is used to filter out target videos from the videos to be detected according to preset rules based on the location features of the content regions in the video frame sequence.

[0109] The video classification and filtering module includes:

[0110] The video frame classification and labeling unit is used to classify and label the video frame sequence according to the relative positional relationship between the content area and the invalid area of ​​the video frame sequence.

[0111] The invalid area can be customized according to the actual situation, such as black borders, solid color areas, blurred areas, fixed logo areas, etc. in video frames.

[0112] Furthermore, during the classification and labeling process, to more clearly determine whether invalid regions exist, horizontal and vertical boundary distance thresholds can be set. These boundary distance thresholds can be fixed values ​​or adaptive ratios set according to different video frame resolutions and sizes. The specific values ​​are not limited in this invention and can be set according to actual conditions. For example, the horizontal boundary distance threshold can be set to 100, and the vertical boundary distance threshold to 50; alternatively, the horizontal boundary distance threshold can be set to 10% of the vertical boundary length of the video frame, and the vertical boundary distance threshold can be set to 10% of the horizontal boundary length of the video frame.

[0113] When the distance difference between one or more edges of the current video frame content region boundary and the corresponding edge of the video frame is greater than a preset boundary distance threshold, it can be determined that there is an invalid region outside the one or more edges of the current video frame content region boundary; when the distance difference between any edge of the current video frame content region boundary and the corresponding edge of the video frame is greater than a preset boundary distance threshold, it can be determined that there is no invalid region in the current video frame content region.

[0114] Understandably, the relative positional relationship between the content area and invalid area of ​​the video frame sequence represents the overall layout tendency of the content area in the video frame, such as: the content area is centered and displayed in full screen, the content area has a black border at the top, the content area is centered but has black borders at the top and bottom, the content is centered but has blurred areas on the top, bottom, left and right sides, etc.

[0115] For example, to filter out target videos with a layout where the content area is centered but there are black borders on all four sides (top, bottom, left, and right), the video frames can be divided into the following five categories:

[0116] (1) Video frames with only black borders at the top and bottom: There are invalid areas at the top and bottom boundaries of the video frame, and the video frame appears as pairs of horizontal black borders at the top and bottom.

[0117] (2) Video frames with only black borders on the left and right: There are invalid areas on the left and right edges of the video frame, and the video frame appears as a pair of vertical black borders on the left and right.

[0118] (3) Video frames with black borders on all four sides: There are invalid areas on the top, bottom, left and right boundaries of the video frame. The video frame appears as if the content area is "squeezed" by the four sides of the structural black border, forming a typical sandwich structure.

[0119] (4) Full-screen video frame: There are no invalid areas on the top, bottom, left and right boundaries of the video frame, and the video frame is displayed in full screen.

[0120] (5) Other types of video frames: There is an invalid area only outside a single boundary, or there is no invalid area outside a single boundary, or there is a black border outside only one boundary in the vertical and horizontal directions respectively, and the video frame appears to have a right-angled (L-shaped) black border.

[0121] The scoring unit is used to assign weight scores to video frames of each category based on the classification results.

[0122] In this unit, it is understood that, in order to filter out target videos, higher scores can be set for target-classified video frames, and correspondingly, lower scores can be set for non-target video frames. The specific scores are not limited by this invention and can be set according to actual needs.

[0123] For example, to filter out target videos with a layout where the content area is centered but there are black bars on all four sides (top, bottom, left, and right), the following weight scores are assigned to video frames with different category labels: Video frames with black bars on all four sides are assigned a higher weight score to represent the target category video frames; video frames with only black bars on the top and bottom or only black bars on the left and right sides are assigned a medium weight score to represent near-target category video frames; and full-screen video frames or other types of video frames are assigned the lowest weight score to represent non-target category video frames. The weight scores can be set as follows: video frames with black bars on all four sides (1 point) > video frames with only black bars on the top and bottom (0.5 points) = video frames with only black bars on the left and right sides (0.5 points) > full-screen video frames (0 points) = other video frames (0 points).

[0124] The scoring calculation unit is used to calculate the comprehensive score of the video to be detected by averaging the scores of all video frames in the video frame sequence.

[0125] In this unit, it should be noted that the method for calculating the mean based on the scores of all video frames in the video frame sequence can be selected according to the actual situation, and this invention does not impose any restrictions. Exemplary mean calculation methods include, but are not limited to: arithmetic mean, weighted average, and median.

[0126] The determination unit is used to compare the comprehensive score with a second preset threshold to determine whether the video to be detected belongs to the target video.

[0127] If the overall score is greater than the second preset threshold, the video to be detected is determined to belong to the target video; otherwise, it does not belong to the target video.

[0128] The video-level content region calculation module is used to aggregate and analyze the content region positioning information of the video frame sequence of the target video to obtain video-level content region positioning information.

[0129] The video-level content region calculation module includes:

[0130] The reference selection unit is used to select any video frame as a reference video frame from the content region positioning information of the video frame sequence, and its content region positioning information is used as the reference positioning information.

[0131] The clustering unit is used to select video frames in the video frame sequence whose video frame content region positioning information matches the benchmark positioning information and cluster them into the same cluster set, with the benchmark positioning information as a reference.

[0132] The "match" means that the difference between the positioning information of any side of the content area of ​​the current video frame and the positioning information of the corresponding side in the reference video frame is within a preset difference range.

[0133] The target cluster set selection unit is used to select the cluster set with the most video frames from all cluster sets as the target cluster set.

[0134] The target cluster set represents the content region location information that is most stable in time and has the highest frequency of occurrence.

[0135] The aggregation calculation unit is used to perform mean calculation on the positioning information of the top, bottom, left and right sides of all content regions in the target cluster set, and combine the statistical results to generate new content region positioning information, which is the video-level content region positioning information of the video to be detected.

[0136] It should be noted that the method used for calculating the mean is not limited in this invention, and the method can be selected according to the actual situation. For example, the method for calculating the mean includes, but is not limited to: arithmetic mean, weighted average, and median.

[0137] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, such as... Figure 6 As shown, the electronic device specifically includes the following components: a processor 601, a memory 602, a communication interface 603, and a communication bus 604.

[0138] The processor 601, memory 602, and communication interface 603 communicate with each other through the communication bus 604; the communication interface 603 is used to realize information transmission between the devices.

[0139] The processor 601 is used to call the computer program in the memory 602, and when the processor executes the computer program, it implements all the steps of the above-described video content region detection method.

[0140] Based on the same inventive concept, another embodiment of the present invention provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements all the steps of the above-described video content region detection method.

[0141] Furthermore, in embodiments of the present invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. In the description herein, phrases such as "in one embodiment," "in another embodiment," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment, which are included in at least one embodiment of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments. Furthermore, those skilled in the art can combine and integrate the different embodiments and features described in this specification without contradiction.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting video content regions, characterized in that, include: Video frame sequence analysis involves sampling the video to be detected to obtain a video frame sequence and acquiring the content region positioning information of the video frame sequence. Video classification and filtering: Based on the positional features of the content regions in a video frame sequence, target videos are filtered out from the videos to be detected according to preset rules; Video-level content region calculation: For the target video, aggregate and analyze the content region positioning information of its video frame sequence to obtain video-level content region positioning information.

2. The video content region detection method according to claim 1, characterized in that, After obtaining the content region location information of the video frame sequence, the method further includes correcting the location information using one or more of the following methods: Based on the pixel features of the video frame sequence, the content area positioning information is corrected; Based on the spatial distribution consistency of content region positioning information in video frame sequences, the content region positioning information is structurally corrected. The content region positioning information of the video frame sequence is aligned and corrected with the video frame boundaries.

3. The video content region detection method according to claim 2, characterized in that, The correction of content region positioning information based on pixel features of video frame sequences includes: Extract the luminance channel of the video frame sequence in the YUV color space; Analyze and calculate the pixel brightness gradient within a preset range centered on the boundary pixel of the content region; The pixel with the most significant and continuous pixel brightness gradient is scanned and the boundary of the content region is adjusted to the position of that pixel.

4. The video content region detection method according to claim 2, characterized in that, The spatial distribution consistency of content region positioning information based on video frame sequences is used to structurally correct the content region positioning information, including: Using the geometric center of the video frame sequence as a reference, statistically analyze the overall distribution trend of the video frame sequence content area in the vertical and / or horizontal directions: When more than a preset proportion of content areas are concentrated in the upper or lower half of a video frame, it can be inferred that the content areas of the video frame should have a vertically symmetrical structure, and the content areas of the video frame should be corrected according to the vertically symmetrical structure. When more than a preset proportion of content areas are concentrated in the left or right half of a video frame, it can be presumed that the content areas of the video frame should have a horizontally symmetrical structure, and the content areas of the video frame should be corrected symmetrically according to the horizontally symmetrical structure.

5. The video content region detection method according to claim 2, characterized in that, The step of aligning and correcting the content region positioning information of the video frame sequence with the video frame boundaries includes: If the distance difference between one or more edges of the content region boundary of the video frame sequence and the corresponding edge of the video frame is less than the first preset threshold, then the positioning information of the edge is attached to the positioning information of the corresponding edge of the video frame, that is, the positioning information of the edge is corrected to the positioning information of the corresponding edge of the video frame. If the distance difference between any side of the content region boundary of the video frame sequence and the corresponding side of the video frame is less than the first preset threshold, then the positioning information of that side remains unchanged.

6. The video content region detection method according to claim 1, characterized in that, The method of selecting target videos from the video to be detected based on the location features of content regions in the video frame sequence according to preset rules includes: Based on the relative positional relationship between the content region and invalid region of the video frame sequence, the video frame sequence is classified and labeled. Based on the classification results, a weight score is assigned to each category of video frames. The comprehensive score of the video to be detected is calculated by averaging the scores of all video frames in the video frame sequence. The comprehensive score is compared with a second preset threshold to determine whether the video to be detected belongs to the target video.

7. The video content region detection method according to claim 1, characterized in that, The step of aggregating and analyzing the content region location information of the video frame sequence of the target video to obtain video-level content region location information includes: From the content region positioning information of the video frame sequence, select any video frame as the reference video frame, and use its content region positioning information as the reference positioning information. Using the reference positioning information as a reference, video frames in the video frame sequence whose video frame content region positioning information matches the reference positioning information are clustered into the same cluster set; Wherein, the matching means that the difference between the positioning information of any side of the content area of ​​the current video frame and the positioning information of the corresponding side in the reference video frame is within a preset difference range. From all cluster sets, select the cluster set with the largest number of video frames as the target cluster set; For all content regions in the target cluster set, the mean values ​​of the positioning information on the top, bottom, left, and right sides are calculated. The statistical results are combined to generate new content region positioning information, which is the video-level content region positioning information of the video to be detected.

8. A video content region detection device, characterized in that, include: The video frame sequence analysis module is used to sample the video to be detected to obtain a video frame sequence and acquire the content region positioning information of the video frame sequence. The video classification and filtering module is used to filter out target videos from the videos to be detected according to preset rules based on the location features of the content regions in the video frame sequence. The video-level content region calculation module is used to aggregate and analyze the content region positioning information of the video frame sequence of the target video to obtain video-level content region positioning information.

9. The video content region detection device according to claim 8, characterized in that, The video frame sequence analysis module also includes a positioning information correction unit; The positioning information correction unit is used to correct the positioning information using one or more of the following methods after obtaining the content region positioning information of the video frame sequence: Based on the pixel features of the video frame sequence, the content area positioning information is corrected; Based on the spatial distribution consistency of content region positioning information in video frame sequences, the content region positioning information is structurally corrected. The content region positioning information of the video frame sequence is aligned and corrected with the video frame boundaries.

10. The video content region detection device according to claim 9, characterized in that, The correction of content region positioning information based on pixel features of video frame sequences includes: Extract the luminance channel of the video frame sequence in the YUV color space; Analyze and calculate the pixel brightness gradient within a preset range centered on the boundary pixel of the content region; The pixel with the most significant and continuous pixel brightness gradient is scanned and the boundary of the content region is adjusted to the position of that pixel.

11. The video content region detection device according to claim 9, characterized in that, The spatial distribution consistency of content region positioning information based on video frame sequences is used to structurally correct the content region positioning information, including: Using the geometric center of the video frame sequence as a reference, statistically analyze the overall distribution trend of the video frame sequence content area in the vertical and / or horizontal directions: When more than a preset proportion of content areas are concentrated in the upper or lower half of a video frame, it can be inferred that the content areas of the video frame should have a vertically symmetrical structure, and the content areas of the video frame should be corrected according to the vertically symmetrical structure. When more than a preset proportion of content areas are concentrated in the left or right half of a video frame, it can be presumed that the content areas of the video frame should have a horizontally symmetrical structure, and the content areas of the video frame should be corrected symmetrically according to the horizontally symmetrical structure.

12. The video content region detection device according to claim 9, characterized in that, The step of aligning and correcting the content region positioning information of the video frame sequence with the video frame boundaries includes: If the distance difference between one or more edges of the content region boundary of the video frame sequence and the corresponding edge of the video frame is less than the first preset threshold, then the positioning information of the edge is attached to the positioning information of the corresponding edge of the video frame, that is, the positioning information of the edge is corrected to the positioning information of the corresponding edge of the video frame. If the distance difference between any side of the content region boundary of the video frame sequence and the corresponding side of the video frame is less than the first preset threshold, then the positioning information of that side remains unchanged.

13. The video content region detection device according to claim 8, characterized in that, The video classification and filtering module includes: The video frame classification and labeling unit is used to classify and label the video frame sequence according to the relative positional relationship between the content area and the invalid area of ​​the video frame sequence; The scoring unit is used to assign weight scores to video frames of each category based on the classification results; The scoring calculation unit is used to calculate the comprehensive score of the video to be detected by averaging the scores of all video frames in the video frame sequence. The determination unit is used to compare the comprehensive score with a second preset threshold to determine whether the video to be detected belongs to the target video.

14. The video content region detection device according to claim 8, characterized in that, The video-level content region calculation module includes: The reference selection unit is used to select any video frame as a reference video frame from the content region positioning information of the video frame sequence, and its content region positioning information is used as the reference positioning information. The clustering unit is used to select video frames in the video frame sequence whose video frame content region positioning information matches the benchmark positioning information and cluster them into the same cluster set, with the benchmark positioning information as a reference. Wherein, the matching means that the difference between the positioning information of any side of the content area of ​​the current video frame and the positioning information of the corresponding side in the reference video frame is within a preset difference range. The target cluster set selection unit is used to select the cluster set with the most video frames from all cluster sets as the target cluster set; The aggregation calculation unit is used to perform mean calculation on the positioning information of the top, bottom, left and right sides of all content regions in the target cluster set, and combine the statistical results to generate new content region positioning information, which is the video-level content region positioning information of the video to be detected.

15. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute a computer program stored in the memory to implement the video content region detection method as described in any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the video content region detection method as described in any one of claims 1 to 7.

Images