A video frame clipping image segmentation processing method

Abstract

The present invention discloses a video frame editing image segmentation processing method, which comprises the following steps: firstly extracting video frame images from a video to be edited, obtaining a continuous frame image sequence of the video as a processing basis according to a time sequence and a timestamp of the video; converting the video into an image sequence, so as to facilitate independent analysis and processing of each frame image in subsequent processing steps; using a preset target object recognition neural network to perform target object recognition detection on each video frame image, and outputting a plurality of target video frame images containing the target object; determining a target object image block of the target object in each target video frame image based on a preset target object contour; iteratively screening and analyzing pixel data in the target object image block, identifying edge pixel points of the target object, and thereby obtaining a clear target object image.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a video frame clipping image segmentation processing method. Background Art

[0002] During the video production process, filmmakers often use a "stand-in" method, using a stand-in to perform dangerous actions of the main actor to shoot a draft video. The stand-in's image in the draft video is then cut out and the main actor's image is replaced with the stand-in's image to edit the complete video.

[0003] Traditional methods extract the target image from the full-length video using simple background subtraction or color distribution segmentation. However, these methods often perform poorly for scenes with complex backgrounds or dramatic lighting changes. Background subtraction is easily affected by background complexity, moving objects, or lighting changes, resulting in the extracted target image containing a large amount of noise or missing parts of the target object. Summary of the Invention

[0004] The purpose of the present invention is to provide a video frame clipping image segmentation processing method to solve the above technical problems pointed out in the prior art.

[0005] The present invention provides a video frame clipping image segmentation processing method, comprising the following steps:

[0006] Obtain a video to be edited; perform a video frame extraction operation on the video to be edited according to the timestamps on the time sequence to obtain multiple video frame images;

[0007] Using a preset target object recognition neural network to identify and detect each of the video frame images, and output a plurality of target video frame images containing the target object;

[0008] Determine a target object image block in the target video frame image according to a preset target object contour;

[0009] Iteratively screening and analyzing pixel data within the target object image block to identify edge pixel points of the target object; performing an image segmentation operation on the target object image block based on the edge pixel points of the target object to obtain a target object image.

[0010] Preferably, the preset target object contour includes the following steps:

[0011] Pre-set the contour image of the target object image;

[0012] The reference area of ​​the target object image in the contour picture is manually marked; and the image signal in the marked reference area is used as the image contour of the target object corresponding to the target object image.

[0013] Preferably, the iterative screening and analyzing of the pixel data in the target object image block to identify the edge pixel points of the target object comprises the following steps:

[0014] grayscale the target object image block, and then perform a bilateral filtering operation to obtain a first preprocessed image;

[0015] Generate a projection image by performing a projection operation on each pixel point of the first preprocessed image;

[0016] Acquiring peak data and peak correlation data of the projection image;

[0017] Calculating and obtaining a height difference between each of the peak data and the peak associated data corresponding to the peak data; determining whether the height difference is greater than or equal to a preset height difference threshold; and if so, determining that the peak data is the first peak data;

[0018] The edge pixels of the target object are identified through secondary screening analysis based on the first peak data and the last peak data in the peak graph.

[0019] Preferably, the projection operation includes a horizontal projection operation and a vertical projection operation;

[0020] The projection diagram includes a horizontal projection diagram and a vertical projection diagram;

[0021] The peak data includes horizontal peak data and vertical peak data;

[0022] The peak correlation data includes horizontal peak correlation data and vertical peak correlation data;

[0023] The horizontal wave peak data includes the horizontal wave peak coordinates and the horizontal wave peak projection value J corresponding to the horizontal wave peak coordinates;

[0024] The vertical peak data includes vertical peak coordinates and vertical peak projection values ​​J' corresponding to the vertical peak coordinates;

[0025] The horizontal peak association data includes the horizontal trough coordinates of the horizontal trough adjacent to each horizontal peak and the horizontal trough projection value J±1 corresponding to the horizontal trough coordinates; wherein J+1 is the horizontal trough projection value of the horizontal trough adjacent to the right of the J-th horizontal peak projection value; and J-1 is the horizontal trough projection value of the horizontal trough adjacent to the left of the J-th horizontal peak projection value.

[0026] The vertical peak association data includes the vertical trough coordinates of the vertical troughs adjacent to each vertical peak and the vertical trough projection values ​​J'±1 corresponding to the vertical trough coordinates; wherein J'+1 is the vertical trough projection value of the vertical trough adjacent to the right of the J'th vertical peak projection value; and J'-1 is the vertical trough projection value of the vertical trough adjacent to the left of the J'th vertical peak projection value;

[0027] Preferably, the step of identifying edge pixels of the target object through secondary screening analysis based on the first peak data and the last peak data in the peak graph includes the following steps:

[0028] Traversing each of the first peak data, and calculating and obtaining a standard deviation between the first peak data and the last peak data in the peak graph;

[0029] Obtaining a standard deviation judgment threshold based on the grayscale values ​​of neighboring pixel points within a preset range of the pixel point corresponding to the first peak data; determining whether the standard deviation is greater than or equal to the standard deviation judgment threshold, and if so, determining the first peak data as the second peak data, and returning to the above operation to continue traversing until all the first peak data are traversed to obtain multiple second peak data;

[0030] Performing morphological processing on the pixel points corresponding to all the second wave peak data to obtain edge pixel points of the target object;

[0031] Preferably, the step of performing a morphological processing operation on the pixel points corresponding to all the second peak data to obtain the edge pixel points of the target object comprises the following steps:

[0032] Clustering the pixel points corresponding to each of the second wave peak data to obtain a plurality of clusters; selecting the largest one among the clusters as a target cluster;

[0033] Get all the bounding pixels (x c ,y c ); for each of the surrounding pixel points (x c ,y c ) Calculate the local binary pattern LBP R (x c ,y c ) features; based on the local binary pattern LBP R (x c 'y c ) Features construct the contour texture descriptor of the regional image composed of the bounding box of the current target cluster;

[0034] Performing a morphological processing operation on each of the second wave peak data to obtain edge pixel points to be determined formed by all of the second wave peak data;

[0035] The edge pixel points to be determined are iteratively optimized based on the contour texture descriptor to obtain edge pixel points of the target object.

[0036] Preferably, the texture descriptor can be constructed by LBP histogram.

[0037] Preferably, the iterative optimization of the edge pixel points to be determined based on the contour texture descriptor to obtain the edge pixel points of the target object includes the following steps:

[0038] Get the processing parameter vector θ of the morphological processing operation t ; The processing parameter vector θ t Including the shape and size of the structural element and the number of expansion and corrosion; and setting an iteration counter and an iterative optimization stop condition; the iterative optimization stop condition is the minimum threshold of the texture similarity measure; the number of iterations of the iteration counter is initially 0;

[0039] Incrementing the number of iterations of the iteration counter by 1, and acquiring an edge surrounding image based on all the edge pixels to be determined; traversing each of the edge pixels to be determined, and acquiring an image of the area to be determined corresponding to the edge pixel point using a search box of a preset size with the edge pixel point to be determined as the center;

[0040] Calculating each pixel point in the image of the area to be determined to obtain a first contour texture descriptor;

[0041] Calculating a texture similarity measure F based on the first contour texture descriptor and the contour texture descriptor;

[0042] Determine whether the texture similarity measure F is greater than or equal to the preset iterative optimization stop condition; if so, output the edge pixel to be determined as the edge pixel of the final target object; if not, based on the number of iterations of the iteration counter and the morphological processing parameter vector θ t And the first contour texture descriptor is calculated to obtain a new morphological processing parameter θ t+1 ;

[0043] The new morphological processing parameter θ t+1 The calculation method is:

[0044]

[0045] The second peak data is processed based on the new morphological processing parameters to obtain new edge pixel points to be determined, and the above operation is returned to until the texture similarity measure F is greater than or equal to the preset iterative optimization stop condition, and the edge pixel points of the target object are output.

[0046] Preferably, the construction method of the first contour texture descriptor is consistent with the construction method of the contour texture descriptor.

[0047] On the other hand, the present invention further provides a storage medium, characterized in that a computer program is stored in the storage medium, and when the computer program is executed by a processor, the above-mentioned video frame clipping image segmentation processing method is implemented.

[0048] Compared with the prior art, the embodiments of the present invention have at least the following technical advantages:

[0049] An analysis of the above-mentioned video frame editing image segmentation processing method provided by the present invention shows that, in a specific application, video frame images are first extracted from the video to be edited, and a continuous frame image sequence of the video is obtained as a basis for processing according to the time series and timestamp of the video; the video is converted into an image sequence, which facilitates the subsequent processing steps to independently analyze and process each frame image; a pre-set target object recognition neural network (such as a convolutional neural network based on YOLO) is used to identify and detect the target object for each video frame image, and multiple target video frame images containing the target object are output, that is, each frame is screened to ensure that only the frames containing the target object are further processed, so that the target object in the video to be edited can be quickly and accurately identified;

[0050] Among them, based on the preset target object contour, the specific image area of ​​the target object in each target video frame image, or the target object image block, is determined, a large amount of unnecessary and redundant image content is removed, and the focus is on the target object area, which improves the speed and accuracy of subsequent processing. The target object image block is determined by the preset contour, which effectively optimizes the image analysis process and reduces the consumption of computing resources; the pixel data in the target object image block is iteratively screened and analyzed to identify the edge pixel points of the target object. These pixel data include color data (RGB), grayscale data and position data, which are used for subsequent image segmentation and edge detection, thereby improving the accuracy and clarity of the target object edge and laying the foundation for further image segmentation operations. Through fine pixel-level analysis, efficient recognition of the target object edge is ensured, thereby obtaining a clear target object image. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of the main flow of a video frame clipping image segmentation processing method provided in the first embodiment of the present invention;

[0052] Figure 2 A schematic diagram of the steps for iteratively screening and analyzing to identify edge pixels of a target object in a video frame clipping image segmentation processing method provided in the first embodiment of the present invention;

[0053] Figure 3 A schematic diagram of the operation steps of obtaining edge pixel points of a target object through morphological processing in a video frame clipping image segmentation processing method provided in the first embodiment of the present invention;

[0054] Figure 4 A schematic diagram of target cluster simulation in a video frame clipping image segmentation processing method provided in the first embodiment of the present invention;

[0055] Figure 5 A schematic diagram of the operation steps for obtaining edge pixel points of a target object in a video frame clipping image segmentation processing method provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0056] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0057] The present invention will be further described in detail below through specific embodiments in conjunction with the accompanying drawings.

[0058] Example 1

[0059] like Figure 1 As shown, the first embodiment of the present invention provides a video frame clipping image segmentation processing method, including the following steps:

[0060] Step S10: obtaining a video to be edited; performing a video frame extraction operation on the video to be edited according to the timestamps in the time sequence to obtain a plurality of video frame images;

[0061] It should be noted that the above embodiment of the present application extracts video frame images through each timestamp of the time sequence of the video to be edited, and obtains all video frame images corresponding to the current video to be edited, which can provide an image basis for subsequent processing of the video frame images;

[0062] Step S20: using a preset target object recognition neural network to identify and detect each of the video frame images, and outputting a plurality of target video frame images containing the target object;

[0063] It should be noted that the above-mentioned embodiment of the present application performs target object recognition operations on video frame images through a neural network model, and can quickly and accurately identify target video frame images containing target objects in the video to be edited. The above-mentioned target object recognition neural network is a convolutional neural network established based on YOLO. YOLO is a popular target detection algorithm. Its core idea is to perform end-to-end target detection through a single neural network model, which is efficient and fast. The method of establishing a convolutional neural network based on YOLO is common knowledge among those skilled in the art and will not be repeated in this application.

[0064] Step S30: determining a target object image block in the target video frame image according to a preset target object contour;

[0065] It should be noted that the above embodiment of the present application determines the image region where the target object outline in the target video frame image is located, i.e., the above target object image block, by using the preset target object outline. By extracting the above target object image block, a large amount of unnecessary and redundant image content can be screened out, which has a significant speed-up effect on the subsequent image segmentation and recognition processing.

[0066] Specifically, in order to more accurately segment the target object image and improve segmentation efficiency, in this embodiment, the target object image block (or target object image area) is first determined in the target video frame image information. The subsequent edge detection image segmentation processing operation is only performed on the target object image block, and the image of other areas in the target video frame image except the target object image block is not subsequently processed, which greatly improves processing efficiency.

[0067] In this embodiment, for some large interference images present in the target video frame image obtained in step S10: for example, if the target video frame image only contains a portion of the target object image and the rest are large interference images, step S20 can also have a screening function. That is, in order to improve the detection efficiency, the above-mentioned large interference images can be excluded through step S20, and only the image pixel data within the target object image block can be processed in the next step;

[0068] In this embodiment, the target object contour can be: pre-setting a contour picture of the target object image, manually marking a reference area of ​​the target object image in the contour picture; and using the image signal in the marked reference area as the image contour of the target object corresponding to the target object image.

[0069] Step S40: Iteratively screening and analyzing the pixel data in the target object image block to identify the edge pixel points of the target object; performing an image segmentation operation on the target object image block based on the edge pixel points of the target object to obtain the target object image.

[0070] In the subsequent operation process, the image segmentation can be performed based on the edge pixels of the target object to obtain the target object image;

[0071] The pixel data includes color data (i.e., RGB data), grayscale data, and position data of each pixel in the target object image block;

[0072] It should be noted that the above-mentioned embodiment of the present application first extracts video frame images from the video to be edited, and obtains a continuous frame image sequence of the video as the basis for processing according to the time series and timestamp of the video, providing a static image representation of the video to be edited, and providing basic data for subsequent target object recognition and image processing; converting the video into an image sequence, so as to facilitate the subsequent processing steps to perform independent analysis and processing of each frame image; using a pre-set target object recognition neural network (such as a convolutional neural network based on YOLO) to perform target object recognition detection on each video frame image, and output a plurality of target video frame images containing the target object, that is, screening each frame to ensure that only frames containing the target object are further processed, and the target object in the video to be edited can be quickly and accurately identified. The efficiency and speed of the YOLO algorithm ensure the rapid processing of a large number of video frames, thereby improving the overall processing efficiency;

[0073] Based on the preset target object contour, the specific image area of ​​the target object in each target video frame image, or the target object image block, is determined, a large amount of unnecessary and redundant image content is removed, and the focus is on the target object area, which improves the speed and accuracy of subsequent processing. The target object image block is determined by the preset contour, which effectively optimizes the image analysis process and reduces the consumption of computing resources; the pixel data in the target object image block is iteratively screened and analyzed to identify the edge pixels of the target object. These pixel data include color data (RGB), grayscale data and position data, which are used for subsequent image segmentation and edge detection, thereby improving the accuracy and clarity of the target object edge and laying the foundation for further image segmentation operations. Through fine pixel-level analysis, efficient recognition of the target object edge is ensured, thereby obtaining a clear target object image.

[0074] Specifically, such as Figure 2 As shown, in step S40, iterative screening and analysis are performed based on the pixel data in the target object image block to identify the edge pixel points of the target object, including the following steps:

[0075] Step S41: grayscale processing is performed on the target object image block, and then a bilateral filtering operation is performed to obtain a first pre-processed image;

[0076] It should be noted that the above-mentioned application performs grayscale processing on the target object image blocks in sequence, and after obtaining the grayscale image, performs bilateral filtering processing on the grayscale image to smooth the grayscale image to obtain a first preprocessed image, which can remove a large amount of noise in the image and make subsequent analysis and recognition more accurate.

[0077] Step S42: generating a projection image by projecting each pixel point of the first pre-processed image;

[0078] The projection operation includes a horizontal projection operation and a vertical projection operation;

[0079] The projection diagram includes a horizontal projection diagram and a vertical projection diagram;

[0080] It should be noted that, in the above-mentioned embodiment of the present application, the first pre-processed image is horizontally projected to generate a horizontal projection image, and the first pre-processed image is vertically projected to generate a vertical projection image. This is because the first pre-processed image contains image content other than the target object image. For example, in a first pre-processed image, the target object image is a cat. However, in the target object image block obtained by the above-mentioned target object contour recognition analysis, there will also be image content such as the sky, grass, and other interfering images that do not belong to the "cat". In the embodiment of the present application, the first pre-processed image is generated into a projection image through horizontal projection and vertical projection, which can more clearly and intuitively use a computer to identify and analyze image content that does not belong to the cat, thereby filtering out image content such as the sky and grass.

[0081] The horizontal projection mentioned above refers to summing the pixels of each row of the image along the vertical direction (usually from top to bottom) to obtain a one-dimensional array. Each element in this array represents the sum of the pixels on the corresponding row, reflecting the pixel distribution of the image in the vertical direction. Horizontal projection can help detect the row spacing, row position and row distribution density in the image;

[0082] The vertical projection mentioned above refers to summing the pixels of each column of the image along the horizontal direction (usually from left to right) to obtain a one-dimensional array. Each element in the array represents the sum of the pixels in the corresponding column, reflecting the pixel distribution of the image in the horizontal direction.

[0083] Different areas in the image can be segmented or separated by the distribution information of the projections in the projection map obtained by the horizontal projection and the vertical projection.

[0084] Step S43: obtaining peak data and peak correlation data of the projection diagram (including the horizontal projection diagram and the vertical projection diagram);

[0085] The peak data includes horizontal peak data and vertical peak data;

[0086] The peak correlation data includes horizontal peak correlation data and vertical peak correlation data;

[0087] The horizontal wave peak data includes the horizontal wave peak coordinates and the horizontal wave peak projection value J corresponding to the horizontal wave peak coordinates;

[0088] The vertical peak data includes vertical peak coordinates and vertical peak projection values ​​J' corresponding to the vertical peak coordinates;

[0089] The horizontal peak association data includes the horizontal trough coordinates of the horizontal trough adjacent to each horizontal peak and the horizontal trough projection value J±1 corresponding to the horizontal trough coordinates; wherein J+1 is the horizontal trough projection value of the horizontal trough adjacent to the right of the J-th horizontal peak projection value; and J-1 is the horizontal trough projection value of the horizontal trough adjacent to the left of the J-th horizontal peak projection value.

[0090] The vertical peak association data includes the vertical trough coordinates of the vertical troughs adjacent to each vertical peak and the vertical trough projection values ​​J'±1 corresponding to the vertical trough coordinates; wherein J'+1 is the vertical trough projection value of the vertical trough adjacent to the right of the J'th vertical peak projection value; and J'-1 is the vertical trough projection value of the vertical trough adjacent to the left of the J'th vertical peak projection value;

[0091] Step S44: Calculate and obtain the height difference between each of the peak data and the peak-related data corresponding to the peak data; determine whether the height difference is greater than or equal to a preset height difference threshold; if so (if not, it is determined that the pixel points of the peak data and the peak-related data corresponding to the peak data are in the same area, that is, the pixel points corresponding to the peak data and the pixel points corresponding to the peak-related data belong to the same pixel points in the target object image or the pixel points of the interference image), then determine that the peak data is the first peak data;

[0092] It should be noted that when the above-mentioned height difference is greater than or equal to the preset height difference threshold, it proves that the peak data domain and the peak associated data corresponding to the peak data do not belong to the same area, that is, the pixel point corresponding to the peak data and the pixel point corresponding to the peak associated data do not belong to the same object (that is, the pixel point of the target object image or the interference image); therefore, the embodiment of the present application determines the peak data as the first peak data for subsequent processing, and analyzes and identifies the edge pixel points of the target object; and the peak data in the embodiment of the present application is the peak or trough phenomenon that occurs at the edge of the target object image and the interference image, and the embodiment of the present application is mainly to identify and analyze the edge pixel points of the target object, and to intercept the target object image as the ultimate goal. Therefore, the embodiment of the present application does not perform the next step of processing for the situation where the peak data and the peak associated data belong to the same area, that is, when the height difference is less than the preset height difference threshold, the peak data and the peak associated data are shelved and no further processing operation is performed.

[0093] Step S45: identifying edge pixels of the target object through secondary screening analysis based on the first peak data and the last peak data in the peak graph.

[0094] It should be noted that the above-mentioned embodiment of the present application first eliminates color information and retains the brightness (grayscale) information of the image through a grayscale processing operation, which helps to improve the stability and efficiency of subsequent processing steps; through bilateral filtering, by considering the spatial distance and grayscale value similarity between pixels, noise is effectively removed, making the image clearer and more accurate, which is conducive to subsequent edge detection and analysis tasks; through horizontal projection and vertical projection operations, the line spacing, line position and distribution density information in the image are extracted to clearly and intuitively display the pixel distribution in the image, which helps the computer to identify and analyze the content of different areas, especially for the distinction between target objects and interfering objects (such as sky and grass); the height difference between peaks is calculated through peak data and peak correlation data to determine the boundary of the area, and according to the preset height difference threshold, it is judged whether the difference in projection values ​​between adjacent peaks is large enough to determine whether they belong to the same area; based on the relationship between the first peak data and the last peak data, the edge pixel points of the target object are further analyzed and identified, the boundary of the target object is determined, and accurate edge information is provided as basic data for further image segmentation;

[0095] The above steps S41 to S44 are the first screening, which divides the image content in the first preprocessed image into the target object image and the interference object image. Step S45 is the second screening, which analyzes and identifies the edge pixels of the target object through the first peak data and the last peak data in the peak image. For specific operations, please refer to the subsequent processing steps S451-S453.

[0096] Specifically, in step S45, edge pixels of the target object are identified through secondary screening analysis based on the first peak data and the last peak data in the peak graph, including the following steps:

[0097] Step S451: traverse each of the first peak data, and calculate and obtain the standard deviation between the first peak data and the last peak data in the peak graph;

[0098] The standard deviation is calculated as:

[0099]

[0100] Where σ is the standard deviation; N is the number of corresponding coordinates; xi is the projection value corresponding to the i-th coordinate; μ is the average projection value;

[0101] It should be noted that by calculating the standard deviation between the first peak data and the last peak data in the peak graph, the change in the grayscale value of the pixel between the two peaks can be evaluated. If the standard deviation is small, it means that the pixel grayscale change in this interval is small, which may be due to the influence of the background or other non-target objects, rather than the real target object edge.

[0102] Step S452: obtaining a standard deviation judgment threshold based on the grayscale values ​​of neighboring pixel points within a preset range of the pixel points corresponding to the first peak data; judging whether the standard deviation is greater than or equal to the standard deviation judgment threshold, if so (it is determined that there is no interfering image between the pixel points of the first peak data and the last peak data, that is, the pixel points between the first peak data and the last peak data all belong to the pixel points of the target object image), then determining the first peak data as the second peak data, and returning to the above operation to continue traversing until all the first peak data are traversed to obtain multiple second peak data;

[0103] If not, then (it is determined that there is an interference image between the pixel point corresponding to the first peak data and the pixel point corresponding to the last peak data in the peak graph, which may also be caused by image fluctuation of the interference image) the first peak data is filtered out and no further processing operation is performed;

[0104] It should be noted that the above-mentioned standard deviation judgment threshold is calculated based on the grayscale values ​​of the neighboring pixels within a preset range of the pixel corresponding to the first peak data currently traversed. Specifically, the grayscale values ​​of the current pixel and the neighboring pixels are first averaged, and then the average value is subtracted from the value of each pixel to obtain the difference. The squares of each difference are further added and divided by the number of pixels to obtain the variance. Finally, the square root of the variance is the standard deviation.

[0105] The advantage of this method is that the standard deviation judgment threshold is adaptively calculated based on the neighborhood pixels of the first peak data. The threshold is dynamically determined based on the actual data of the pixels corresponding to the first peak data. Because different image regions may have different grayscale or color distributions, this adaptive method can more accurately distinguish the edge pixels of the target object from the background or interference pixels.

[0106] Interference images tend to have a relatively uniform grayscale distribution. This is because interference is usually the background or other non-target objects, and its grayscale changes are not significant, unlike the edge of the target object, which has obvious grayscale changes.

[0107] Therefore, the peak data belonging to the target object image in the first peak data (ie, the second peak data) can be determined and analyzed by using the standard deviation obtained by projection.

[0108] Step S453: performing morphological processing on the pixel points corresponding to all the second peak data to obtain edge pixel points of the target object;

[0109] It should be noted that the above morphological processing operations include basic processing parameters, such as:

[0110] 1. Structural element. The structural element is the core of morphological operations, which determines the effect and characteristics of the processing. Morphological operations are based on structural elements to process edge pixels. These operations include corrosion, expansion, opening and closing operations, etc. The shapes of structural elements include: rectangle: suitable for processing horizontal and vertical edges; circle: has a uniform effect on edges in all directions and is suitable for processing circular or curved features; cross: can retain details and have a certain ability to remove noise; line: used to detect edges in a specific direction; the size of the structural element is as follows: small size (such as 3x3): retains more details, but has weaker noise removal ability; large size (such as 7x7 or larger): can remove more noise, but may lose some details; the choice of structural element directly affects the processing effect. For example, the structural element used to remove noise should be smaller than the smallest target feature to be retained, while the structural element used to fill holes should be larger than the largest hole to be filled.

[0111] 2. Processing parameters for dilation and erosion. Dilation: Increasing the dilation times will expand the bright area (foreground) and can fill small holes and gaps; excessive dilation may lead to loss of details and rough edges; Erosion: Increasing the erosion times will shrink the bright area and remove small noise points; excessive erosion may cause small features to disappear. The dilation and erosion times need to be balanced based on image characteristics and processing goals. For example, to remove small noise points, you can first perform a small amount of erosion, followed by the same number of dilations (i.e., opening operations);

[0112] The basic operations of morphological processing operations include opening and closing operations. Opening operations: first erosion and then dilation, remove small objects, smooth the outlines of larger objects, and separate objects; closing operations: first dilation and then erosion, fill small holes, connect neighboring objects, smooth the outlines, and obtain image edges. The above morphological processing operations are prior art and will not be described in detail in this application. In addition, the focus of the embodiments of this application is not on obtaining edge pixels through morphological processing, but on iteratively optimizing the processing parameters of the morphological processing, so as to use the optimized processing parameters to detect and identify edge pixels of the target object image.

[0113] In the above-mentioned embodiment of the present application, first, it is determined whether there is an interfering image based on the calculated standard deviation, and the standard deviation judgment threshold of the neighborhood grayscale value of the pixel point corresponding to the first peak data is calculated to adaptively determine whether there is an interfering image. This method can more accurately distinguish the edge pixels of the target object and the background or interfering pixels; further, the peak data without interference (i.e., the above-mentioned second peak data) is effectively screened out through the judgment conditions to ensure the accuracy of subsequent processing steps. Through dynamic calculation based on the current pixel point and its neighborhood grayscale value, it can adapt to the grayscale or color distribution of different image areas, thereby improving the robustness and accuracy of the algorithm; the peak data of the target object is processed using morphological processing (such as dilation, erosion, etc.) to obtain its edge pixel points, thereby improving the accuracy and stability of edge detection.

[0114] During the specific implementation of the above-mentioned embodiments of the present application, technicians found that traditional morphological processing methods usually use fixed structural elements to perform dilation and erosion operations to obtain interference images, but this method is only suitable for image segmentation of simple scenes. In complex scenes, such as the "cat" in the above example, where the morphology changes greatly and the color changes of the cat are complex, this will lead to inaccurate edge extraction of the target object. Therefore, the use of morphological processing operations to process the pixel points corresponding to the second peak data to obtain the target object image still requires further research by technicians.

[0115] Specifically, if Figure 3 As shown, in step S453, all pixel points corresponding to the second peak data are subjected to morphological processing operations to obtain edge pixel points of the target object, including the following operation steps:

[0116] Step S4531: clustering the pixel points corresponding to each of the second wave peak data to obtain a plurality of clusters; selecting the largest one among the clusters as the target cluster;

[0117] It should be noted that the above-mentioned embodiment of the present application clusters the pixel points of the second peak data, and obtains block images of the target object image obtained by merging multiple pixel points of the current target object image. For example, the largest cluster in the clusters is selected as the target cluster, and the image composed of all pixels in this target cluster may be an area in the target object image. For example, if the entire target object image is a "cat", the corresponding area composed of the largest cluster is selected as a part of the cat's body or the entire body (for example, the corresponding area composed of the largest cluster is selected as the cat's head, or the corresponding area composed of the largest cluster is selected as the cat's tail); by selecting the target cluster, it can be used as the texture basis of the entire target object image, so that the overall target object image can be obtained in subsequent operations;

[0118] In addition, it should be noted that the second peak data extracted above are usually the peak data corresponding to the pixel points of the target object image, and inside the target object image, there are peak data corresponding to some pixels. Because the pixel points inside the image are evenly distributed, they are not determined as the second peak data. Therefore, the second peak data reflects the pixel points at the edge of the target object image and the interference image (such as Figure 4 shown).

[0119] Step S4532: Obtain all bounding pixel points (x c ,y c ); for each of the surrounding pixel points (x c ,y c ) Calculate the local binary pattern LBP R (x c ,y c ) features; based on the local binary pattern LBP R (x c ,y c ) Features construct the contour texture descriptor of the regional image composed of the bounding box of the current target cluster;

[0120] It should be noted that texture features are an important basis for recognition in target object recognition. By calculating texture descriptors, we can better characterize the characteristics of the target area and provide a basis for subsequent edge optimization.

[0121] The local binary pattern describes the texture features of each pixel by comparing the grayscale values ​​of the central pixel with the surrounding pixels; suppose an area of ​​the cat's head is selected, and then the local binary pattern LBP of each pixel in the area is calculated R (x c ,y c) features, by comparing the texture features of the cat's head hair, a contour texture descriptor can be calculated to evaluate the difference between the head area and the background;

[0122] The above texture descriptor can be constructed through the LBP histogram, that is, by counting the frequency of occurrence of each LBP value in the entire area as a texture descriptor; at the same time, it can also be constructed through statistical features, that is, calculating the mean, variance, skewness, kurtosis and other statistics of the LBP value as a texture descriptor.

[0123] Step S4533: performing a morphological processing operation on each of the second wave peak data to obtain edge pixel points to be determined formed by all the second wave peak data;

[0124] It should be noted that the above-mentioned embodiment of the present application selects the shape and size (i.e., dimension) of the structural element, as well as the number of dilation and erosion operations through conventional morphological processing operations, and obtains the edge pixel points to be determined through these basic operations; the above-mentioned conventional morphological processing is common knowledge among those skilled in the art and will not be repeated in the embodiments of the present application.

[0125] Step S4534: iteratively optimize the edge pixel points to be determined based on the contour texture descriptor to obtain edge pixel points of the target object.

[0126] It should be noted that, in the above embodiment of the present application, the pixel points corresponding to the second peak data are first clustered to obtain multiple clusters, and the largest cluster is selected as the target cluster. The target cluster represents a major part of the target object image, such as the body or head of a cat, and provides a texture basis for the entire target object image for subsequent operations.

[0127] The main parts of the target object are identified through clustering, providing a basis for subsequent texture analysis and edge optimization; further, by obtaining all the surrounding pixels within the bounding box of the target cluster, the local binary pattern (LBP) features of the surrounding pixels are calculated, and a contour texture descriptor is constructed based on the LBP features. The local texture information of the target area is captured through the LBP features; the contour texture descriptor is constructed to represent the texture features of the target area, providing important texture information for subsequent edge optimization; then, the second peak data is morphologically processed to obtain the edge pixels to be determined, and the edge information is preliminarily processed through morphological operations to provide candidate edge pixels for subsequent edge optimization; the edge pixels to be determined are iteratively optimized based on the contour texture descriptor to obtain the final target object edge pixels, and the edge pixels are optimized using texture information. The accuracy of edge recognition is improved through an iterative process.

[0128] Specifically, if Figure 5As shown, in step S4534, the edge pixel points to be determined are iteratively optimized based on the contour texture descriptor to obtain the edge pixel points of the target object, including the following operation steps:

[0129] Step S45341: Obtain the processing parameter vector θ of the morphological processing operation t ; The processing parameter vector θ t Including the shape and size of the structural element and the number of expansion and corrosion; and setting an iteration counter and an iterative optimization stop condition; the iterative optimization stop condition is the minimum threshold of the texture similarity measure; the number of iterations of the iteration counter is initially 0;

[0130] Step S45342: Increment the number of iterations of the iteration counter by 1, and simultaneously obtain an edge surrounding image based on all the edge pixels to be determined; traverse each of the edge pixels to be determined, and obtain the image of the area to be determined corresponding to the edge pixel point using a search box of a preset size with the edge pixel point to be determined as the center;

[0131] Step S45343: Calculating each pixel point in the image of the area to be determined to obtain a first contour texture descriptor;

[0132] It should be noted that the first contour texture descriptor is also obtained by using the local binary pattern LBP of the image in the area to be determined. R (x c ′y c ) Features are constructed by LBP histogram or statistical features;

[0133] Step S45344: Calculating and obtaining a texture similarity measure F according to the first contour texture descriptor and the contour texture descriptor;

[0134] It should be noted that the above texture similarity measure can be calculated by histogram similarity or statistical feature distance. If the first contour texture descriptor and the contour texture descriptor are constructed using LBP histogram, the histogram intersection or chi-square distance can be calculated to obtain the texture similarity measure;

[0135] If the first contour texture descriptor and the contour texture descriptor are constructed using statistical features, a Euclidean distance or a Mahalanobis distance may be calculated to calculate a texture similarity measure;

[0136] It should also be noted that the construction method of the first contour texture descriptor is consistent with the construction method of the contour texture descriptor.

[0137] Step S45345: Determine whether the texture similarity measure F is greater than or equal to a preset iterative optimization stop condition; if so, output the edge pixel to be determined as the edge pixel of the final target object; if not, based on the number of iterations of the iteration counter and the morphological processing parameter vector θ t And the first contour texture descriptor is calculated to obtain a new morphological processing parameter θ t+1 ;

[0138] The new morphological processing parameter θ t+1 The calculation method is:

[0139]

[0140] Step S45346: Process the second peak data based on the new morphological processing parameters to obtain new edge pixel points to be determined, and return to the above operation until the texture similarity measure F is greater than or equal to the preset iterative optimization stop condition, and output the edge pixel points of the target object.

[0141] It should be noted that the above-mentioned embodiment of the present application first obtains the parameter vector required for the morphological processing operation, including the shape and size of the structural element, and the number of dilations and erosions. These parameters determine the accuracy and effect of the morphological operation in edge detection; the edge surrounding image of each edge pixel to be determined is used to provide a data basis for subsequent morphological parameter optimization to output the edge pixel of the target object image; further, for each pixel in the image of the area to be determined, a first contour texture descriptor is calculated to represent the local texture features around each pixel, and the texture similarity measure F is calculated by using the first contour texture descriptor and the known contour texture descriptor to measure the accuracy and closeness to the expected edge pixel of the target object in the current iteration; when the expected situation is not met, the morphological processing parameters are optimized through iterative optimization to obtain new edge pixel points to be determined, until the texture similarity measure F reaches the preset optimization stop condition, ensuring that the algorithm can gradually optimize the precision and accuracy of edge detection through multiple rounds of iterations.

[0142] Example 2

[0143] The present invention also provides a storage medium, in which a computer program is stored. When the computer program is executed by a processor, the above-mentioned video frame clipping image segmentation processing method is implemented.

[0144] In summary, the video frame clipping image segmentation processing method proposed in the example of the present invention first extracts video frame images from the video to be clipped, obtains a continuous frame image sequence of the video as the basis for processing according to the time series and timestamp of the video, and provides a static image representation of the video to be clipped; after converting the video into an image sequence, a preset target object recognition neural network is used to identify and detect the target object for each video frame image, and multiple target video frame images containing the target object are output, that is, each frame is screened to ensure that only frames containing the target object are further processed; based on the preset target object contour, the specific target object image block of the target object in each target video frame image is determined, a large amount of unnecessary and redundant image content is removed, and the focus is on the target object area, thereby improving the speed and accuracy of subsequent processing, and determining the target object image block through the preset contour effectively optimizes the image analysis process and reduces the consumption of computing resources; the pixel data in the target object image block is iteratively screened and analyzed to identify the edge pixel points of the target object;

[0145] In the specific operation, first, the grayscale processing operation is performed to eliminate the color information and retain the brightness (grayscale) information of the image, which is conducive to the stability and efficiency of the subsequent processing steps; the bilateral filtering process effectively removes noise by considering the spatial distance and grayscale value similarity between pixels, making the image clearer and more accurate, which is conducive to subsequent edge detection and analysis tasks; the horizontal projection and vertical projection operations are used to extract the line spacing, line position and distribution density information in the image, and clearly and intuitively display the pixel distribution in the image, which helps the computer to identify and analyze the content of different areas, especially for the distinction between target objects and interference objects (such as sky, grass); the height difference between the peaks is calculated through the peak data and the peak correlation data to determine the boundary of the area, and according to the preset height difference threshold, it is judged whether the difference in the projection values ​​between adjacent peaks is large enough to determine whether they belong to the same area; according to the relationship between the first peak data and the last peak data, the edge pixel points of the target object are further analyzed and identified, the boundary of the target object is determined, and accurate edge information is provided as the basic data for further image segmentation;

[0146] During the second screening and analysis, the presence of an interfering image is determined based on the calculated standard deviation. The standard deviation judgment threshold of the neighborhood grayscale value of the pixel point corresponding to the first peak data is calculated to adaptively determine whether an interfering image exists. This method can more accurately distinguish between the edge pixels of the target object and the background or interfering pixels. Furthermore, the peak data without interference is effectively screened out through the judgment conditions to ensure the accuracy of subsequent processing steps. Through dynamic calculation based on the grayscale value of the current pixel point and its neighborhood, it can adapt to the grayscale or color distribution of different image areas, thereby improving the robustness and accuracy of the algorithm. The peak data of the target object is morphologically processed using the morphological processing parameters obtained through iterative optimization to obtain its edge pixel points, thereby improving the accuracy and stability of edge detection, thereby obtaining a clear image of the target object.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. A person skilled in the art may modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video frame clipping image segmentation processing method, characterized in that: The steps are as follows: Obtain a video to be edited; perform a video frame extraction operation on the video to be edited according to the timestamps on the time sequence to obtain multiple video frame images; Using a preset target object recognition neural network to identify and detect each of the video frame images, and output a plurality of target video frame images containing the target object; Determine a target object image block in the target video frame image according to a preset target object contour; Iteratively screening and analyzing pixel data within the target object image block to identify edge pixel points of the target object; performing an image segmentation operation on the target object image block based on edge pixels of the target object to obtain a target object image; The iterative screening and analysis of pixel data in the target object image block to identify edge pixels of the target object comprises the following steps: grayscale the target object image block, and then perform a bilateral filtering operation to obtain a first preprocessed image; Generate a projection image by performing a projection operation on each pixel point of the first preprocessed image; Acquiring peak data and peak correlation data of the projection image; Calculate and obtain the height difference between each of the peak data and the peak associated data corresponding to the peak data; Determining whether the height difference is greater than or equal to a preset height difference threshold; If yes, determining that the peak data is the first peak data; Identifying edge pixels of the target object through secondary screening analysis based on the first peak data and the last peak data in the peak graph; The identifying edge pixel points of the target object through secondary screening analysis based on the first peak data and the last peak data in the peak graph includes the following steps: Traversing each of the first peak data, and calculating and obtaining a standard deviation between the first peak data and the last peak data in the peak graph; Obtaining a standard deviation judgment threshold based on the grayscale values ​​of neighboring pixels within a preset range of the pixel corresponding to the first wave peak data; Determine whether the standard deviation is greater than or equal to a standard deviation determination threshold; if so, determine the first peak data as the second peak data, and return to the above operation to continue traversing until all first peak data are traversed to obtain multiple second peak data; Performing morphological processing on the pixel points corresponding to all the second wave peak data to obtain edge pixel points of the target object; The step of performing morphological processing on the pixel points corresponding to all the second peak data to obtain the edge pixel points of the target object includes the following steps: Clustering the pixel points corresponding to each of the second wave peak data to obtain a plurality of clusters; selecting the largest one among the clusters as a target cluster; Get all the bounding pixels within the bounding box of the target cluster ; For each of the surrounding pixels Compute local binary patterns Features; Based on the local binary pattern Features construct the contour texture descriptor of the regional image composed of the bounding box of the current target cluster; Performing a morphological processing operation on each of the second wave peak data to obtain edge pixel points to be determined formed by all of the second wave peak data; Iteratively optimizing the edge pixel points to be determined based on the contour texture descriptor to obtain edge pixel points of the target object; The iterative optimization of the edge pixel points to be determined based on the contour texture descriptor to obtain the edge pixel points of the target object includes the following steps: Get the processing parameter vector of the morphological processing operation ; The processing parameter vector This includes the shape and size of structural elements and the number of expansions and erosions; And set the iteration counter and iterative optimization stop condition; The iterative optimization stopping condition is a texture similarity metric minimum threshold; the number of iterations of the iteration counter is initially 0; Incrementing the number of iterations of the iteration counter by 1, and acquiring an edge surrounding image based on all the edge pixels to be determined; traversing each of the edge pixels to be determined, and acquiring an image of the area to be determined corresponding to the edge pixel point using a search box of a preset size with the edge pixel point to be determined as the center; Calculating each pixel point in the image of the area to be determined to obtain a first contour texture descriptor; Calculating a texture similarity measure F based on the first contour texture descriptor and the contour texture descriptor; Determine whether the texture similarity measure F is greater than or equal to the preset iterative optimization stop condition; if so, output the edge pixel to be determined as the edge pixel of the final target object; if not, based on the number of iterations of the iteration counter and the morphological processing parameter vector And the first contour texture descriptor is calculated to obtain new morphological processing parameters ; The new morphological processing parameters The calculation method is: ； The second peak data is processed based on the new morphological processing parameters to obtain new edge pixel points to be determined, and the above operation is returned to until the texture similarity measure F is greater than or equal to the preset iterative optimization stop condition, and the edge pixel points of the target object are output.

2. A video frame clipping image segmentation processing method according to claim 1, characterized in that: The preset of the target object contour includes the following steps: Pre-set the contour image of the target object image; The reference area of ​​the target object image in the contour picture is manually marked; and the image signal in the marked reference area is used as the image contour of the target object corresponding to the target object image.

3. A video frame clipping image segmentation processing method according to claim 2, characterized in that: The projection operation includes a horizontal projection operation and a vertical projection operation; The projection diagram includes a horizontal projection diagram and a vertical projection diagram; The peak data includes horizontal peak data and vertical peak data; The peak correlation data includes horizontal peak correlation data and vertical peak correlation data; The horizontal wave peak data includes the horizontal wave peak coordinates and the horizontal wave peak projection value J corresponding to the horizontal wave peak coordinates; The vertical peak data includes vertical peak coordinates and vertical peak projection values ​​J' corresponding to the vertical peak coordinates; The horizontal peak association data includes the horizontal trough coordinates of the horizontal trough adjacent to each horizontal peak and the horizontal trough projection value J±1 corresponding to the horizontal trough coordinates; wherein J+1 is the horizontal trough projection value of the horizontal trough adjacent to the right of the J-th horizontal peak projection value; and J-1 is the horizontal trough projection value of the horizontal trough adjacent to the left of the J-th horizontal peak projection value. The vertical peak association data includes the vertical trough coordinates of the vertical trough adjacent to each vertical peak and the vertical trough projection value J'±1 corresponding to the vertical trough coordinates; wherein J'+1 is the vertical trough projection value of the vertical trough adjacent to the right of the J'th vertical peak projection value; and J'-1 is the vertical trough projection value of the vertical trough adjacent to the left of the J'th vertical peak projection value.

4. A video frame clipping image segmentation processing method according to claim 3, characterized in that: The texture descriptor can be constructed by using an LBP histogram.

5. A video frame clipping image segmentation processing method according to claim 4, characterized in that: The construction method of the first contour texture descriptor is consistent with the construction method of the contour texture descriptor.

6. A storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by the processor, the method for video frame clipping and image segmentation processing according to any one of claims 1 to 5 is implemented.

Images