Video frame clip image segmentation processing method

Abstract

The invention discloses a video frame clip image segmentation processing method, which comprises the following steps of: firstly, extracting video frame images from a video to be clipped, and acquiring a continuous frame image sequence of the video as a processing basis according to a time sequence and a timestamp of the video; the video is converted into an image sequence, so that each frame of image can be independently analyzed and processed in a subsequent processing step; performing target object recognition detection on each video frame image by using a preset target object recognition neural network, and outputting a plurality of target video frame images containing target objects; determining a target object image block of a target object in each target video frame image based on a preset target object contour; and carrying out iterative screening and analysis on the pixel data in the target object image block, and identifying edge pixel points of the target object so as to obtain a clear target object image.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a video frame clipping image segmentation processing method. Background Art

[0002] In the process of video production, filming personnel often use the "stand-in actor" method, using the stand-in actor to complete the dangerous actions of the main actor to shoot a draft video, split and cut out the image of the stand-in actor in the draft video, and then replace the image of the main actor with the image of the stand-in actor, so as to edit and obtain a complete video;

[0003] Traditional methods extract the target image based on simple background difference or color distribution segmentation of the complete draft video. However, these methods often do not work well for scenes with complex backgrounds or drastic lighting changes. Background difference is easily disturbed by background complexity, moving objects or lighting changes, resulting in the extracted target image containing a lot of noise or missing parts of the target object. Summary of the invention

[0004] The purpose of the present invention is to provide a video frame clipping image segmentation processing method to solve the above technical problems pointed out in the prior art.

[0005] The present invention provides a video frame clipping image segmentation processing method, comprising the following steps:

[0006] Acquire a video to be edited; perform a video frame extraction operation on the video to be edited according to a timestamp in a time series to obtain a plurality of video frame images;

[0007] Using a preset target object recognition neural network to recognize and detect each of the video frame images, and output a plurality of target video frame images containing the target object;

[0008] Determine a target object image block in the target video frame image according to a preset target object contour;

[0009] Iteratively screening and analyzing pixel data in the target object image block to identify edge pixel points of the target object; performing an image segmentation operation on the target object image block based on the edge pixel points of the target object to obtain a target object image.

[0010] Preferably, the preset of the target object contour includes the following steps:

[0011] Pre-set the contour image of the target object image;

[0012] A reference area of ​​the target object image in the contour picture is manually marked; and an image signal in the marked reference area is used as an image contour of the target object corresponding to the target object image.

[0013] Preferably, the iterative screening and analyzing of the pixel data in the target object image block to identify the edge pixel points of the target object comprises the following steps:

[0014] grayscale the target object image block, and then perform a bilateral filtering operation to obtain a first preprocessed image;

[0015] Generate a projection image by projecting each pixel point of the first preprocessed image;

[0016] Acquire each peak data and peak correlation data of the projection image;

[0017] Calculate and obtain the height difference between each of the peak data and the peak associated data corresponding to the peak data; determine whether the height difference is greater than or equal to a preset height difference threshold; if so, determine that the peak data is the first peak data;

[0018] The edge pixels of the target object are identified through secondary screening analysis based on the first peak data and the last peak data in the peak graph.

[0019] Preferably, the projection operation includes a horizontal projection operation and a vertical projection operation;

[0020] The projection diagram includes a horizontal projection diagram and a vertical projection diagram;

[0021] The peak data includes horizontal peak data and vertical peak data;

[0022] The peak correlation data includes horizontal peak correlation data and vertical peak correlation data;

[0023] The horizontal wave crest data includes horizontal wave crest coordinates and horizontal wave crest projection values ​​J corresponding to the horizontal wave crest coordinates;

[0024] The vertical wave crest data includes the vertical wave crest coordinates and the vertical wave crest projection value J' corresponding to the vertical wave crest coordinates;

[0025] The horizontal peak associated data includes the horizontal trough coordinates of the horizontal troughs adjacent to each horizontal peak and the horizontal trough projection values ​​J±1 corresponding to the horizontal trough coordinates; wherein J+1 is the horizontal trough projection value of the horizontal trough adjacent to the right of the J-th horizontal peak projection value; and J-1 is the horizontal trough projection value of the horizontal trough adjacent to the left of the J-th horizontal peak projection value.

[0026] The vertical peak associated data includes the vertical trough coordinates of the vertical troughs adjacent to each vertical peak and the vertical trough projection values ​​J'±1 corresponding to the vertical trough coordinates; wherein J'+1 is the vertical trough projection value of the vertical trough adjacent to the right of the J'th vertical peak projection value; and J'-1 is the vertical trough projection value of the vertical trough adjacent to the left of the J'th vertical peak projection value;

[0027] Preferably, the step of identifying edge pixels of the target object through secondary screening analysis based on the first peak data and the last peak data in the peak graph includes the following steps:

[0028] Traversing each of the first peak data, and calculating and obtaining the standard deviation between the first peak data and the last peak data in the peak graph;

[0029] Obtaining a standard deviation judgment threshold based on the grayscale values ​​of neighboring pixel points within a preset range of the pixel point corresponding to the first wave peak data; determining whether the standard deviation is greater than or equal to the standard deviation judgment threshold, and if so, determining the first wave peak data as the second wave peak data, and returning to the above operation to continue traversing until all the first wave peak data are traversed to obtain multiple second wave peak data;

[0030] Performing morphological processing on all pixel points corresponding to the second wave peak data to obtain edge pixel points of the target object;

[0031] Preferably, the step of performing a morphological processing operation on the pixel points corresponding to all the second wave peak data to obtain the edge pixel points of the target object comprises the following steps:

[0032] Clustering the pixel points corresponding to each of the second wave peak data to obtain a plurality of clusters; selecting the largest one among the clusters as a target cluster;

[0033] Get all the bounding pixels (x c ,y c ); for each of the enclosed pixel points (x c ,y c ) Calculate the local binary pattern LBP R (x c ,y c ) feature; based on the local binary pattern LBP R (x c 'y c ) feature constructs the contour texture descriptor of the regional image formed by the bounding box of the current target cluster;

[0034] Performing a morphological processing operation on each of the second wave peak data to obtain edge pixel points to be determined formed by all the second wave peak data;

[0035] The edge pixel points to be determined are iteratively optimized based on the contour texture descriptor to obtain the edge pixel points of the target object.

[0036] Preferably, the texture descriptor can be constructed by using an LBP histogram.

[0037] Preferably, the iterative optimization of the edge pixel points to be determined based on the contour texture descriptor to obtain the edge pixel points of the target object includes the following operation steps:

[0038] Get the processing parameter vector θ of the morphological processing operation t ; The processing parameter vector θ t Including the shape and size of the structural element and the number of expansions and erosions; and setting an iteration counter and an iteration optimization stop condition; the iteration optimization stop condition is a texture similarity metric minimum threshold; the number of iterations of the iteration counter is initially 0;

[0039] The iteration count of the iteration counter is increased by 1, and an edge enclosing image is obtained based on all the edge pixels to be determined; each edge pixel to be determined is traversed, and the image of the area to be determined corresponding to the edge pixel is obtained with a search box of a preset size with the edge pixel to be determined as the center;

[0040] Calculating each pixel point in the image of the area to be determined to obtain a first contour texture descriptor;

[0041] Calculating and acquiring a texture similarity measure F according to the first contour texture descriptor and the contour texture descriptor;

[0042] Determine whether the texture similarity measure F is greater than or equal to the preset iterative optimization stop condition; if so, output the edge pixel to be determined as the edge pixel of the final target object; if not, based on the number of iterations of the iteration counter and the morphological processing parameter vector θ t And the first contour texture descriptor is calculated to obtain a new morphological processing parameter θ t+1 ;

[0043] The new morphological processing parameter θ t+1 The calculation method is:

[0044]

[0045] The second peak data is processed based on the new morphological processing parameters to obtain new edge pixel points to be determined, and the above operation is returned until the texture similarity measure F is greater than or equal to the preset iterative optimization stop condition, and the edge pixel points of the target object are output.

[0046] Preferably, the construction method of the first contour texture descriptor is consistent with the construction method of the contour texture descriptor.

[0047] On the other hand, the present invention further provides a storage medium, characterized in that a computer program is stored in the storage medium, and when the computer program is executed by a processor, the above-mentioned video frame clipping image segmentation processing method is implemented.

[0048] Compared with the prior art, the embodiments of the present invention have at least the following technical advantages:

[0049] By analyzing the above-mentioned video frame editing image segmentation processing method provided by the present invention, it can be known that in a specific application, firstly, video frame images are extracted from the video to be edited, and a continuous frame image sequence of the video is obtained as a basis for processing according to the time series and timestamp of the video; the video is converted into an image sequence, so as to facilitate the subsequent processing steps to independently analyze and process each frame image; a preset target object recognition neural network (for example, a convolutional neural network based on YOLO) is used to perform recognition and detection of the target object on each video frame image, and a plurality of target video frame images containing the target object are output, that is, each frame is screened to ensure that only the frames containing the target object are further processed, so that the target object in the video to be edited can be quickly and accurately identified;

[0050] Among them, based on the preset target object contour, the specific image area of ​​the target object in each target video frame image, or the target object image block, is determined, a large amount of unnecessary and redundant image content is removed, and the focus is on the target object area, which improves the speed and accuracy of subsequent processing. The target object image block is determined by the preset contour, which effectively optimizes the image analysis process and reduces the consumption of computing resources; the pixel data in the target object image block is iteratively screened and analyzed to identify the edge pixels of the target object. These pixel data include color data (RGB), grayscale data and position data, which are used for subsequent image segmentation and edge detection, which improves the accuracy and clarity of the edge of the target object and lays the foundation for further image segmentation operations. Through fine pixel-level analysis, efficient recognition of the edge of the target object is ensured, thereby obtaining a clear target object image. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of the main flow of a video frame clipping image segmentation processing method provided in the first embodiment of the present invention;

[0052] Figure 2 A schematic diagram of the operation steps of iteratively screening and analyzing to identify edge pixels of a target object in a video frame clipping image segmentation processing method provided in the first embodiment of the present invention;

[0053] Figure 3 A schematic diagram of the operation steps of obtaining edge pixel points of a target object through morphological processing operations in a video frame clipping image segmentation processing method provided in the first embodiment of the present invention;

[0054] Figure 4 A schematic diagram of target cluster simulation in a video frame clipping image segmentation processing method provided in the first embodiment of the present invention;

[0055] Figure 5 A schematic diagram of the operation steps for obtaining edge pixel points of a target object in a video frame clipping image segmentation processing method provided in Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0056] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] The present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings.

[0058] Embodiment 1

[0059] like Figure 1 As shown, the first embodiment of the present invention provides a video frame clipping image segmentation processing method, including the following operation steps:

[0060] Step S10: obtaining a video to be edited; performing a video frame extraction operation on the video to be edited according to a timestamp on a time series to obtain a plurality of video frame images;

[0061] It should be noted that the above embodiment of the present application extracts video frame images through each timestamp of the time sequence of the video to be edited, and obtains all video frame images corresponding to the current video to be edited, which can provide an image basis for subsequent processing of the video frame images;

[0062] Step S20: using a preset target object recognition neural network to recognize and detect each of the video frame images, and outputting a plurality of target video frame images containing the target object;

[0063] It should be noted that the above-mentioned embodiment of the present application performs target object recognition operation on video frame images through a neural network model, and can quickly and accurately identify the target video frame image containing the target object in the video to be edited. The above-mentioned target object recognition neural network is a convolutional neural network established based on YOLO. YOLO is a popular target detection algorithm. Its core idea is to perform end-to-end target detection through a single neural network model, which is efficient and fast. The method of establishing a convolutional neural network based on YOLO is common knowledge among technicians in this field and will not be repeated in this application.

[0064] Step S30: determining a target object image block in the target video frame image according to a preset target object contour;

[0065] It should be noted that the above embodiment of the present application determines the image region where the target object contour in the target video frame image is located by a preset target object contour, that is, the above target object image block; through the extraction operation of the above target object image block, a large amount of unnecessary and redundant image content can be screened out, which has a strong speed-up effect on the subsequent image segmentation and recognition processing;

[0066] Specifically, in order to more accurately segment the target object image and improve the segmentation efficiency, in this embodiment, the target object image block (or target object image area) is first determined in the target video frame image information, and the subsequent edge detection image segmentation processing operation is only performed on the target object image block, and the images of other areas in the target video frame image except the target object image block are not subsequently processed, which greatly improves the processing efficiency;

[0067] In this embodiment, for some large interference images existing in the target video frame image obtained in step S10: for example, in the case where the target video frame image only contains part of the target object image and the rest are large interference images, step S20 can also have a screening function, that is, in order to improve the detection efficiency, the above-mentioned large interference images can be excluded through step S20, and only the image pixel data in the target object image block is processed in the next step;

[0068] In this embodiment, the target object contour can be: pre-setting a contour picture of the target object image, manually marking a reference area of ​​the target object image in the contour picture; and using the image signal in the marked reference area as the image contour of the target object corresponding to the target object image.

[0069] Step S40: Iteratively screening and analyzing the pixel data in the target object image block to identify the edge pixel points of the target object; performing an image segmentation operation on the target object image block based on the edge pixel points of the target object to obtain the target object image.

[0070] In the subsequent operation process, image segmentation can be performed based on the edge pixels of the target object to obtain the target object image;

[0071] The pixel data includes color data (i.e., RGB data), grayscale data, and position data of each pixel in the target object image block;

[0072] It should be noted that the above-mentioned embodiment of the present application first extracts video frame images from the video to be edited, and obtains a continuous frame image sequence of the video as a basis for processing according to the time series and timestamp of the video, thereby providing a static image representation of the video to be edited, and providing basic data for subsequent target object recognition and image processing; converting the video into an image sequence, so as to facilitate the subsequent processing steps to independently analyze and process each frame image; using a pre-set target object recognition neural network (such as a convolutional neural network based on YOLO) to perform target object recognition detection on each video frame image, and output multiple target video frame images containing the target object, that is, screening each frame to ensure that only the frames containing the target object are further processed, so as to quickly and accurately identify the target object in the video to be edited, and the efficiency and speed of the YOLO algorithm ensure the rapid processing of a large number of video frames, thereby improving the overall processing efficiency;

[0073] Based on the preset target object contour, the specific image area of ​​the target object in each target video frame image, or the target object image block, is determined, a large amount of unnecessary and redundant image content is removed, and the focus is on the target object area, which improves the speed and accuracy of subsequent processing. The target object image block is determined by the preset contour, which effectively optimizes the image analysis process and reduces the consumption of computing resources; the pixel data in the target object image block is iteratively screened and analyzed to identify the edge pixels of the target object. These pixel data include color data (RGB), grayscale data and position data, which are used for subsequent image segmentation and edge detection, which improves the accuracy and clarity of the edge of the target object and lays the foundation for further image segmentation operations. Through fine pixel-level analysis, efficient recognition of the edge of the target object is ensured, thereby obtaining a clear target object image.

[0074] Specifically, Figure 2 As shown, in step S40, iterative screening and analysis are performed to identify edge pixels of the target object according to the pixel data in the target object image block, including the following operation steps:

[0075] Step S41: grayscale the target object image block, and then perform a bilateral filtering operation to obtain a first preprocessed image;

[0076] It should be noted that the above-mentioned application performs grayscale processing on the target object image blocks in sequence, and after obtaining the grayscale image, performs bilateral filtering processing on the grayscale image to smooth the grayscale image to obtain a first preprocessed image, which can remove a large amount of noise in the image and make subsequent analysis and identification more accurate.

[0077] Step S42: generating a projection image by projecting each pixel point of the first preprocessed image;

[0078] The projection operation includes a horizontal projection operation and a vertical projection operation;

[0079] The projection diagram includes a horizontal projection diagram and a vertical projection diagram;

[0080] It should be noted that, in the above-mentioned embodiment of the present application, the first pre-processed image is horizontally projected to generate a horizontal projection image, and the first pre-processed image is vertically projected to generate a vertical projection image; this is because in the first pre-processed image, there are image contents other than the target object image, for example, in a first pre-processed image, the target object image is a cat, however, in the target object image block obtained by the above-mentioned target object contour recognition analysis, there are also image contents such as the sky, grass, etc. that are interference images that do not belong to the "cat"; the embodiment of the present application generates a projection image from the first pre-processed image through horizontal projection and vertical projection, which can more clearly and intuitively use a computer to identify and analyze the image content that does not belong to the cat, thereby performing a screening process on the image content such as the sky and grass;

[0081] The above-mentioned horizontal projection refers to summing up each row of pixels in the image along the vertical direction (usually from top to bottom) to obtain a one-dimensional array. Each element in this array represents the sum of the pixels on the corresponding row, reflecting the pixel distribution of the image in the vertical direction. Horizontal projection can help detect the row spacing, row position and row distribution density in the image;

[0082] The vertical projection refers to summing up the pixels of each column of the image along the horizontal direction (usually from left to right) to obtain a one-dimensional array. Each element in the array represents the sum of the pixels on the corresponding column, reflecting the pixel distribution of the image in the horizontal direction.

[0083] The distribution information of the projections in the projection map obtained by the horizontal projection and the vertical projection can be used to segment or separate different areas in the image.

[0084] Step S43: obtaining each peak data and peak correlation data of the projection diagram (including the horizontal projection diagram and the vertical projection diagram);

[0085] The peak data includes horizontal peak data and vertical peak data;

[0086] The peak correlation data includes horizontal peak correlation data and vertical peak correlation data;

[0087] The horizontal wave crest data includes horizontal wave crest coordinates and horizontal wave crest projection values ​​J corresponding to the horizontal wave crest coordinates;

[0088] The vertical wave crest data includes the vertical wave crest coordinates and the vertical wave crest projection value J' corresponding to the vertical wave crest coordinates;

[0089] The horizontal peak associated data includes the horizontal trough coordinates of the horizontal troughs adjacent to each horizontal peak and the horizontal trough projection values ​​J±1 corresponding to the horizontal trough coordinates; wherein J+1 is the horizontal trough projection value of the horizontal trough adjacent to the right of the J-th horizontal peak projection value; and J-1 is the horizontal trough projection value of the horizontal trough adjacent to the left of the J-th horizontal peak projection value.

[0090] The vertical peak associated data includes the vertical trough coordinates of the vertical troughs adjacent to each vertical peak and the vertical trough projection values ​​J'±1 corresponding to the vertical trough coordinates; wherein J'+1 is the vertical trough projection value of the vertical trough adjacent to the right of the J'th vertical peak projection value; and J'-1 is the vertical trough projection value of the vertical trough adjacent to the left of the J'th vertical peak projection value;

[0091] Step S44: Calculate and obtain the height difference between each of the peak data and the peak associated data corresponding to the peak data; determine whether the height difference is greater than or equal to a preset height difference threshold; if so (if not, determine that the pixel points of the peak data and the peak associated data corresponding to the peak data are in the same area, that is, the pixel points corresponding to the peak data and the pixel points corresponding to the peak associated data belong to the same pixel points in the target object image or the pixel points of the interference image), then determine that the peak data is the first peak data;

[0092] It should be noted that when the above-mentioned height difference is greater than or equal to the preset height difference threshold, it proves that the peak data domain and the peak associated data corresponding to the peak data do not belong to the same area, that is, the pixel point corresponding to the peak data and the pixel point corresponding to the peak associated data do not belong to the same object (that is, the pixel point of the target object image or the interference image); therefore, the embodiment of the present application determines the peak data as the first peak data for subsequent processing to analyze and identify the edge pixel points of the target object; and the peak data in the embodiment of the present application is the peak or trough phenomenon that occurs at the edge of the target object image and the interference image, and the embodiment of the present application is mainly to identify and analyze the edge pixel points of the target object, and to capture the target object image as the ultimate goal. Therefore, the embodiment of the present application does not perform the next step of processing for the situation where the peak data and the peak associated data belong to the same area, that is, when the height difference is less than the preset height difference threshold, the peak data and the peak associated data are shelved and the next step of processing is not performed.

[0093] Step S45: identifying edge pixels of the target object through secondary screening analysis based on the first peak data and the last peak data in the peak graph.

[0094] It should be noted that the above-mentioned embodiment of the present application first eliminates color information and retains the brightness (grayscale) information of the image through grayscale processing, which is conducive to the stability and efficiency of subsequent processing steps; through bilateral filtering, by considering the spatial distance and grayscale value similarity between pixels, noise is effectively removed to make the image clearer and more accurate, which is conducive to subsequent edge detection and analysis tasks; through horizontal projection and vertical projection operations, the line spacing, line position and distribution density information in the image are extracted to clearly and intuitively display the pixel distribution in the image, which is helpful for the computer to identify and analyze the content of different areas, especially for the distinction between target objects and interfering objects (such as sky and grass); the height difference between the peaks is calculated through the peak data and the peak correlation data to determine the boundary of the area, and according to the preset height difference threshold, it is judged whether the difference in projection values ​​between adjacent peaks is large enough to determine whether they belong to the same area; according to the relationship between the first peak data and the last peak data, the edge pixel points of the target object are further analyzed and identified, the boundary of the target object is determined, and accurate edge information is provided as basic data for further image segmentation;

[0095] The above steps S41 to S44 are the first screening, which divides the image content in the first preprocessed image into a target object image and an interference image. Step S45 is the second screening, which analyzes and identifies the edge pixels of the target object through the first peak data and the last peak data in the peak image. For specific operations, please refer to the subsequent processing steps S451-S453.

[0096] Specifically, in step S45, the edge pixel points of the target object are identified through secondary screening analysis according to the first peak data and the last peak data in the peak graph, including the following operation steps:

[0097] Step S451: traverse each of the first peak data, and calculate and obtain the standard deviation between the first peak data and the last peak data in the peak graph;

[0098] The standard deviation is calculated as:

[0099]

[0100] Where σ is the standard deviation; N is the number of corresponding coordinates; xi is the projection value corresponding to the i-th coordinate; μ is the average projection value;

[0101] It should be noted that by calculating the standard deviation between the first peak data and the last peak data in the peak graph, the change in the grayscale value of the pixel between the two peaks can be evaluated. If the standard deviation is small, it means that the pixel grayscale change in this interval is small, which may be due to the influence of the background or other non-target objects, rather than the real edge of the target object.

[0102] Step S452: obtaining a standard deviation judgment threshold based on the grayscale values ​​of neighboring pixel points within a preset range of the pixel points corresponding to the first peak data; determining whether the standard deviation is greater than or equal to the standard deviation judgment threshold, if so (then determining that there is no interfering image between the pixel points of the first peak data and the last peak data, that is, the pixel points between the first peak data and the last peak data all belong to the pixel points of the target object image), then determining the first peak data as the second peak data, and returning to the above operation to continue traversing until all the first peak data are traversed to obtain multiple second peak data;

[0103] If not, then (it is determined that there is an interference image between the pixel point corresponding to the first peak data and the pixel point corresponding to the last peak data in the peak image, which may also be caused by the image fluctuation of the interference image) the first peak data is screened out, and no further processing operation is performed;

[0104] It should be noted that the above standard deviation judgment threshold is calculated based on the grayscale values ​​of the neighboring pixels within a preset range of the pixel corresponding to the first wave peak data currently traversed. Specifically, the grayscale values ​​of the current pixel and the neighboring pixels are averaged first, and then the average value is subtracted from the value of each pixel to obtain the difference, and then the square of each difference is added and divided by the number of pixels to obtain the variance, and finally the square root of the variance is the standard deviation;

[0105] The advantage of this method is that the standard deviation judgment threshold is obtained by adaptive calculation based on the neighborhood pixel points of the first wave peak data, and the threshold is dynamically determined based on the actual data of the pixel points corresponding to the first wave peak data. Because different image areas may have different grayscale or color distributions, this adaptive method can more accurately distinguish the edge pixels of the target object from the background or interference pixels;

[0106] Distractor images tend to have a relatively uniform grayscale distribution because distractors are usually backgrounds or other non-target objects, whose grayscale changes are not significant, unlike the edge of the target object, which has obvious grayscale changes.

[0107] Therefore, the peak data belonging to the target object image in the first peak data (ie, the second peak data) can be determined and analyzed by the standard deviation obtained by projection.

[0108] Step S453: performing morphological processing on all pixel points corresponding to the second wave peak data to obtain edge pixel points of the target object;

[0109] It should be noted that the above morphological processing operations include basic processing parameters, such as:

[0110] 1. Structural element. Structural element is the core of morphological operation, which determines the effect and characteristics of processing. Morphological operation is to process edge pixels based on structural elements. These operations include corrosion, expansion, opening and closing operations, etc. The shapes of structural elements include: rectangle: suitable for processing horizontal and vertical edges; circle: has a uniform effect on edges in all directions, suitable for processing circular or curved features; cross: can retain details and have a certain ability to remove noise; line: used to detect edges in a specific direction; the size of structural elements is as follows: small size (such as 3x3): retains more details, but has a weaker ability to remove noise; large size (such as 7x7 or larger): can remove more noise, but may lose some details; the choice of structural elements directly affects the processing effect. For example, the structural element used to remove noise should be smaller than the smallest target feature to be retained, and the structural element used to fill holes should be larger than the largest hole to be filled;

[0111] 2. Processing parameters of the number of dilation and erosion. Number of dilation: Increasing the number of dilation will expand the bright area (foreground) and fill small holes and gaps; too much dilation may cause loss of details and coarsening of edges; Number of erosion: Increasing the number of erosion will reduce the bright area and remove small noise points; too much erosion may cause small features to disappear; the number of dilation and erosion needs to be balanced according to image features and processing goals. For example, to remove small noise points, a small amount of erosion can be performed first, and then the same number of dilation (i.e., opening operation) can be performed;

[0112] The basic operations of morphological processing operations include opening and closing operations. Opening operations: corrode first and then expand, remove small objects, smooth the contours of larger objects, and separate objects; closing operations: expand first and then corrode, fill small holes, connect neighboring objects, smooth the contours, and obtain image edges; the above-mentioned morphological processing operations are prior art and will not be described in detail in this application; and the focus of the embodiments of this application is not on obtaining edge pixels by morphological processing, but on iteratively optimizing the processing parameters of the morphological processing, so as to use the optimized processing parameters to detect and identify the edge pixels of the target object image;

[0113] In the above-mentioned embodiment of the present application, firstly, it is determined whether there is an interfering image based on the calculated standard deviation, and the standard deviation judgment threshold of the neighborhood grayscale value of the pixel point corresponding to the first peak data is calculated to adaptively determine whether there is an interfering image. This method can more accurately distinguish the edge pixels of the target object and the background or interfering pixels; further, the peak data without interference (i.e., the above-mentioned second peak data) is effectively screened out through the judgment conditions to ensure the accuracy of the subsequent processing steps. Through dynamic calculation based on the current pixel point and its neighborhood grayscale value, it can adapt to the grayscale or color distribution of different image areas, thereby improving the robustness and accuracy of the algorithm; the peak data of the target object is processed using morphological processing (such as dilation, erosion, etc.) to obtain its edge pixels, thereby improving the accuracy and stability of edge detection.

[0114] During the specific implementation of the above-mentioned embodiments of the present application, technicians have discovered that traditional morphological processing methods usually use fixed structural elements to perform dilation and erosion operations to obtain interference images, but this method is only suitable for image segmentation of simple scenes. In complex scenes, such as the "cat" in the above example, where the morphology changes greatly and the color changes of the cat are complex, this will lead to inaccurate edge extraction of the target object. Therefore, the use of morphological processing operations to process the pixel points corresponding to the second peak data to obtain the target object image still requires further research by technicians.

[0115] Specifically, Figure 3 As shown, in step S453, all pixel points corresponding to the second peak data are subjected to morphological processing operations to obtain edge pixel points of the target object, including the following operation steps:

[0116] Step S4531: clustering the pixel points corresponding to each of the second wave peak data to obtain a plurality of clusters; selecting the largest one among the clusters as the target cluster;

[0117] It should be noted that the above-mentioned embodiment of the present application clusters the pixel points of the second peak data, and obtains block images of the target object image obtained by merging multiple pixel points of the current target object image. For example, the largest cluster in the clusters is selected as the target cluster, and the image composed of all the pixels in this target cluster may be a region in the target object image. For example, if the entire target object image is a "cat", the corresponding region composed of the largest cluster is selected as a part of the cat's body or the entire body (for example, the corresponding region composed of the largest cluster is selected as the cat's head, or the corresponding region composed of the largest cluster is selected as the cat's tail); by selecting the target cluster, it can be used as the texture basis of the entire target object image, so as to obtain the overall target object image in subsequent operations;

[0118] In addition, it should be noted that the second peak data extracted above are usually the peak data corresponding to the pixel points of the target object image, and there are peak data corresponding to some pixel points inside the target object image, which are not determined as the second peak data because the pixel points inside the image are evenly distributed. Therefore, the second peak data above reflects the pixel points at the edge of the target object image and the interference image (such as Figure 4 shown).

[0119] Step S4532: Obtain all bounding pixel points (x c ,y c ); for each of the enclosed pixel points (x c ,y c ) Calculate the local binary pattern LBP R (x c ,y c ) features; based on the local binary pattern LBP R (x c ,y c ) feature constructs the contour texture descriptor of the regional image formed by the bounding box of the current target cluster;

[0120] It should be noted that in target object recognition, texture features are an important basis for recognition. By calculating texture descriptors, we can better characterize the characteristics of the target area and provide a basis for subsequent edge optimization;

[0121] The local binary pattern describes the texture features of each pixel by comparing the grayscale values ​​of the central pixel with the surrounding pixels; suppose an area of ​​the cat's head is selected, and then the local binary pattern LBP of each pixel in the area is calculated. R (x c ,y c) features, by comparing the texture features of the cat’s head hair, a contour texture descriptor can be calculated to evaluate the difference between the head area and the background;

[0122] The above texture descriptor can be constructed through the LBP histogram, that is, by counting the frequency of occurrence of each LBP value in the entire area as a texture descriptor; at the same time, it can also be constructed through statistical features, that is, calculating the mean, variance, skewness, kurtosis and other statistics of the LBP value as a texture descriptor.

[0123] Step S4533: performing a morphological processing operation on each of the second wave peak data to obtain edge pixel points to be determined formed by all the second wave peak data;

[0124] It should be noted that the above-mentioned embodiment of the present application selects the shape and size (i.e., dimension) of the structural element, as well as the number of dilation and erosion operations through conventional morphological processing operations, and obtains the edge pixel points to be determined through these basic operations; the above-mentioned conventional morphological processing is common knowledge of those skilled in the art, and the embodiments of the present application will not be repeated here.

[0125] Step S4534: iteratively optimize the edge pixel points to be determined based on the contour texture descriptor to obtain edge pixel points of the target object.

[0126] It should be noted that, in the above-mentioned embodiment of the present application, the pixel points corresponding to the second peak data are first clustered to obtain multiple clusters, and the largest cluster is selected as the target cluster. The target cluster represents a main part of the target object image, such as the body or head of a cat, and provides a texture basis for the entire target object image for subsequent operations.

[0127] The main part of the target object is identified through clustering, which provides a basis for subsequent texture analysis and edge optimization; further, by obtaining all the surrounding pixels in the bounding box of the target cluster, the local binary pattern (LBP) features of the surrounding pixels are calculated, and the contour texture descriptor is constructed based on the LBP features. The local texture information of the target area is captured through the LBP features; the contour texture descriptor is constructed to represent the texture features of the target area, providing important texture information for subsequent edge optimization; then, the second wave peak data is morphologically processed to obtain the edge pixels to be determined, and the edge information is preliminarily processed through morphological operations to provide candidate edge pixels for subsequent edge optimization; based on the contour texture descriptor, the edge pixels to be determined are iteratively optimized to obtain the final target object edge pixels, and the edge pixels are optimized using texture information, and the accuracy of edge recognition is improved through an iterative process.

[0128] Specifically, Figure 5As shown, in step S4534, the edge pixel points to be determined are iteratively optimized based on the contour texture descriptor to obtain the edge pixel points of the target object, including the following operation steps:

[0129] Step S45341: Obtain the processing parameter vector θ of the morphological processing operation t ; The processing parameter vector θ t Including the shape and size of the structural element and the number of expansions and erosions; and setting an iteration counter and an iteration optimization stop condition; the iteration optimization stop condition is a texture similarity metric minimum threshold; the number of iterations of the iteration counter is initially 0;

[0130] Step S45342: adding 1 to the iteration count of the iteration counter, and acquiring an edge surrounding image based on all the edge pixels to be determined; traversing each of the edge pixels to be determined, taking the edge pixel to be determined as the center, and acquiring an image of the area to be determined corresponding to the edge pixel using a search box of a preset size;

[0131] Step S45343: Calculate each pixel point in the image of the area to be determined to obtain a first contour texture descriptor;

[0132] It should be noted that the first contour texture descriptor is also obtained by using the local binary pattern LBP of the image in the to-be-determined area. R (x c ′y c ) Features are constructed by LBP histogram or statistical features;

[0133] Step S45344: Calculate and obtain a texture similarity measure F according to the first contour texture descriptor and the contour texture descriptor;

[0134] It should be noted that the above texture similarity metric can be calculated by histogram similarity or statistical feature distance. If the first contour texture descriptor and the contour texture descriptor are constructed using an LBP histogram, the histogram intersection or chi-square distance can be calculated to obtain the texture similarity metric;

[0135] If the first contour texture descriptor and the contour texture descriptor are constructed using statistical features, a Euclidean distance or a Mahalanobis distance may be calculated to calculate a texture similarity measure;

[0136] It should also be noted that the construction method of the first contour texture descriptor is consistent with the construction method of the contour texture descriptor.

[0137] Step S45345: Determine whether the texture similarity measure F is greater than or equal to a preset iterative optimization stop condition; if so, output the edge pixel to be determined as the edge pixel of the final target object; if not, based on the number of iterations of the iteration counter and the morphological processing parameter vector θ t And the first contour texture descriptor is calculated to obtain a new morphological processing parameter θ t+1 ;

[0138] The new morphological processing parameter θ t+1 The calculation method is:

[0139]

[0140] Step S45346: Process the second peak data based on the new morphological processing parameters to obtain new edge pixel points to be determined, and return to the above operation until the texture similarity measure F is greater than or equal to the preset iterative optimization stop condition, and output the edge pixel points of the target object.

[0141] It should be noted that the above-mentioned embodiment of the present application first obtains the parameter vector required for the morphological processing operation, including the shape and size of the structural element, and the number of expansions and erosions. These parameters determine the accuracy and effect of the morphological operation in edge detection; the edge surrounding image of each edge pixel to be determined is used to provide a data basis for subsequent morphological parameter optimization to output the edge pixel of the target object image; further, for each pixel in the image of the area to be determined, a first contour texture descriptor is calculated to represent the local texture features around each pixel, and the texture similarity measure F is calculated by using the first contour texture descriptor and the known contour texture descriptor to measure the accuracy of the edge pixel of the target object in the current iteration and the degree of closeness to the expectation; when the expected situation is not met, the morphological processing parameters are optimized by iterative optimization to obtain new edge pixels to be determined, until the texture similarity measure F reaches the preset optimization stop condition, thereby ensuring that the algorithm can gradually optimize the precision and accuracy of edge detection through multiple rounds of iterations.

[0142] Embodiment 2

[0143] The present invention also provides a storage medium, in which a computer program is stored. When the computer program is executed by a processor, the above-mentioned video frame clipping image segmentation processing method is implemented.

[0144] In summary, a video frame clipping image segmentation processing method proposed in an example of the present invention first extracts video frame images from the video to be clipped, obtains a continuous frame image sequence of the video as a processing basis according to the time series and timestamp of the video, and provides a static image representation of the video to be clipped; after converting the video into an image sequence, a preset target object recognition neural network is used to perform target object recognition detection on each video frame image, and multiple target video frame images containing the target object are output, that is, each frame is screened to ensure that only frames containing the target object are further processed; based on the preset target object contour, the specific target object image block of the target object in each target video frame image is determined, a large amount of unnecessary and redundant image content is removed, and the focus is on the target object area, thereby improving the speed and accuracy of subsequent processing, and the target object image block is determined by the preset contour, which effectively optimizes the image analysis process and reduces the consumption of computing resources; the pixel data in the target object image block is iteratively screened and analyzed to identify the edge pixel points of the target object;

[0145] In the specific operation, firstly, the grayscale processing operation is performed to eliminate the color information and retain the brightness (grayscale) information of the image, which is conducive to the stability and efficiency of the subsequent processing steps; the bilateral filtering processing is performed by considering the spatial distance and grayscale value similarity between pixels to effectively remove noise, make the image clearer and more accurate, and facilitate the subsequent edge detection and analysis tasks; the horizontal projection and vertical projection operations are used to extract the line spacing, line position and distribution density information in the image, and the pixel distribution in the image is clearly and intuitively displayed, which is helpful for the computer to identify and analyze the content of different areas, especially for the distinction between target objects and interfering objects (such as sky and grass); the height difference between the peaks is calculated through the peak data and the peak correlation data to determine the boundary of the area, and according to the preset height difference threshold, it is judged whether the difference in projection values ​​between adjacent peaks is large enough to determine whether they belong to the same area; according to the relationship between the first peak data and the last peak data, the edge pixel points of the target object are further analyzed and identified, the boundary of the target object is determined, and accurate edge information is provided as the basic data for further image segmentation;

[0146] During the second screening and analysis, the presence of an interfering image is determined based on the calculated standard deviation. The presence of an interfering image is adaptively determined by calculating the standard deviation judgment threshold of the neighborhood grayscale value of the pixel point corresponding to the first peak data. This method can more accurately distinguish the edge pixels of the target object from the background or interfering pixels. Furthermore, the peak data without interference is effectively screened out through the judgment conditions to ensure the accuracy of the subsequent processing steps. The grayscale or color distribution of different image areas can be adapted through dynamic calculation based on the current pixel point and its neighborhood grayscale value, thereby improving the robustness and accuracy of the algorithm. The peak data of the target object is morphologically processed using the morphological processing parameters obtained through iterative optimization to obtain its edge pixels, thereby improving the accuracy and stability of edge detection, thereby obtaining a clear image of the target object.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. A person skilled in the art may modify the technical solutions described in the above embodiments, or replace part or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video frame clipping image segmentation processing method, characterized in that: The steps are as follows: Acquire a video to be edited; perform a video frame extraction operation on the video to be edited according to a timestamp in a time series to obtain a plurality of video frame images; Using a preset target object recognition neural network to recognize and detect each of the video frame images, and output a plurality of target video frame images containing the target object; Determine a target object image block in the target video frame image according to a preset target object contour; Iteratively screening and analyzing pixel data in the target object image block to identify edge pixel points of the target object; An image segmentation operation is performed on the target object image block based on edge pixels of the target object to obtain a target object image.

2. A video frame clipping image segmentation processing method as claimed in claim 1, characterized in that: The preset of the target object contour includes the following steps: Pre-set the contour image of the target object image; A reference area of ​​the target object image in the contour picture is manually marked; and an image signal in the marked reference area is used as an image contour of the target object corresponding to the target object image.

3. A video frame clipping image segmentation processing method as claimed in claim 2, characterized in that: The iterative screening and analysis to identify edge pixels of the target object according to the pixel data in the target object image block comprises the following steps: grayscale the target object image block, and then perform a bilateral filtering operation to obtain a first preprocessed image; Generate a projection image by projecting each pixel point of the first preprocessed image; Acquire each peak data and peak correlation data of the projection image; Calculate and obtain the height difference between each of the peak data and the peak associated data corresponding to the peak data; Determining whether the height difference is greater than or equal to a preset height difference threshold; If yes, determining that the peak data is the first peak data; The edge pixels of the target object are identified through secondary screening analysis based on the first peak data and the last peak data in the peak graph.

4. A video frame clipping image segmentation processing method as claimed in claim 3, characterized in that: The projection operation includes a horizontal projection operation and a vertical projection operation; The projection diagram includes a horizontal projection diagram and a vertical projection diagram; The peak data includes horizontal peak data and vertical peak data; The peak correlation data includes horizontal peak correlation data and vertical peak correlation data; The horizontal wave crest data includes horizontal wave crest coordinates and horizontal wave crest projection values ​​J corresponding to the horizontal wave crest coordinates; The vertical wave crest data includes the vertical wave crest coordinates and the vertical wave crest projection value J' corresponding to the vertical wave crest coordinates; The horizontal peak associated data includes the horizontal trough coordinates of the horizontal troughs adjacent to each horizontal peak and the horizontal trough projection values ​​J±1 corresponding to the horizontal trough coordinates; wherein J+1 is the horizontal trough projection value of the horizontal trough adjacent to the right of the J-th horizontal peak projection value; and J-1 is the horizontal trough projection value of the horizontal trough adjacent to the left of the J-th horizontal peak projection value. The vertical peak associated data includes the vertical trough coordinates of the vertical troughs adjacent to each vertical peak and the vertical trough projection values ​​J'±1 corresponding to the vertical trough coordinates; wherein J'+1 is the vertical trough projection value of the vertical trough adjacent to the right of the J'th vertical peak projection value; and J'-1 is the vertical trough projection value of the vertical trough adjacent to the left of the J'th vertical peak projection value.

5. A video frame clipping image segmentation processing method as claimed in claim 4, characterized in that: The step of identifying edge pixels of the target object through secondary screening analysis based on the first peak data and the last peak data in the peak graph includes the following steps: Traversing each of the first peak data, and calculating and obtaining the standard deviation between the first peak data and the last peak data in the peak graph; Obtaining a standard deviation judgment threshold based on the grayscale values ​​of neighboring pixels within a preset range of pixels corresponding to the first wave peak data; Determine whether the standard deviation is greater than or equal to a standard deviation determination threshold, and if so, determine the first wave peak data as the second wave peak data, and return to the above operation to continue traversing until all the first wave peak data are traversed to obtain a plurality of second wave peak data; The pixel points corresponding to all the second wave peak data are subjected to morphological processing operations to obtain edge pixel points of the target object.

6. A video frame clipping image segmentation processing method as claimed in claim 5, characterized in that: The step of performing morphological processing on the pixel points corresponding to all the second wave peak data to obtain the edge pixel points of the target object comprises the following steps: Clustering the pixel points corresponding to each of the second wave peak data to obtain a plurality of clusters; selecting the largest one among the clusters as a target cluster; Get all the bounding pixels (x c ,y c ); for each of the enclosed pixel points (x c ,y c ) Calculate the local binary pattern LBP R (x c ,y c )feature; Based on the local binary pattern LBP R (x c ,y c ) feature constructs the contour texture descriptor of the regional image formed by the bounding box of the current target cluster; Performing a morphological processing operation on each of the second wave peak data to obtain edge pixel points to be determined formed by all the second wave peak data; The edge pixel points to be determined are iteratively optimized based on the contour texture descriptor to obtain the edge pixel points of the target object.

7. A video frame clipping image segmentation processing method as claimed in claim 6, characterized in that: The texture descriptor can be constructed by using an LBP histogram.

8. A video frame clipping image segmentation processing method as claimed in claim 7, characterized in that: The iterative optimization of the edge pixel points to be determined based on the contour texture descriptor to obtain the edge pixel points of the target object includes the following operation steps: Get the processing parameter vector θ of the morphological processing operation t ; The processing parameter vector θ t This includes the shape and size of structural elements and the number of dilations and erosions; And set the iteration counter and iterative optimization stop condition; The iterative optimization stopping condition is the texture similarity metric minimum threshold; the number of iterations of the iteration counter is initially 0; The iteration count of the iteration counter is increased by 1, and an edge enclosing image is obtained based on all the edge pixels to be determined; each edge pixel to be determined is traversed, and the image of the area to be determined corresponding to the edge pixel is obtained with a search box of a preset size with the edge pixel to be determined as the center; Calculating each pixel point in the image of the area to be determined to obtain a first contour texture descriptor; Calculating and acquiring a texture similarity measure F according to the first contour texture descriptor and the contour texture descriptor; Determine whether the texture similarity measure F is greater than or equal to the preset iterative optimization stop condition; if so, output the edge pixel to be determined as the edge pixel of the final target object; if not, based on the number of iterations of the iteration counter and the morphological processing parameter vector θ t And the first contour texture descriptor is calculated to obtain a new morphological processing parameter θ t+1 ; The new morphological processing parameter θ t+1 The calculation method is: The second peak data is processed based on the new morphological processing parameters to obtain new edge pixel points to be determined, and the above operation is returned until the texture similarity measure F is greater than or equal to the preset iterative optimization stop condition, and the edge pixel points of the target object are output.

9. A video frame clipping image segmentation processing method as claimed in claim 8, characterized in that: The construction method of the first contour texture descriptor is consistent with the construction method of the contour texture descriptor.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the video frame clipping image segmentation processing method described in any one of claims 1 to 9 is implemented.

Images