Occluded vehicle image generation method and device, and electronic device

Abstract

The application discloses a kind of shelter vehicle image generation method, device and electronic equipment.Therein, the method includes: obtaining vehicle image and shelter image;According to shelter image, determine rough segmentation shelter image;Determine the inflation edge region corresponding to rough segmentation shelter image;Determine the tripartite graph corresponding to inflation edge region, wherein the tripartite graph includes foreground area, background area and mixed pixel area;According to shelter image and tripartite graph, generate fine segmentation shelter foreground image;According to vehicle image, fine segmentation shelter foreground image and vehicle sheltered position coordinates, obtain shelter vehicle image.The application solves the technical problem that accurate, actual shelter vehicle image is difficult to generate in related art.

Description

Technical Field

[0001] This invention relates to the field of images, and more specifically, to a method, apparatus, and electronic device for generating images of occluded vehicles. Background Technology

[0002] Today, artificial intelligence (AI) is ubiquitous in daily life, with autonomous driving, license plate recognition, and facial recognition becoming key technologies in smart cities. The development of AI relies heavily on big data; most models require high-quality data for effective training. However, currently, data in many scenarios is insufficient for training needs, and issues such as poor data quality and difficulties in data labeling hinder the improvement of model performance.

[0003] In fields such as license plate recognition and autonomous driving, the diversity and quality of vehicle images are crucial for improving model generalization. Obtaining occluded vehicle data is particularly challenging. Occluded vehicle images refer to images where parts of a vehicle are obscured by objects in different scenarios, resulting in the loss of some features. Due to the complexity of occlusions and surrounding environments in various scenarios, manual screening is extremely resource-intensive and cannot create suitable datasets for diverse situations. For example, in license plate recognition tasks, vehicle license plates are often easily obscured by various objects, leading to misjudgments by image- or video-based license plate recognition algorithms. Therefore, it is necessary to optimize the model for such scenarios. However, license plate occlusion scenarios for license plateless vehicles are extremely complex, including occlusion by pedestrians, other motor vehicles and non-motor vehicles, and malicious occlusion. In these cases, generating occluded vehicle image data for the required scenarios and expanding the data can solve most of the data shortage problem and significantly improve the model's generalization ability.

[0004] Currently, obtaining transparent foreground images of occluded objects mainly involves semantic segmentation, classifying each pixel of the image to obtain specific semantic regions, such as human bodies and vehicles. However, such methods require a large amount of manually labeled data for training, and obtaining training images for some semantics, such as tree branches and plastic bags, is quite difficult. This leads to significant limitations in these methods.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This invention provides a method, apparatus, and electronic device for generating occluded vehicle images, thereby at least solving the technical problem in the related art of generating accurate and realistic occluded vehicle images.

[0007] According to one aspect of the present invention, a method for generating an occluded vehicle image is provided, comprising: acquiring a vehicle image and an occluding object image; determining a coarsely segmented occluding object image based on the occluding object image; determining an inflated edge region corresponding to the coarsely segmented occluding object image; determining a tripartite image corresponding to the inflated edge region, wherein the tripartite image includes a foreground region, a background region, and a mixed pixel region; generating a finely segmented occluding object foreground image based on the occluding object image and the tripartite image; and acquiring an occluded vehicle image based on the vehicle image, the finely segmented occluding object foreground image, and the coordinates of the occluded vehicle position, wherein the coordinates of the occluded vehicle position are coordinates in a target coordinate system, the target coordinate system being established based on the vehicle image.

[0008] Optionally, determining the coarsely segmented occluded image based on the occluded image includes: extracting image features of the occluded image based on a salient object detection algorithm; and determining the coarsely segmented occluded image based on the image features.

[0009] Optionally, determining the dilated edge region corresponding to the coarsely segmented occluded image includes: upsampling the coarsely segmented occluded image using the bilinear interpolation method to obtain a first sub-occluded image; and downsampling the first sub-occluded image using the mean method to determine the dilated edge region.

[0010] Optionally, determining the tri-image corresponding to the dilated edge region includes: determining a background-removed image corresponding to the dilated edge region based on a region growing algorithm; determining a mixed pixel image corresponding to the dilated edge region based on an edge detection method; determining a pixel segmentation image corresponding to the dilated edge region based on a maximum inter-class variance algorithm; and determining the tri-image based on the background-removed image, the mixed pixel image, the pixel segmentation image, and the coarsely segmented occlusion image.

[0011] Optionally, before obtaining the occluded vehicle image based on the vehicle image and the finely segmented foreground image of the occluded object and the coordinates of the occluded vehicle position, the method further includes: determining the coordinates of the occluded vehicle position based on an object detection algorithm.

[0012] Optionally, obtaining the occluded vehicle image based on the vehicle image, the finely segmented foreground image of the occluder, and the coordinates of the occluded vehicle position includes: obtaining a random offset; and obtaining the occluded vehicle image based on the random offset, the vehicle image, the finely segmented foreground image of the occluder, and the coordinates of the occluded vehicle position.

[0013] Optionally, after obtaining the occluded vehicle image based on the vehicle image, the finely segmented foreground image of the occluder, and the coordinates of the occluded vehicle position, the method further includes: determining an occluder position label based on the vehicle image, the finely segmented foreground image of the occluder, and the coordinates of the occluded vehicle position; and optimizing the color parameters of the occluded vehicle image using a generator based on the occluder position label and the occluded vehicle image to obtain an optimized occluded vehicle image.

[0014] According to one aspect of the present invention, an apparatus for generating an occluded vehicle image is provided, comprising: a first acquisition module for acquiring a vehicle image and an occluder image; a first determination module for determining a coarsely segmented occluder image based on the occluder image; a second determination module for determining an inflated edge region corresponding to the coarsely segmented occluder image; a third determination module for determining a tripartite image corresponding to the inflated edge region, wherein the tripartite image includes a foreground region, a background region, and a mixed pixel region; a generation module for generating a finely segmented occluder foreground image based on the occluder image and the tripartite image; and a second acquisition module for acquiring an occluded vehicle image based on the vehicle image, the finely segmented occluder foreground image, and the coordinates of the occluded vehicle position, wherein the coordinates of the occluded vehicle position are coordinates in a target coordinate system, the target coordinate system being established based on the vehicle image.

[0015] According to one aspect of the present invention, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the occluded vehicle image generation method described in any of the preceding embodiments.

[0016] According to one aspect of the present invention, a computer-readable storage medium is provided, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the occluded vehicle image generation method described above.

[0017] In this embodiment of the invention, by acquiring a vehicle image and an occlusion image, a coarse segmentation of the occlusion image is determined based on the occlusion image. Then, the dilated edge region corresponding to the coarse segmentation of the occlusion image is determined, and the corresponding triangulation is determined. Finally, based on the occlusion image and the triangulation, a fine segmentation of the occlusion foreground image is generated, resulting in a detailed occlusion segmentation foreground image. Since a detailed occlusion segmentation foreground image is obtained, the occluded vehicle image is clear, accurate, and realistic when the occluded vehicle image and the coordinates of the occluded vehicle position are finely segmented, achieving the goal of obtaining an occluded vehicle image. This solves the technical problem in related technologies of the difficulty in generating accurate and realistic occluded vehicle images. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0019] Figure 1 This is a flowchart of a method for generating an image of an obscured vehicle according to an embodiment of the present invention;

[0020] Figure 2 This is a general framework flowchart provided by an optional embodiment of the present invention;

[0021] Figure 3 This is a general flowchart of the generation of finely segmented foreground images of occluded objects provided by an optional embodiment of the present invention;

[0022] Figure 4 These are the main steps of the generator optimizing the image provided in the optional embodiments of the present invention;

[0023] Figure 5 This is a structural block diagram of an occluded vehicle image generation device according to an embodiment of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0027] The YOLOv5 object detection method: Prior to YOLOv1, the R-CNN series of object detection algorithms were essentially classification problems. The basic idea was to slide a window across the image, traversing the entire image and classifying each window. Then, regression methods were used to adjust the precise bounding boxes of objects to achieve detection and localization. The later improved Fast R-CNN series improved speed. The basic process involves first generating a large number of region proposals (potential target regions) using a CNN, then using another CNN to extract features from these potential target regions and classify them. YOLOv5 is an improvement based on YOLOv1.

[0028] U2Net is a salient object detection algorithm with a two-level embedded U-shaped network structure. This novel network structure does not require pre-training and can achieve excellent results when trained from scratch. This network structure maintains high resolution while increasing the number of network layers.

[0029] Seeded Region Growing (SRG) algorithm: The basic idea of ​​SRG is to group pixels with similar properties together to form regions. First, for each region to be segmented, a seed pixel is found as the growth point. Then, pixels in the surrounding neighborhood of the seed pixel that have the same or similar properties are merged into the region containing the seed pixel. New pixels continue to grow outwards as seeds until no more pixels meeting the conditions can be included, thus forming a region.

[0030] Sobel edge detection method: The Sobel operator is commonly used for edge detection. It is the most commonly used edge detection operator with coarse accuracy and has been widely used for decades.

[0031] OTUUS (Maximum Inter-Class Variance) algorithm: This method mainly uses a threshold to segment the foreground and background. The optimal threshold is determined by maximizing the inter-class variance. It divides the image into background and foreground based on the grayscale characteristics of the image. Segmentation that maximizes the inter-class variance means minimizing the probability of misclassification.

[0032] Image matting, also known as image matting technology, focuses on optimizing uncertain regions. The most direct way to understand a task is to look at its evaluation metrics, which truly reflect the task's requirements. One metric in matting is the difference in gradient calculations, which clearly targets the prediction of blurry areas. Compared to segmentation, which focuses more on the overall picture, matting may be more detail-oriented. Furthermore, matting typically provides a ternary map to indicate defined and uncertain regions. The output of matting is more like a probability map, corresponding to the probability of belonging to the foreground.

[0033] Example 1

[0034] According to an embodiment of the present invention, an embodiment of a method for generating an occluded vehicle image is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0035] Figure 1 This is a flowchart of a method for generating occluded vehicle images according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0036] Step S102: Acquire vehicle image and occlusion image;

[0037] In step S102 described in this application, the vehicle image is a complete image of the vehicle, i.e., an image that is not obstructed by any object. The vehicle can be facing any direction, as long as the vehicle can be fully displayed. The obstruction image is an image of the obstruction, and the obstruction is any object that can obstruct the vehicle, such as other vehicles, green plants, traffic light poles, etc., which are not limited here.

[0038] Step S104: Determine the coarsely segmented occlusion image based on the occlusion image;

[0039] In step S104 described in this application, the coarsely segmented occluded image can be understood as a coarsely cut-out image, that is, an image in which the occluded object is coarsely cut out. This is so that a finely segmented occluded foreground image can be obtained from the coarsely segmented occluded image.

[0040] Step S106: Determine the dilated edge region corresponding to the coarsely segmented occluded image;

[0041] In step S106 of this application, determining the dilated edge region corresponding to the coarsely segmented occluded image ensures that the occluded object can be completely separated without fragmentation, avoiding unsuitable images with partial cutout and partial residue. Furthermore, it reduces the computation time of subsequent algorithms and minimizes interference from redundant foreground and background information, thus enabling better execution of the method described in this application.

[0042] Step S108: Determine the three-part map corresponding to the dilated edge region, wherein the three-part map includes the foreground region, the background region, and the blended pixel region;

[0043] In step S108 described in this application, determining the three-part image corresponding to the bulging edge region can lay the foundation for subsequent fine image matting.

[0044] Step S110: Generate a finely segmented foreground image of the occluded object based on the occluded object image and the three-part image;

[0045] In step S110 described in this application, since the three-part image includes a foreground region, a background region, and a mixed pixel region, the determination of different regions enables the subsequent generation of a finely segmented foreground image of the occluded object to take into account multiple aspects, resulting in a more accurate image.

[0046] Step S112: Based on the vehicle image, the foreground image of the occluder and the coordinates of the occluded vehicle position are further segmented to obtain the occluded vehicle image. The coordinates of the occluded vehicle position are the coordinates in the target coordinate system, which is established based on the vehicle image.

[0047] In step S112 described in this application, the purpose of obtaining an image of an occluded vehicle is achieved, which can be applied to a variety of scenarios.

[0048] Through the above steps, by acquiring vehicle and occlusion images, a coarse segmentation of the occlusion image is determined based on the occlusion image. Then, the dilated edge region corresponding to the coarse segmentation is identified, and the corresponding triangulation is determined. Finally, based on the occlusion image and the triangulation, a fine segmentation of the occlusion foreground image is generated, resulting in a detailed occlusion foreground image. Because a detailed occlusion foreground image is obtained, the occluded vehicle image is clear, accurate, and realistic when obtained by finely segmenting the occlusion foreground image and the vehicle's occluded position coordinates based on the vehicle image. This achieves the goal of acquiring occluded vehicle images and solves the technical problem in related technologies of generating accurate and realistic occluded vehicle images.

[0049] As an optional embodiment, determining a coarsely segmented occluded image based on the occluded image includes: extracting image features of the occluded image based on a salient object detection algorithm; and determining the coarsely segmented occluded image based on the image features.

[0050] In this embodiment, a salient object detection algorithm is used to extract image features from the occluded object image. This algorithm is effective because it can quickly and efficiently extract useful regions from an image for further analysis, detect obvious occlusions, and segment the object's accurate region. Therefore, it is suitable for determining coarsely segmented occluded object images, ensuring the accuracy of the determined coarsely segmented occluded object image.

[0051] As an optional embodiment, determining the dilated edge region corresponding to the coarsely segmented occluded image includes: upsampling the coarsely segmented occluded image according to the bilinear interpolation method to obtain a first sub-occluded image; and downsampling the first sub-occluded image according to the mean method to determine the dilated edge region.

[0052] In this embodiment, bilinear interpolation for upsampling and mean interpolation for downsampling are used sequentially to determine the dilated edge region. Upsampling refers to enlarging the image. Bilinear interpolation is effective at upsampling images, and it calculates one point in the new image from four (2*2) points in the original image, resulting in better performance than nearest neighbor interpolation and faster speed than bicubic interpolation, achieving a better balance between speed and performance. Downsampling restores resolution. Downsampling using the mean interpolation method effectively restores resolution, ensuring that the final determined dilated edge region is clear.

[0053] As an optional embodiment, determining the tri-image corresponding to the dilated edge region includes: determining the background-removed image corresponding to the dilated edge region based on a region growing algorithm; determining the mixed pixel image corresponding to the dilated edge region based on an edge detection method; determining the pixel segmentation image corresponding to the dilated edge region based on a maximum inter-class variance algorithm; and determining the tri-image based on the background-removed image, the mixed pixel image, the pixel segmentation image, and the coarsely segmented occlusion image.

[0054] In this embodiment, different algorithms are used to determine different parts of the trisection image. Among them, the region growing algorithm is a traditional region-based image segmentation algorithm. Region growing is the process of continuously combining pixels or small regions into larger regions according to predefined growth rules. Specifically, region growing starts from a set of initial seed points, and through predefined region growing rules, adds neighboring pixels with similar properties to each seed point, forming the final grown region when the termination condition of region growing is met. This effectively obtains the background-free image corresponding to the dilated edge region. The edge detection algorithm can detect and identify sets of pixels with drastic brightness changes in the image, thus clearly distinguishing edge regions from non-edge regions and better determining the pixel segmentation image corresponding to the dilated edge region. The maximum inter-class variance algorithm divides all pixels in the image into two classes, unaffected by image brightness contrast, thus better determining the pixel segmentation image corresponding to the dilated edge region. After obtaining the above results, the linear operations on these results can accurately obtain the classification of foreground, background, and mixed pixels, thereby generating the trisection image. This ensures the accuracy and effectiveness of the trisection image.

[0055] As an optional embodiment, before obtaining the image of the occluded vehicle, the foreground image of the occluded object and the coordinates of the occluded vehicle position are further segmented based on the vehicle image. The method also includes determining the coordinates of the occluded vehicle position based on an object detection algorithm.

[0056] In this embodiment, for example, the occluded position of the vehicle can be automatically located based on the YOLOv5 target detection method, so that the determined occluded position of the vehicle is consistent with reality.

[0057] As an optional embodiment, based on the vehicle image, the foreground image of the occluder and the coordinates of the occluded vehicle position are further segmented to obtain the occluded vehicle image, including: obtaining a random offset; based on the random offset and the vehicle image, the foreground image of the occluder and the coordinates of the occluded vehicle position are further segmented to obtain the occluded vehicle image.

[0058] In this embodiment, random offset pairs can be generated as needed, and vehicle images and finely segmented foreground images of occluded objects can be superimposed to obtain a synthesized image of the occluded vehicle. With different offsets, the number of samples can be greatly increased when the generated occluded vehicle images are used for training, making them better suited to this scenario.

[0059] As an optional embodiment, after obtaining the occluded vehicle image by finely segmenting the foreground image of the occluded object and the coordinates of the occluded vehicle position based on the vehicle image, the method further includes: finely segmenting the foreground image of the occluded object and the coordinates of the occluded vehicle position based on the vehicle image to determine the occluded object position label; and using a generator to optimize the color parameters of the occluded vehicle image based on the occluded object position label and the occluded vehicle image to obtain an optimized occluded vehicle image.

[0060] First, it should be noted that the occlusion location label can be determined simultaneously with the vehicle image, by finely segmenting the foreground image of the occluded object and the coordinates of the occluded vehicle location, thus obtaining the occluded vehicle image. This simultaneous determination method accelerates the process of this embodiment. In this embodiment, the occlusion location label and the synthesized image can be input into a generator based on a generative adversarial network-based image harmonization algorithm. The color parameters of the occluded area, such as hue, are adjusted to make the colors more harmonious, achieving the goal of unifying the colors of the occluded object and the vehicle image. For example, when synthesizing daytime occlusion images with nighttime vehicle images, the resulting optimized occluded vehicle image has unified colors, avoiding a strong sense of discontinuity due to the difference between day and night. By harmonizing the colors of the occluded objects in the vehicle occlusion image, the occlusion object and vehicle hues are kept harmonious, resulting in a final vehicle occlusion image that more closely resembles the real image.

[0061] Based on the above embodiments and optional embodiments, an optional implementation method is provided, which is described in detail below.

[0062] An optional embodiment of this invention provides an end-to-end vehicle occlusion image generation method. This method generates high-quality occluded vehicle images for various scenarios without human interaction, based solely on images of the occluded object and the vehicle itself. The generated images can be used in the construction of most vehicle-related datasets to improve model performance. Furthermore, this method is also applicable to image generation in other non-vehicle scenarios.

[0063] This invention provides a method for generating occluded vehicle images based on image synthesis. Figure 2 This is a general framework flowchart provided by an optional embodiment of the present invention. Figure 3 This is a general flowchart of the fine-segmented foreground image generation method provided by an optional embodiment of the present invention, such as... Figure 2 , 3 As shown, the optional embodiments of the present invention will be described below:

[0064] S1, acquire vehicle image and occlusion image.

[0065] S2, extract image features of the occluded object image based on the salient object detection algorithm, and determine the coarse segmentation of the occluded object image based on the image features.

[0066] Specifically, the U2-Ne salient object detection algorithm can be used to extract image features of occluded objects using an encoder and decoder, predict salient values, and obtain coarse segmentation of occluded objects based on the salient values.

[0067] S3. Upsample the coarsely segmented occluded image using the bilinear interpolation method to obtain the first sub-occluded image. Downsample the first sub-occluded image using the mean method to determine the dilated edge region.

[0068] Specifically, the image is upsampled using bilinear interpolation, and dilation and erosion are performed. The size of the dilation and erosion kernel is automatically controlled by detecting the overall proportion of the foreground pixels. Then, the mean method is used to downsample and restore the resolution, thus determining the dilation edge region.

[0069] The formula for determining the size of the expansion corrosion core is defined as follows:

[0070]

[0071] Where I kernel Where W is the dimension of the kernel, max(W,H) is the number of pixels on the longest side of the coarse segmentation result, F is the number of foreground pixels in the coarse segmentation image, X is the number of pixels in the overall image, and the parameter β controls the expansion ratio of dilation and erosion.

[0072] It should be noted that the purpose of obtaining the dilated edge region is to reduce the computation time of subsequent algorithms and to minimize interference from redundant foreground and background information. The calculation formula is as follows:

[0073]

[0074] Where I represents a single pixel, A represents an RGB image, i,j represents the position of the pixel, and I(i,j) is the pixel value of the pixel at row i, column j in I. i,j∈D represents the set of pixels in the dilated region. i,j∈E represents the set of pixels in the eroded region. This operation is performed on all channels for an RGB three-channel image.

[0075] S4, determine the three-part diagram corresponding to the expansion edge region.

[0076] S4.1, Based on the Region Growth Algorithm (SRG), determine the background-removed image corresponding to the dilated edge region;

[0077] Specifically, after obtaining the edge region GB, a region growing algorithm is used to preprocess the dilation boundaries of dilation and erosion as seed points S. d However, this method obtains overly dense seed points, resulting in poor background removal during low-threshold growth. Here, Manhattan distance is used to evaluate the seed point set S. d The formula for sparsity processing is defined as follows:

[0078] d(I 1 ,I 2 )=|I 1 i -I 2 j |+|I 1 i -I 2 j |I 1 ,I 2 ∈S d ;

[0079]

[0080] In the formula, d(I1,I2) represents I 1 with I 2 The Manhattan distance between two pixels, and the set of seed points after sparsening are S. m GB will be based on the seed point set S. m Region growing is performed to remove redundant background information without interaction.

[0081] S4.2, Based on the Sobel edge detection method, determine the mixed pixel image corresponding to the dilated edge region;

[0082] Specifically, the Sobel operator detects image edges by calculating gray levels, and performs convolution operations on each pixel using two convolution operators, one horizontal and one vertical. The formulas for the two convolution operators are as follows:

[0083]

[0084] Where A is the original image, G x With G y These are the gradient images for the horizontal and vertical directions, respectively. The output image is controlled as a 16-bit signed integer grayscale image. Then, the gradients in both directions are superimposed, and the 16-bit image is converted to 8 bits to obtain the overall gradient image G. The gradient superposition formula is shown below:

[0085]

[0086] Next, the image is denoised using the Non-Local Means (NML) algorithm, followed by binarization. The purpose of binarization is to clearly distinguish edge regions from non-edge regions based on the NML denoising results. A binarization threshold of 50 is defined to remove redundant background while retaining more details. The specific formula is as follows:

[0087]

[0088] If the gray value G(i,j) of a pixel is greater than 50, it is set to 255; if it is less than or equal to 50, it is set to 0. The resulting image Z contains the interference of the boundary between the foreground and the transparent area of ​​the GB image.

[0089] S4.3, based on the Otsu's algorithm, determine the pixel segmentation image corresponding to the dilated edge region;

[0090] Using the Otsu's method, we assume a threshold TH divides all pixels in the image into two classes: regions D1 with values ​​less than the threshold and regions D2 with values ​​greater than the threshold. The mean gray values ​​of each region are m1 and m2, and the probabilities of a pixel being divided into two regions are p1 and p2, respectively. The threshold selection formula for OTSU is:

[0091] σ 2 =p1p2(m1-m2) 2 ;

[0092] σ in the above formula 2 The threshold that is maximized is the OTSU threshold. This threshold can then be used to segment the image, yielding the OTSU thresholded segmentation result M. O .

[0093] S4.4, Based on the background-removed image, the blended pixel image, the pixel-segmented image, and the coarsely segmented occluded image, determine the three-part image.

[0094] The above processing of the GB image yielded image M after background removal using the region growing method. S ; and the mixed pixel region M obtained after edge detection B ; and the result M after threshold segmentation using the OTSU algorithm O ; M S M B In M O And a three-part image is generated under the constraint of the coarse segmentation result M, where M S With M O The black area represents the foreground and part of the background. B The black areas are the edge-blended pixel areas. The specific formula is defined as follows:

[0095]

[0096]

[0097] T(i,j)=128 M B (i,j)=0,GBM(i,j)=255;

[0098] GBM is the grayscale label image of GB, with white representing the foreground, black representing the background, and gray representing the boundary region. Through the linear operations on the above results, the classification of foreground, background, and blended pixels can be accurately obtained, thus generating a triangulation image. After generating the triangulation image, the image matting method is selected to generate a fine-grained transparency mask, obtaining the transparent foreground of the occluded object.

[0099] S5 takes the original occluded image and the generated tri-image as input, and uses the image matting method to predict the foreground and background transparency to obtain a detailed foreground of the occluded object.

[0100] S6, determine the coordinates of the occluded position of the vehicle based on the target detection algorithm.

[0101] Specifically, vehicle image attributes are detected using object detection algorithms, and the coordinates of the occluded vehicle location are determined using logic.

[0102] S7. Obtain the random offset. Based on the random offset, the vehicle image is further segmented into the foreground image of the occluder and the coordinates of the occluded vehicle position to obtain the occluded vehicle image.

[0103] Specifically, the image overlay position is obtained based on different occlusion attributes and a random offset β. Then, the occlusion image and the vehicle image are overlaid according to the overlay position to obtain the composite image.

[0104] S8, Figure 4 These are the main steps of the generator optimizing the image provided in the optional embodiments of the present invention, such as... Figure 4 As shown, based on the vehicle image, the foreground image of the occluder and the coordinates of the occluded vehicle position are finely segmented to determine the occlusion position label. Based on the occlusion position label and the occluded vehicle image, a generator is used to optimize the color parameters of the occluded vehicle image to obtain the optimized occluded vehicle image.

[0105] Specifically, an encoder is used to extract color features of the vehicle background image based on the synthesized image, and a generator is used to harmonize the hue and other attributes of the occluded areas in the synthesized image according to the occlusion location labels. This solves the appearance consistency problem and obtains a more realistic image of the occluded vehicle.

[0106] The above optional implementation methods can achieve at least the following beneficial effects:

[0107] (1) A salient object detection method is used as a coarse segmentation method. By detecting salient regions in an image, the salient object detection method can extract the foreground in the image without being limited by semantics. Good results can be obtained without retraining the segmentation model for different semantics.

[0108] (2) The three-part image generation algorithm is based on the coarse segmentation results of salient target detection and automatically generates three-part images using traditional image processing methods. It does not require manual production and achieves the end-to-end purpose of the whole process.

[0109] (3) The matting method can predict the transparency of the foreground and background of each pixel. The generated transparent foreground does not contain the background interference of the original image, and can achieve a matting effect as fine as a hair. In subsequent image compositing, a more realistic effect can be achieved.

[0110] (4) Image harmonization algorithms can optimize the tonal differences between occluders and the background in an image, making the entire image closer to a real image and improving generalization ability when trained in the application scenario of the model.

[0111] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0112] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0113] Example 2

[0114] According to embodiments of the present invention, an apparatus for implementing the above-described method for generating occluded vehicle images is also provided. Figure 5 This is a structural block diagram of an occluded vehicle image generation device according to an embodiment of the present invention, such as... Figure 5 As shown, the device includes: a first acquisition module 502, a first determination module 504, a second determination module 506, a third determination module 508, a generation module 510, and a second acquisition module 512. The device will be described in detail below.

[0115] The first acquisition module 502 is used to acquire a vehicle image and an occlusion image; the first determination module 504, connected to the first acquisition module 502, is used to determine a coarsely segmented occlusion image based on the occlusion image; the second determination module 506, connected to the first determination module 504, is used to determine the dilated edge region corresponding to the coarsely segmented occlusion image; the third determination module 508, connected to the second determination module 506, is used to determine a three-part image corresponding to the dilated edge region, wherein the three-part image includes a foreground region, a background region, and a mixed pixel region; the generation module 510, connected to the third determination module 508, is used to generate a finely segmented occlusion foreground image based on the occlusion image and the three-part image; the second acquisition module 512, connected to the generation module 510, is used to obtain an occluded vehicle image by finely segmenting the occlusion foreground image and the coordinates of the occluded vehicle position based on the vehicle image, wherein the coordinates of the occluded vehicle position are coordinates in a target coordinate system, which is established based on the vehicle image.

[0116] It should be noted that the first acquisition module 502, the first determination module 504, the second determination module 506, the third determination module 508, the generation module 510, and the second acquisition module 512 correspond to steps S102 to S112 in the method for generating images of occluded vehicles. The instances and application scenarios implemented by the multiple modules and the corresponding steps are the same, but are not limited to the content disclosed in the above embodiment 1.

[0117] Example 3

[0118] According to another aspect of the present invention, an electronic device is also provided, comprising: a processor; and a memory for storing processor-executable instructions, wherein the processor is configured to execute instructions to implement the occluded vehicle image generation method of any of the above embodiments.

[0119] Example 4

[0120] According to another aspect of the present invention, a computer-readable storage medium is also provided, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the occluded vehicle image generation method described above.

[0121] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0122] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0123] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0124] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0125] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0126] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0127] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for generating an image of an occluded vehicle, characterized in that, include: Acquire vehicle images and images of occluded objects; Based on the image of the occluded object, a coarsely segmented image of the occluded object is determined; Determining the dilated edge region corresponding to the coarsely segmented occlusion image includes: upsampling the coarsely segmented occlusion image using bilinear interpolation, controlling the size of the dilation and erosion kernel by detecting the overall proportion of foreground pixels to obtain a first sub-occlusion image; downsampling the first sub-occlusion image using the mean method to restore resolution and determine the dilated edge region; wherein, the dilation and erosion kernel is calculated as follows: ， in, The size of the expansion corrosion core, The number of pixels on the longest side of the coarsely segmented occluded image. The number of foreground pixels in the coarsely segmented occluded image. The parameter is the number of pixels in the overall image of the coarsely segmented occluded object image. To control the expansion ratio of expansion corrosion; Determine a tripartite image corresponding to the dilated edge region, wherein the tripartite image includes a foreground region, a background region, and a blended pixel region; Based on the occluder image and the three-part image, a finely segmented foreground image of the occluder is generated; Based on the vehicle image, the foreground image of the occluder and the coordinates of the occluded vehicle position are segmented to obtain the occluded vehicle image, wherein the coordinates of the occluded vehicle position are coordinates in the target coordinate system, and the target coordinate system is established based on the vehicle image; The step of determining the tri-image corresponding to the dilated edge region includes: determining a background-removed image corresponding to the dilated edge region based on a region growing algorithm; determining a mixed pixel image corresponding to the dilated edge region based on an edge detection method; determining a pixel segmentation image corresponding to the dilated edge region based on a maximum inter-class variance algorithm; and determining the tri-image based on the background-removed image, the mixed pixel image, the pixel segmentation image, and the coarsely segmented occlusion image.

2. The method according to claim 1, characterized in that, The step of determining the coarsely segmented occlusion image based on the occlusion image includes: Image features of the occluded object image are extracted based on a salient object detection algorithm; Based on the image features, the coarsely segmented occluded image is determined.

3. The method according to claim 1, characterized in that, Before obtaining the image of the occluded vehicle based on the vehicle image and the finely segmented foreground image of the occluded object and the coordinates of the occluded vehicle position, the method further includes: The coordinates of the obscured position of the vehicle are determined based on the target detection algorithm.

4. The method according to claim 1, characterized in that, The step of obtaining the occluded vehicle image based on the vehicle image, the finely segmented foreground image of the occluded object, and the coordinates of the occluded vehicle position includes: Get the random offset; Based on the random offset, the vehicle image, the finely segmented foreground image of the occluder, and the coordinates of the occluded vehicle position, the occluded vehicle image is obtained.

5. The method according to any one of claims 1 to 4, characterized in that, After obtaining the occluded vehicle image based on the vehicle image, the method further includes: (The method involves finely segmenting the foreground image of the occluded object and the coordinates of the occluded vehicle position.) Based on the vehicle image, the finely segmented foreground image of the occluder and the coordinates of the occluded position of the vehicle are used to determine the occluder position label; Based on the location label of the obstruction and the image of the obstructed vehicle, a generator is used to optimize the color parameters of the image of the obstructed vehicle to obtain an optimized image of the obstructed vehicle.

6. A device for generating an image of an obscured vehicle, characterized in that, include: The first acquisition module is used to acquire vehicle images and images of occluders; The first determining module is used to determine a coarsely segmented occlusion image based on the occlusion image; The second determining module is used to determine the dilated edge region corresponding to the coarsely segmented occluded image. It is used to upsample the coarsely segmented occluded image using bilinear interpolation, control the size of the dilation and erosion kernel by detecting the overall proportion of foreground pixels, and obtain a first sub-occluded image. The first sub-occluded image is then downsampled using the mean method to restore resolution and determine the dilated edge region. The calculation method for the dilation and erosion kernel is as follows: ， in, The size of the expansion corrosion core, The number of pixels on the longest side of the coarsely segmented occluded image. The number of foreground pixels in the coarsely segmented occluded image. The parameter is the number of pixels in the overall image of the coarsely segmented occluded object image. To control the expansion ratio of expansion corrosion; The third determining module is used to determine the three-part map corresponding to the expanded edge region, wherein the three-part map includes a foreground region, a background region, and a mixed pixel region; The generation module is used to generate a finely segmented foreground image of the occluded object based on the occluded object image and the three-part image; The second acquisition module is used to acquire an image of an occluded vehicle based on the vehicle image, the finely segmented foreground image of the occluder and the coordinates of the occluded position of the vehicle, wherein the coordinates of the occluded position of the vehicle are coordinates in a target coordinate system, and the target coordinate system is established based on the vehicle image. The generation module is further configured to: determine the background-removed image corresponding to the dilated edge region based on a region growing algorithm; determine the mixed pixel image corresponding to the dilated edge region based on an edge detection method; determine the pixel segmentation image corresponding to the dilated edge region based on a maximum inter-class variance algorithm; and determine the trisection image based on the background-removed image, the mixed pixel image, the pixel segmentation image, and the coarse segmentation occlusion image.

7. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the occluded vehicle image generation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the occluded vehicle image generation method as described in any one of claims 1 to 5.

Images