Image processing method and device and readable storage medium

Abstract

The invention discloses an image processing method and device and a readable storage medium, and relates to the technical field of image processing. The image processing method comprises the steps that a first image is input into an image detection model, an image shielding area output by the image detection model is obtained, the first image is an image of a first object, and the image shielding area is an area where the first object is shielded in the first image; and complementing the image shielding region in the first image through the generative adversarial network to obtain a second image. The information integrity of the second image is improved, and the image quality of the second image is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device and readable storage medium. Background Art

[0002] With the development of intelligent transportation systems, autonomous driving technology has gradually become the main direction of future transportation. During the autonomous driving process, vehicles need to collect images of traffic lights and accurately identify the traffic lights in the images to make correct driving decisions. However, in actual road environments, due to obstacles such as trees, buildings, billboards, or large vehicles, the traffic lights in the image are partially blocked. Therefore, existing image processing methods have technical problems such as poor image integrity. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, an apparatus, and a readable storage medium, which are used to solve technical problems such as poor image integrity in the prior art.

[0004] According to a first aspect of the embodiments of the present application, there is provided an image processing method, comprising:

[0005] Inputting the first image into the image detection model to obtain an image occlusion region output by the image detection model, wherein the first image is an image of the first object, and the image occlusion region is a region in the first image where the first object is occluded;

[0006] By generating an adversarial network, the image occluded area in the first image is completed to obtain a second image.

[0007] The image processing method in this embodiment, by inputting the first image into an image detection model to obtain the image occlusion area output by the image detection model, and then inputting the first image into a generative adversarial network to obtain a second image output by the generative adversarial network, ensures the image integrity of the second image, and further improves the image quality of the second image.

[0008] According to a second aspect of the embodiments of the present application, there is provided an image processing device, including:

[0009] A processing unit, configured to input a first image into an image detection model to obtain an image occlusion region output by the image detection model, wherein the first image is an image of a first object, and the image occlusion region is a region in the first image where the first object is occluded;

[0010] The processing unit is further used to complete the image occluded area in the first image through a generative adversarial network to obtain a second image.

[0011] The image processing device in this embodiment, by inputting the first image into an image detection model to obtain the image occlusion area output by the image detection model, and then inputting the first image into a generative adversarial network to obtain the second image output by the generative adversarial network, ensures the image integrity of the second image, and further improves the image quality of the second image.

[0012] In a third aspect of the embodiments of the present application, another image processing device is provided, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the image processing method in any of the above embodiments are implemented. Therefore, the image processing device has all the beneficial effects of the image processing method in any of the above embodiments, which will not be described in detail here.

[0013] In a fourth aspect of the embodiments of the present application, a readable storage medium is provided, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the image processing method in any of the above embodiments are implemented. Therefore, the readable storage medium has all the beneficial effects of the image processing method in any of the above embodiments, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0015] Figure 1 A flowchart of an image processing method provided in an embodiment of the present application;

[0016] Figure 2 A functional module block diagram of an image processing device provided in an embodiment of the present application;

[0017] Figure 3 This is a structural block diagram of the image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to better understand the technical solutions provided by the embodiments of this specification, the technical solutions of the embodiments of this specification are described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of this specification and the specific features in the embodiments are detailed descriptions of the technical solutions of the embodiments of this specification, rather than limitations on the technical solutions of this specification. In the absence of conflict, the embodiments of this specification and the technical features in the embodiments can be combined with each other.

[0019] In this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the statement "comprise one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. The term "more than two" includes two or more than two situations.

[0020] In some embodiments, Figure 1 A flowchart of an image processing method provided in an embodiment of the present application, such as Figure 1 As shown, an image processing method is proposed, comprising:

[0021] Step S101: input a first image into an image detection model to obtain an image occlusion area output by the image detection model.

[0022] In this embodiment, a first image of a first object is acquired, wherein the first object is an object to be processed and the first image is an image containing the first object.

[0023] Exemplarily, the first object may be a traffic light, and the first image may be a two-dimensional image including the traffic light.

[0024] An image detection model corresponding to the first object is created, wherein the image detection model is a depth model for processing the first image.

[0025] Exemplarily, the image detection model may be a convolutional neural network model.

[0026] The first image is input into the image detection model to obtain an image occlusion region output by the image detection model, wherein the image occlusion region is a region in the first image where the first object is occluded.

[0027] Exemplarily, the image occlusion area is an image area in which the first object in the first image is occluded and not displayed.

[0028] Exemplarily, the first object may be a traffic light, and the image blocked area is an area in the first image where the traffic light is blocked by obstacles such as trees, buildings, billboards or large vehicles.

[0029] Step S102: completing the image occluded area in the first image by generating an adversarial network to obtain a second image.

[0030] A generative adversarial network corresponding to the first image is created, wherein the generative adversarial network is used to complete the first image.

[0031] The first image is input into a generative adversarial network to obtain a second image output by the generative adversarial network, wherein the second image is an image after the first image is completed.

[0032] It should be noted that the generative adversarial network can accurately complete the first image to obtain the second image, thereby improving the information completeness of the second image, and further can accurately identify the first object in the second image.

[0033] The image processing method in this embodiment, by inputting the first image into an image detection model to obtain the image occlusion area output by the image detection model, and then inputting the first image into a generative adversarial network to obtain a second image output by the generative adversarial network, ensures the image integrity of the second image, and further improves the image quality of the second image.

[0034] In some embodiments, inputting the first image into an image detection model to obtain an image occlusion area output by the image detection model includes:

[0035] Step S201: input a first image into a feature extraction layer to obtain image features in the first image.

[0036] In this embodiment, the image detection model includes a feature extraction layer, a data output layer and a loss function, wherein the feature extraction layer is used to extract features in the first image, the data output layer is used to output the features in the first image, and the loss function is used to optimize the output data of the data output layer.

[0037] The first image is input into the feature extraction layer to obtain image features output by the feature extraction layer, wherein the image features are features in the first image.

[0038] Exemplarily, the image feature can represent a feature of the first object in the first image.

[0039] Step S202, inputting the image features into the data output layer, and obtaining the occlusion probability, the first region coordinates and the region size output by the data output layer.

[0040] The image features are input into the data output layer to obtain the occlusion probability, first region coordinates and region size output by the data output layer, wherein the occlusion probability represents the probability that the first object in the first image is occluded, the first region coordinates are the coordinates of the occluded region of the first object in the first image, and the region size is the size information of the occluded region of the first object in the first image.

[0041] Exemplarily, the occlusion probability may be 90%, indicating that the probability that the first object in the first image is occluded is 90%.

[0042] Step S203: updating the first region coordinates by using a loss function to obtain second region coordinates.

[0043] The first region coordinates are updated with data through the loss function to obtain second region coordinates, wherein the second region coordinates are coordinate data optimized for the first region coordinates.

[0044] Exemplarily, the loss function is used to optimize the predicted image occlusion area, and the loss function may be:

[0045]

[0046] Among them, P i is the real bounding box coordinate, i.e. the first region coordinate, is the predicted bounding box coordinate, that is, the first region coordinate, N is the total number, i is the number, and L is the loss function.

[0047] Exemplarily, an optimization algorithm may be used to optimize the loss function. Specifically, a stochastic gradient descent optimization algorithm is used, and the update formula is:

[0048]

[0049] Where η is the learning rate, is the gradient of the loss function, L is the loss function, W t is the loss function, W t+1 is the gradient at t+1, W t is the gradient at t.

[0050] Step S204: determining the image occlusion region according to the occlusion probability, the second region coordinates and the region size.

[0051] An image occlusion region in the first image is determined based on the occlusion probability, the second region coordinates, and the region size.

[0052] The image processing method in this embodiment determines the occlusion probability, second area coordinates and area size based on the first image, and then determines the image occlusion area according to the occlusion probability, second area coordinates and area size, thereby ensuring the data accuracy of the image occlusion area and further ensuring the accuracy of the second image.

[0053] In some embodiments, inputting the first image into a feature extraction layer to obtain image features in the first image includes:

[0054] Step S301: input a first image into an input layer to extract a third image from the first image.

[0055] In this embodiment, the feature extraction layer includes an input layer, a convolution layer and a pooling layer, wherein the input layer is used to receive an input first image, the convolution layer is used to extract data in the first image, and the pooling layer is used to simplify the size of the image.

[0056] The first image is input to the input layer to extract a third image from the first image, wherein the third image is a cropped image of the area where the first object is located.

[0057] Exemplarily, the first object may be a traffic light, and the third image may be a cropped image of the candidate area of ​​the traffic light.

[0058] Step S302: input the third image into the convolution layer to obtain feature data output by the convolution layer.

[0059] The third image is input into the convolution layer to obtain feature data output by the convolution layer.

[0060] Exemplarily, the convolutional layers may include a 1st convolution layer, a 2nd convolution layer, a 3rd convolution layer, and a 4th convolution layer.

[0061] Step S303, input the feature data into the pooling layer to obtain the image features output by the pooling layer.

[0062] The feature data is input into the pooling layer to obtain the image features output by the pooling layer.

[0063] Exemplarily, the pooling layers may include a 1st pooling layer, a 2nd pooling layer, a 3rd pooling layer, and a 4th pooling layer.

[0064] Exemplarily, the network architecture of the feature extraction layer is designed as follows:

[0065] (1) Input layer: 128×128×3 image. Cropped image of the candidate traffic light region.

[0066] (2) Convolution layer 1: 32 3×3 kernels, stride 1, padding ‘same’, ReLU activation, output 128×128×32. Extract low-level features.

[0067] (3) Pooling layer 1: 2×2 max pooling, stride 2, output 64×64×32. Reduce the size of the feature map and enhance the invariance of the features.

[0068] (4) The second convolution layer: 64 3×3 convolution kernels, stride 1, padding 'same', ReLU activation, output 64×64×64. Extract higher-level features.

[0069] (5) Second pooling layer: 2×2 max pooling, stride 2, output 32×32×64. This further reduces the size of the feature map.

[0070] (6) Convolution layer 3: 128 3×3 kernels, stride 1, padding ‘same’, ReLU activation, output 32×32×128. Extracts higher-level features for occlusion detection.

[0071] (7) Pooling layer 3: 2×2 max pooling, stride 2, output 16×16×128. Further reduce the size and aggregate features.

[0072] (8) Convolutional layer 4: 256 3×3 kernels, stride 1, padding ‘same’, ReLU activation, output 16×16×256. Extracts the highest level features and detects complex occlusions.

[0073] (9) Pooling layer 4: 2×2 max pooling, stride 2, output 8×8×256. Reduce the size to fit the input of the fully connected layer.

[0074] The image processing method in this embodiment obtains the image features of the first image through the input layer, the convolution layer and the pooling layer, thereby ensuring the data accuracy of the image features and the data accuracy of the image occluded area.

[0075] In some embodiments, the image features are input into the data output layer, and the occlusion probability, the first region coordinates, and the region size output by the data output layer are obtained, including:

[0076] Step S401: input the image feature into the first output branch to obtain the occlusion probability output by the first output branch.

[0077] In this embodiment, the data output layer includes a first output branch and a second output branch, wherein the first output branch and the second output branch are two output branches independent of each other.

[0078] The image feature is input into the first output branch to obtain the occlusion probability output by the first output branch.

[0079] Step S402: input the image features to the second output branch to obtain the first region coordinates and region size output by the second output branch.

[0080] The image features are input into the second output branch to obtain the first region coordinates and region size output by the second output branch.

[0081] Exemplarily, the first output branch may be branch 1, which is used for occlusion detection and includes:

[0082] Fully connected layer: A fully connected layer for occlusion detection, which outputs two neurons to predict whether there is occlusion or not.

[0083] Output layer: Softmax (normalized exponential function) activation function, output category probability (occluded / unoccluded).

[0084] The first output branch may be branch 2, which is used for bounding box prediction and includes:

[0085] Fully connected layer: A fully connected layer for bounding box regression that outputs four neurons corresponding to the x, y coordinates, width (w), and height (h) of the bounding box.

[0086] Output layer: Linear activation function, directly outputs the coordinates and size of the bounding box.

[0087] The image processing method in this embodiment obtains the occlusion probability, the first area coordinates and the area size through the first output branch and the second output branch, respectively, thereby ensuring the data accuracy of the occlusion probability, the first area coordinates and the area size, and ensuring the data accuracy of the image occlusion area.

[0088] In some embodiments, completing an image occluded area in a first image by generating an adversarial network to obtain a second image includes:

[0089] Step S501, input the first image to the generator to obtain a fourth image output by the generator.

[0090] In this embodiment, the generative adversarial network includes a generator and a discriminator. The generator is responsible for generating a completed image, and the discriminator is used to judge the authenticity of the generated image.

[0091] The first image is input into the generator to obtain a fourth image output by the generator, wherein the fourth image is an image after the first image is completed.

[0092] Exemplarily, the generator includes:

[0093] (1) Input, size is 128×128×3.

[0094] (2) Encoder, consisting of 4 layers of convolution, each followed by LeakyReLU (an activation function) and InstanceNormalization (a normalization function).

[0095] (3) Decoder, which consists of 4 layers of deconvolution, each followed by a ReLU (an activation function) and a skip connection

[0096] (4) Output, size is 128×128×3.

[0097] Step S502: input the fourth image into the discriminator to obtain the true image probability of the fourth image.

[0098] The fourth image is input into the discriminator to obtain the image true probability of the fourth image, wherein the image true probability represents the authenticity of the fourth image.

[0099] Exemplarily, the discriminator includes:

[0100] (1) Input, size is 128×128×3.

[0101] (2) 5 layers of convolution, each followed by LeakyReLU (an activation function) and Dropout (a regularization function).

[0102] (3) Finally, the result is output through the fully connected layer.

[0103] Step S503: when the true probability of the image is greater than the probability threshold, the fourth image is used as the second image.

[0104] When the true probability of the image is greater than the probability threshold, it is indicated that the fourth image is a true image, and the fourth image is used as the second image, wherein the probability threshold is a preset probability threshold.

[0105] Exemplarily, the probability threshold may be 80%.

[0106] The image processing method in this embodiment obtains the second image through the generator and the discriminator, thereby ensuring the image accuracy of the second image and the data integrity of the second image.

[0107] In some embodiments, the specific steps of acquiring the first image include:

[0108] Step S601: collect an image of a first object to obtain an original image.

[0109] In this embodiment, an image of the first object is acquired to obtain an original image, wherein the original image is the original image of the first object.

[0110] Exemplarily, the original image may be a real-time road image, which is acquired in real time using a high-definition camera installed in front of the vehicle. The camera should have a high dynamic range function to adapt to different lighting conditions (such as day, night, backlight, etc.).

[0111] Step S602: De-noising and enhancing the original image to obtain a fifth image.

[0112] The original image is subjected to denoising and enhancement processing to obtain a fifth image, wherein the fifth image is an image after the original image is enhanced.

[0113] For example, in order to improve the clarity of the original image, the median filter is used to pre-process the original image to reduce the impact of noise on subsequent recognition. At the same time, the contrast-limited histogram equalization (CLAHE) algorithm is used to enhance the image contrast, especially in low light conditions, to improve the visibility of traffic lights.

[0114] g(x,y)=med{f(xm,yn),(m,n∈w)};

[0115] Among them, f(x,y) represents the original image, g(x,y) represents the processed image, w is an odd-sized window in the image, m,n represent the offset relative to the current pixel in the window w, and the med function is a function used for signal processing, mainly used for deconvolution and denoising.

[0116] Step S603: perform target detection processing on the fifth image to obtain the first image.

[0117] Perform target detection processing on the fifth image to obtain the first image.

[0118] Exemplarily, the candidate region detection is performed on the fifth image to obtain the first image. Specifically, the HSV (Hue Saturation Value) color space segmentation algorithm and edge detection algorithm are used to preliminarily locate the candidate region of the traffic light in the image. In order to speed up the processing speed, the sliding window technology is used to scan the image block by block to mark the area that may contain the traffic light.

[0119] The image processing method in this embodiment ensures the image accuracy of the first image and the image real-time performance of the first image.

[0120] In some embodiments, performing target detection processing on the fifth image to obtain the first image includes:

[0121] Step S701, performing color conversion and color segmentation processing on the fifth image to obtain a sixth image.

[0122] In this embodiment, the fifth image is subjected to color conversion and color segmentation processing to obtain a sixth image, wherein the sixth image is an image obtained by color conversion and color segmentation of the fifth image.

[0123] Exemplarily, the fifth image is converted from an RGB (red, green, and blue) color space to an HSV color space. The HSV color space is closer to the way the human eye perceives colors, and therefore has an advantage in color segmentation.

[0124] (1) RGB color space representation.

[0125] Assume that a pixel of the original image is represented as R, G, B in the RGB color space, and its value range is [0, 255].

[0126] (2) Normalized RGB values.

[0127] For the convenience of calculation, first normalize R, G, and B to [0, 1]: R' = R / 255, G' = G / 255, B' = B / 255.

[0128] (3) Calculate V (brightness).

[0129] The brightness V is expressed as: V=max(R', G', B').

[0130] (4) Calculate S (saturation).

[0131] The calculation formula of saturation S is as follows:

[0132]

[0133] Among them, V is brightness, R', G', B' are the normalized color values ​​of red, green, and blue, and S is saturation.

[0134] (5) Calculate H (hue).

[0135] The calculation of hue H is divided into three cases according to the relative sizes of the three RGB channels:

[0136]

[0137] Among them, V is brightness, R', G', B' are the normalized color values ​​of red, green, and blue, and H is hue.

[0138] The calculated H is in the range of [0,360°]. If H<0, then H=H+360°.

[0139] Step S702: perform edge detection processing on the sixth image to determine edge data in the sixth image.

[0140] Exemplarily, after conversion to the HSV color space, the image can be color segmented. The basic idea of ​​color segmentation is to set a threshold range for the H, S, and V components of the HSV space to identify a specific color.

[0141] (1) Set the threshold range.

[0142] Set the corresponding H, S, and V threshold ranges according to the colors to be segmented.

[0143] (2) Color segmentation.

[0144] For each pixel, determine whether its HSV value is within the set threshold range. If the condition is met, the pixel is considered to be the target color, otherwise it is the background.

[0145] Set a binary mask image M(x,y), where a value of 1 indicates that the pixel belongs to the target color, and a value of 0 indicates the background:

[0146]

[0147] Among them, M(x,y) is a binary mask image, S is saturation, V is brightness, and H is hue.

[0148] Exemplarily, the sixth image may be post-processed.

[0149] After obtaining the segmented binary mask image, morphological operations are performed to remove noise and connect the segmented areas.

[0150] The first step is morphological operation.

[0151] Dilation: Expands white areas in the image, fills small black holes, and connects segmented areas.

[0152] Erosion: Shrinks the white areas in the image and removes small noise points.

[0153] The second step is connected area analysis.

[0154] Filter out small areas to get more accurate segmentation results.

[0155] Exemplarily, the edge detection algorithm steps are:

[0156] (1) Gaussian filtering.

[0157] In order to reduce the noise in the image, the image is first subjected to Gaussian filtering. The purpose of Gaussian filtering is to smooth the image so that subsequent edge detection is not interfered by high-frequency noise.

[0158] The formula for the Gaussian filter is as follows:

[0159]

[0160] Among them, x and y are the coordinates of the pixel, and σ is the standard deviation of the Gaussian kernel. Choosing a suitable σ value is an important step in Gaussian filtering. A larger σ value can achieve a stronger smoothing effect.

[0161] Convolve the original image I(x,y) with the Gaussian filter G(x,y) to get the smoothed image I G (x,y):

[0162] I G(x,y)=I(x,y)*G(x,y);

[0163] Among them, * is the convolution symbol.

[0164] (2) Gradient calculation.

[0165] The image gradient is calculated on the smoothed image to determine the direction in which the grayscale changes fastest in the image, that is, the edge direction.

[0166] This is achieved by calculating the gradient of the image in the horizontal direction (x direction) and the vertical direction (y direction):

[0167]

[0168] Use the Sobel operator to calculate G x (x,y) and G y (x,y). The Sobel operators are:

[0169]

[0170] Apply the Sobel operator to the image and get the gradient map G x (x,y) and G y (x,y).

[0171] The gradient magnitude M(x,y) and θ(x,y) are calculated by the following formula:

[0172]

[0173] (3) Non-maximum suppression.

[0174] In the non-maximum suppression step, the goal is to refine the edges and remove non-edge points. By processing the gradient magnitude image, only the points with the local maximum value are retained and non-edge points are removed.

[0175] According to the gradient direction θ(x,y), the gradient magnitude of each pixel is compared with the gradient magnitudes of the two adjacent pixels along the gradient direction.

[0176] If the gradient magnitude of the current pixel is not a local maximum, its gradient magnitude is set to 0, indicating that the point cannot be part of the edge.

[0177] The specific operations are as follows:

[0178] Assume that the current pixel is M(x,y), its gradient direction is θ(x,y), and the gradient direction is quantized into four main directions: 0°, 45°, 90°, and 135°.

[0179] M(x,y) is compared with the gradient magnitude of two neighboring pixels according to the direction.

[0180] If M(x,y) is larger, keep it, otherwise set it to 0.

[0181] Exemplarily, dual threshold detection is performed on .

[0182] By setting two thresholds, strong edge points and weak edge points are screened out. This step can further distinguish between true edges and noise.

[0183] Set two thresholds: high threshold T high and low threshold T low .

[0184] If the gradient amplitude of a pixel is greater than T high , it is marked as a strong edge.

[0185] If the gradient amplitude of a pixel point is T low and T high If the edges are between , they are marked as weak edges.

[0186] If the gradient amplitude of a pixel is less than T low , then it is suppressed as a non-edge point.

[0187] Step S703: Perform edge connection processing on the sixth image based on the edge data to obtain the first image.

[0188] Based on the edge data, edge connection processing is performed on the sixth image to obtain the first image.

[0189] Exemplarily, a final edge map, ie, the first image, is formed by connecting weak edges to strong edges.

[0190] Starting from all pixels marked as strong edges, check whether there are weak edge pixels in their 8-neighborhood. If so, keep these weak edge pixels as true edges.

[0191] If a weak edge pixel is not connected to any strong edge pixel, it is suppressed.

[0192] The image processing method in this embodiment ensures the image accuracy of the first image and the image real-time performance of the first image.

[0193] In some embodiments, Figure 2 The functional module block diagram of the image processing device provided in the embodiment of the present application is as follows: Figure 2 As shown, an embodiment of the present application provides an image processing device 700, including:

[0194] A processing unit 702 is used to input a first image into an image detection model to obtain an image occlusion region output by the image detection model, wherein the first image is an image of a first object, and the image occlusion region is a region in the first image where the first object is occluded;

[0195] The processing unit 702 is further configured to complete the image occluded area in the first image by generating an adversarial network to obtain a second image.

[0196] In this embodiment, a first image of a first object is acquired, wherein the first object is an object to be processed and the first image is an image containing the first object.

[0197] Exemplarily, the first object may be a traffic light, and the first image may be a two-dimensional image including the traffic light.

[0198] An image detection model corresponding to the first object is created, wherein the image detection model is a depth model for processing the first image.

[0199] Exemplarily, the image detection model may be a convolutional neural network model.

[0200] The first image is input into the image detection model to obtain an image occlusion region output by the image detection model, wherein the image occlusion region is a region in the first image where the first object is occluded.

[0201] Exemplarily, the image occlusion area is an image area in which the first object in the first image is occluded and not displayed.

[0202] Exemplarily, the first object may be a traffic light, and the image blocked area is an area in the first image where the traffic light is blocked by obstacles such as trees, buildings, billboards or large vehicles.

[0203] A generative adversarial network corresponding to the first image is created, wherein the generative adversarial network is used to complete the first image.

[0204] The first image is input into a generative adversarial network to obtain a second image output by the generative adversarial network, wherein the second image is an image after the first image is completed.

[0205] It should be noted that the generative adversarial network can accurately complete the first image to obtain the second image, thereby improving the information completeness of the second image, and further can accurately identify the first object in the second image.

[0206] The image processing device 702 in this embodiment inputs the first image into an image detection model to obtain the image occlusion area output by the image detection model, and then inputs the first image into a generative adversarial network to obtain the second image output by the generative adversarial network, thereby ensuring the image integrity of the second image and improving the image quality of the second image.

[0207] In some embodiments, an image processing device 700 is provided in an embodiment of the present application, further comprising:

[0208] The processing unit 702 is further configured to input the first image into the feature extraction layer to obtain image features in the first image;

[0209] The processing unit 702 is further used to input the image features into the data output layer to obtain the occlusion probability, the first region coordinates and the region size output by the data output layer;

[0210] The processing unit 702 is further used to update the first region coordinates by using a loss function to obtain second region coordinates;

[0211] The processing unit 702 is further configured to determine the image occlusion region according to the occlusion probability, the second region coordinates and the region size.

[0212] In some embodiments, an image processing device 700 is provided in an embodiment of the present application, further comprising:

[0213] The processing unit 702 is further configured to input the first image into an input layer to extract a third image from the first image, where the third image is a cropped image of the area where the first object is located;

[0214] The processing unit 702 is further configured to input the third image into the convolution layer to obtain feature data output by the convolution layer;

[0215] The processing unit 702 is further used to input the feature data into the pooling layer to obtain the image features output by the pooling layer.

[0216] In some embodiments, an image processing device 700 is provided in an embodiment of the present application, further comprising:

[0217] The processing unit 702 is further configured to input the image feature into the first output branch to obtain the occlusion probability output by the first output branch;

[0218] The processing unit 702 is further configured to input the image feature into the second output branch to obtain the first region coordinates and region size output by the second output branch.

[0219] In some embodiments, an image processing device 700 is provided in an embodiment of the present application, further comprising:

[0220] The processing unit 702 is further configured to input the first image into the generator to obtain a fourth image output by the generator;

[0221] The processing unit 702 is further configured to input the fourth image into the discriminator to obtain a true image probability of the fourth image;

[0222] The processing unit 702 is further configured to use the fourth image as the second image when the true probability of the image is greater than the probability threshold.

[0223] In some embodiments, an image processing device 700 is provided in an embodiment of the present application, further comprising:

[0224] The processing unit 702 is further used to collect an image of the first object to obtain an original image;

[0225] The processing unit 702 is further used to perform denoising and enhancement processing on the original image to obtain a fifth image;

[0226] The processing unit 702 is further configured to perform target detection processing on the fifth image to obtain the first image.

[0227] In some embodiments, an image processing device 700 is provided in an embodiment of the present application, further comprising:

[0228] The processing unit 702 is further configured to perform color conversion and color segmentation processing on the fifth image to obtain a sixth image;

[0229] The processing unit 702 is further configured to perform edge detection processing on the sixth image to determine edge data in the sixth image;

[0230] The processing unit 702 is further configured to perform edge connection processing on the sixth image based on the edge data to obtain the first image.

[0231] In some embodiments, Figure 3 A structural block diagram of an image processing device provided in an embodiment of the present application, such as Figure 3 As shown, an image processing device 800 is proposed, and the image processing device 800 includes a processor 802 and a memory 804. The memory 804 stores a computer program, and when the computer program is executed by the processor 802, the steps of the image processing method in any of the above embodiments are implemented. Therefore, the image processing device 800 has all the beneficial effects of the image processing method in any of the above embodiments, which will not be described in detail here.

[0232] In some embodiments, a readable storage medium is provided, on which a program is stored. When the program is executed by a processor, the steps of the image processing method in any of the above embodiments are implemented, thereby having all the beneficial technical effects of the image processing method in any of the above embodiments.

[0233] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0234] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-readable program code.

[0235] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0236] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0237] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0238] An embodiment of the present application further provides a computer program product, which includes computer software instructions. When the computer software instructions are executed on a processing device, the processing device executes the process of the image processing method.

[0239] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integration. Available media can be magnetic media, (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid state drive (solid state disk, SSD)), etc.

[0240] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0241] In the several embodiments provided in the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0242] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0243] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0244] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0245] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0246] Although the preferred embodiments of this specification have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of this specification.

[0247] Obviously, those skilled in the art can make various changes and modifications to this specification without departing from the spirit and scope of this specification. Thus, if these modifications and variations of this specification fall within the scope of the claims of this specification and their equivalents, this specification is also intended to include these modifications and variations.

Claims

1. An image processing method, characterized in that: The method comprises: Inputting a first image into an image detection model to obtain an image occlusion region output by the image detection model, wherein the first image is an image of a first object, and the image occlusion region is a region of the first image where the first object is occluded; The image occluded area in the first image is completed by generating an adversarial network to obtain a second image.

2. The method according to claim 1, characterized in that The image detection model includes a feature extraction layer, a data output layer and a loss function. The step of inputting the first image into the image detection model to obtain the image occlusion area output by the image detection model includes: Inputting the first image to the feature extraction layer to obtain image features in the first image; Inputting the image features into the data output layer, obtaining the occlusion probability, the first region coordinates and the region size output by the data output layer; By using the loss function, updating the first area coordinates to obtain the second area coordinates; The image occlusion area is determined according to the occlusion probability, the second area coordinates and the area size.

3. The method according to claim 2, characterized in that The feature extraction layer includes an input layer, a convolution layer and a pooling layer. The step of inputting the first image to the feature extraction layer to obtain image features in the first image includes: Inputting the first image to the input layer to extract a third image from the first image, wherein the third image is a cropped image of the area where the first object is located; Inputting the third image into the convolution layer to obtain feature data output by the convolution layer; The feature data is input into the pooling layer to obtain the image features output by the pooling layer.

4. The method according to claim 2, characterized in that: The data output layer comprises a first output branch and a second output branch, The step of inputting the image features into the data output layer to obtain the occlusion probability, the first region coordinates and the region size output by the data output layer includes: Inputting the image feature into the first output branch to obtain the occlusion probability output by the first output branch; The image feature is input into the second output branch to obtain the first region coordinates and the region size output by the second output branch.

5. The method according to claim 1, characterized in that: The generative adversarial network includes a generator and a discriminator. The method of completing the image occluded area in the first image by generating an adversarial network to obtain a second image includes: Inputting the first image into the generator to obtain a fourth image output by the generator; Inputting the fourth image into the discriminator to obtain a true image probability of the fourth image; When the true probability of the image is greater than a probability threshold, the fourth image is used as the second image.

6. The method according to any one of claims 1 to 5, characterized in that The specific steps of acquiring the first image include: Capturing an image of the first object to obtain an original image; Performing denoising and enhancement processing on the original image to obtain a fifth image; Perform target detection processing on the fifth image to obtain the first image.

7. The method according to claim 6, characterized in that The performing target detection processing on the fifth image to obtain the first image includes: Performing color conversion and color segmentation processing on the fifth image to obtain a sixth image; Performing edge detection processing on the sixth image to determine edge data in the sixth image; Based on the edge data, edge connection processing is performed on the sixth image to obtain the first image.

8. An image processing device, characterized in that: The device comprises: a processing unit, configured to input a first image into an image detection model to obtain an image occlusion region output by the image detection model, wherein the first image is an image of a first object, and the image occlusion region is a region in the first image where the first object is occluded; The processing unit is further used to complete the image occluded area in the first image through a generative adversarial network to obtain a second image.

9. An image processing device, characterized in that: include: processor; A memory, wherein a program or instruction is stored in the memory, and when the processor executes the program or instruction in the memory, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

Images