Image Processing Method, Apparatus and Electronic Device

Abstract

The present invention provides an image processing method, apparatus and electronic device, including: obtaining a to-be-processed image containing a target object and a target object mask corresponding to the to-be-processed image; inputting the to-be-processed image and the target object mask into a distortion model to obtain a deformation grid map corresponding to the to-be-processed image; and performing deformation correction on the to-be-processed image based on the deformation grid map to obtain a corrected image of the to-be-processed image. The present invention uses a distortion model to process the to-be-processed image and the target object mask, directly obtains a deformation grid map corresponding to the to-be-processed image, and then performs deformation correction on the to-be-processed image based on the deformation grid map. This process does not require parameter optimization, can greatly improve the operation efficiency and enhance the correction effect; the use of the target object mask is beneficial to strengthening the model's attention to the area where the target object is located; the deformation grid map is used to correct the entire image, making the boundary between the target object and the background in the corrected image more natural.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to an image processing method, apparatus, and electronic device. Background Art

[0002] With the development of shooting devices, wide-angle and ultra-wide-angle lenses are widely used to capture images. When a wide-angle or ultra-wide-angle lens is used to shoot a target object, the imaging points of the target object will shift, resulting in various distorted images. For example, Figure 1 As shown, when shooting a target object p through a wide-angle lens, the imaging point P will shift differently on the imaging plane, obtaining imaging points P1 and P2, and correspondingly obtaining two types of distorted images, namely, barrel distortion images and pillow distortion images. The above distorted images will affect the application value of the images. Therefore, it is necessary to correct the distorted images through an image undistoration method to obtain undistorted images that conform to human visual characteristics, which plays an increasingly important role in image and video processing. Usually, the distortion degrees of the foreground (i.e., the shooting target) and the background of the image are different. If an undistorted image that conforms to human visual characteristics is desired, different distortion correction methods need to be applied to the two.

[0003] Traditional distortion correction methods are as follows: obtaining the distortion parameters of the camera lens through calibration, and then deforming the distorted image back into a normal image. During the deforming process, stretching will inevitably occur at the four corners of the distorted image. When the target object is located on the image boundary, it will cause the target object to deform. The current mainstream algorithms are to first obtain the position of the target object, and then adopt different distortion correction methods for the background and the target object during the distortion correction process, or first adopt the same distortion correction method for the entire image, and then separately adjust the part of the target object. In this way, it can be ensured that both the background and the target object are corrected to conform to visual characteristics. Even so, the above image processing methods still have several problems: one is that the optimization process is slow. In order to solve the optimization algorithm, it usually takes a very large number of iterations to converge to a correct solution, and there is also a risk of converging to a local minimum; the second is that the parameter adaptability is poor. The correction parameters suitable for image A are not necessarily suitable for image B, and it is difficult to find correction parameters that are suitable for all images or most images; the third is that the optimization effect is poor. Unnatural effects such as line distortion often appear at the boundaries between the target object and the background.

[0004] In summary, the existing image processing methods have technical problems of poor effect and low efficiency. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide an image processing method, apparatus, and electronic device to alleviate the technical problems of poor effect and low efficiency of the existing image processing methods.

[0006] In a first aspect, an embodiment of the present invention provides an image processing method, including: obtaining a to-be-processed image including a target object and a target object mask corresponding to the to-be-processed image; inputting the to-be-processed image and the target object mask into a distortion model to obtain a deformed mesh image corresponding to the to-be-processed image, wherein the pixel value of each pixel point of the deformed mesh image represents the offset of the corresponding pixel point in the to-be-processed image; and performing deformation correction on the to-be-processed image based on the deformed mesh image to obtain a corrected image of the to-be-processed image.

[0007] Further, the method further includes: scaling the to-be-processed image to a first preset scale to obtain a to-be-processed image of the first preset scale.

[0008] Further, obtaining the target object mask corresponding to the to-be-processed image includes: inputting the to-be-processed image of the first preset scale into a segmentation model to obtain the target object mask.

[0009] Further, the method further includes: scaling the to-be-processed image and the target object mask to a second preset scale respectively to obtain a to-be-processed image of the second preset scale and a target object mask of the second preset scale.

[0010] Further, inputting the to-be-processed image and the target object mask into the distortion model includes: inputting the to-be-processed image of the second preset scale and the target object mask of the second preset scale into the distortion model to obtain a deformed mesh image corresponding to the to-be-processed image.

[0011] Further, performing deformation correction on the to-be-processed image based on the deformed mesh image includes: scaling the deformed mesh image to a target scale to obtain a deformed mesh image of the target scale, wherein the target scale is equal to the scale of the to-be-processed image; and offsetting the corresponding pixel points in the to-be-processed image according to the offsets represented by the pixel values of each pixel point in the deformed mesh image of the target scale to obtain a corrected image of the to-be-processed image.

[0012] Further, the method further includes: obtaining a training sample set, wherein the training sample set includes: a training object image, a training object mask corresponding to the training object image, and a deformed mesh image corresponding to the training object image; and training an original distortion model through the training sample set to obtain the distortion model.

[0013] In a second aspect, an embodiment of the present invention further provides an image processing apparatus, including: an acquisition unit configured to acquire a to-be-processed image including a target object and a target object mask corresponding to the to-be-processed image; a processing unit configured to input the to-be-processed image and the target object mask into a distortion model to obtain a deformed grid map corresponding to the to-be-processed image, where the pixel value of each pixel point of the deformed grid map represents the offset of the corresponding pixel point in the to-be-processed image; and a deformation correction unit configured to perform deformation correction on the to-be-processed image based on the deformed grid map to obtain a corrected image of the to-be-processed image.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the method according to any one of the above first aspects are implemented.

[0015] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium having non-volatile program code executable by a processor, where the program code causes the processor to execute the steps of the method according to any one of the above first aspects.

[0016] In the embodiment of the present invention, first, a to-be-processed image including a target object and a target object mask corresponding to the to-be-processed image are acquired. Then, the to-be-processed image and the target object mask are input into a distortion model to obtain a deformed grid map corresponding to the to-be-processed image, where the pixel value of each pixel point of the deformed grid map represents the offset of the corresponding pixel point in the to-be-processed image. Deformation correction is performed on the to-be-processed image based on the deformed grid map to obtain a corrected image of the to-be-processed image. Through the above description, it can be seen that the present invention uses a distortion model to process the to-be-processed image and the target object mask, directly obtains a deformed grid map corresponding to the to-be-processed image, and then performs deformation correction on the to-be-processed image based on the deformed grid map. This process does not require parameter optimization, can greatly improve the operation efficiency and enhance the correction effect. The target object mask is used in the process of training and using the distortion model, which is beneficial to strengthening the model's attention to the area where the target object is located. The deformed grid map output by the distortion model is used to correct the entire image, and no longer distinguishes between the target object and the background, making the boundary between the target object and the background in the corrected image more natural. Description of the Drawings

[0017] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 Schematic diagram of generating a distorted image provided by an embodiment of the present invention;

[0019] Figure 2 Schematic diagram of an electronic device provided by an embodiment of the present invention;

[0020] Figure 3 Flowchart of an image processing method provided by an embodiment of the present invention;

[0021] Figure 4 Schematic diagram of an image to be processed provided by an embodiment of the present invention;

[0022] Figure 5 corresponding to provided by an embodiment of the present invention Figure 4 target object mask;

[0023] Figure 6 corresponding to provided by an embodiment of the present invention Figure 4 and Figure 5 deformation grid diagram corresponding to the head mask of the portrait in;

[0024] Figure 7 Schematic diagram of the corrected image provided by an embodiment of the present invention Figure 4 ;

[0025] Figure 8 Process schematic diagram of the image processing method provided by an embodiment of the present invention;

[0026] FIG. 9(a) is a schematic diagram of an image to be processed provided by an embodiment of the present invention;

[0027] FIG. 9(b) is a schematic diagram of the corrected image of FIG. 9(a) provided by an embodiment of the present invention;

[0028] Figure 10 Schematic diagram of an image processing apparatus provided by an embodiment of the present invention. Detailed implementation manners

[0029] Next, the technical solutions of the present invention will be described clearly and completely in conjunction with the embodiments. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0030] Embodiment 1:

[0031] First, with reference to Figure 2 to describe the electronic device 100 for implementing the embodiments of the present invention, which can be used to run the image processing methods of the embodiments of the present invention.

[0032] As Figure 2 shown, the electronic device 100 includes one or more processors 102, one or more memories 104, an input device 106, an output device 108, and a camera 110. These components are interconnected via a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 2 the components and structure of the electronic device 100 shown are exemplary and not restrictive. According to requirements, the electronic device may also have other components and structures.

[0033] The processor 102 may be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic array (PLA), and an application-specific integrated circuit (ASIC). The processor 102 may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.

[0034] The memory 104 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage media, and the processor 102 may run the program instructions to implement the client functions (implemented by the processor) in the embodiments of the present invention described below and / or other desired functions. Various application programs and various data may also be stored in the computer-readable storage media, such as various data used and / or generated by the application programs, etc.

[0035] The input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, and a touch screen, etc.

[0036] The output device 108 can output various information (such as images or sounds) to the outside (e.g., to a user), and can include one or more of a display, a speaker, etc.

[0037] The camera 110 is used to collect an image to be processed. After the image to be processed collected by the camera is processed by the image processing method, a corrected image is obtained. For example, the camera can capture an image desired by the user (such as a photo, a video, etc.), and then, after the image is processed by the image processing method, a corrected image is obtained. The camera can also store the captured image in the memory 104 for use by other components.

[0038] Exemplarily, an electronic device for implementing the image processing method according to an embodiment of the present invention can be implemented as a smart mobile terminal such as a smart phone, a tablet computer, etc.

[0039] Embodiment 2:

[0040] According to an embodiment of the present invention, an image processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0041] Figure 3 is a flowchart of an image processing method according to an embodiment of the present invention, as Figure 3 shown, the method includes the following steps:

[0042] Step S302, obtain an image to be processed including a target object and a target object mask corresponding to the image to be processed;

[0043] In an embodiment of the present invention, the above-mentioned target object can be a person, an animal, or any other physical object, and the embodiment of the present invention does not specifically limit the above-mentioned target object.

[0044] The above-mentioned image to be processed can be an image frame including a target object in a video stream obtained in real time, or an image including a target object obtained by taking a photo before. That is, the method can process the image frame including a target object in the video stream in real time, or perform post-processing on the captured image including a target object. The embodiment of the present invention does not limit the specific form of the above-mentioned image to be processed.

[0045] In an embodiment of the present invention, the target object mask corresponding to the image to be processed is used to represent the positions of the pixel points of the target object that need to be concerned by the distortion model. The above-mentioned target object mask corresponding to the image to be processed can be the entire target object or a part of the target object. For example, the target object mask can be the outer contour of the target object or a partial area of the target object, such as the outer contour of the head. When only a part of the target object needs to be concerned by the distortion model, the target object mask only needs the positions of the pixel points corresponding to a part of the target object; when the distortion correction needs to keep the entire target object accurate, the target object mask needs to include the positions of all pixel points of the target object. In the target object mask, the positions corresponding to the target object can be 1, and the other positions can be 0.

[0046] When the target object is a person, if the image to be processed is as Figure 4 shown, Figure 5 shows two target object masks corresponding to Figure 4 , one is the entire portrait mask and the other is the head mask of the portrait.

[0047] Step S304: Input the image to be processed and the target object mask into the distortion model to obtain a deformed grid map corresponding to the image to be processed, where the pixel value of each pixel point of the deformed grid map represents the offset of the corresponding pixel point in the image to be processed;

[0048] The above-mentioned distortion model is a pre-trained distortion model, and the above-mentioned deformed grid map can be a two-channel deformation matrix, where the deformation matrix of each channel is a two-dimensional matrix, including two directions of x and y. The pixel value of each pixel point in the deformation matrix represents the offset of the corresponding pixel point in the image to be processed. For example, if the pixel value corresponding to the pixel point (1, 1) in the two channels of the two-channel deformation matrix is (2, 2), then it can be determined that the pixel point (1, 1) in the image to be processed should be offset 2 pixel points to the right and 2 pixel points upward to obtain the corrected image. The pixel value of each pixel point in the deformation matrix can also represent the position of the corresponding pixel point in the corrected image in the image to be processed. For example, if the pixel value corresponding to the pixel point (1, 1) in the two channels of the two-channel deformation matrix is (2, 2), then it can be determined that the position of the pixel point (1, 1) in the image to be processed in the corrected image is (2, 2).

[0049] In an example, inputting the image to be processed and the target object mask into the distortion model means inputting the R layer, G layer, B layer of the RGB image to be processed and the target object mask as four layers of images into the distortion model. Figure 6 shows the visualization effect of the deformed grid map corresponding to the head mask of the portrait in Figure 4 and Figure 5 .

[0050] Step S306: Based on the deformed grid map, perform deformation correction on the image to be processed to obtain the corrected image of the image to be processed.

[0051] Specifically, the position of a certain pixel point A in the image to be processed in the corrected image can be calculated based on the deformed grid map, and the pixel value of the pixel point at the corresponding position in the corrected image is modified to the pixel value of pixel point A in the image to be processed, thereby realizing the offset of the pixel points in the image to be processed.

[0052] It can be understood that if, according to the deformation matrix, multiple pixel points in the image to be processed correspond to the same position B in the corrected image, the pixel values of the multiple pixel points in the image to be processed can be fused to obtain the pixel value at position B in the corrected image. If, according to the deformation matrix, position C in the corrected image does not correspond to any pixel point in the image to be processed, the pixel value at position C can be obtained by interpolation based on the pixel values of the surrounding positions of position C in the corrected image.

[0053] Figure 7 is shown based on Figure 6 the deformed grid map in Figure 4 the corrected image obtained after performing deformation correction on the image to be processed in

[0054] In the embodiment of the present invention, first, an image to be processed including a target object and a target object mask corresponding to the image to be processed are obtained. Then, the image to be processed and the target object mask are input into the distortion model to obtain a deformed grid map corresponding to the image to be processed. Among them, the pixel value of each pixel point in the deformed grid map represents the offset amount of the corresponding pixel point in the image to be processed; based on the deformed grid map, perform deformation correction on the image to be processed to obtain the corrected image of the image to be processed. Through the above description, it can be seen that the present invention uses a distortion model to process the image to be processed and the target object mask, directly obtains a deformed grid map corresponding to the image to be processed, and then performs deformation correction on the image to be processed based on the deformed grid map. This process does not require parameter optimization, can greatly improve the operation efficiency and enhance the correction effect; the target object mask is used in the process of training and using the distortion model, which is beneficial to strengthening the model's attention to the area where the target object is located; the deformed grid map output by the distortion model is used to correct the entire image, and no longer distinguish between the target object and the background, making the boundary between the target object and the background in the corrected image smoother.

[0055] The above content briefly introduces the image processing method of the present invention. The following will describe the specific content involved in detail.

[0056] In an alternative embodiment of the present invention, in step S302, the step of obtaining the target object mask corresponding to the image to be processed includes: inputting the image to be processed with a first preset scale into a segmentation model to obtain the target object mask. The first preset scale is smaller than the scale of the image to be processed.

[0057] The image to be processed with the first preset scale is obtained by scaling the image to be processed to the first preset scale after obtaining the image to be processed containing the target object and before inputting the image to be processed into the segmentation model, thereby obtaining the image to be processed with the first preset scale, and then inputting the image to be processed with the first preset scale into the segmentation model.

[0058] Generally, the original resolution of the image to be processed is relatively large. After scaling the image to be processed to the first preset scale (which can be 640*480, and the present invention embodiment does not specifically limit the above first preset scale), and then inputting it into the segmentation model, the computational amount of the model can be greatly reduced, the computational efficiency can be improved, and thus the entire image processing method can be run in real time on a terminal such as a mobile phone.

[0059] The above segmentation model can be any image segmentation model or instance segmentation model. By inputting the image to be processed with the first preset scale, extracting features of different scales of the image through a feature network with multiple layers of convolution and downsampling, and then fusing the features of different scales through a decoding network that continuously upsamples, the target object mask is output. In the present invention, a real-time human portrait segmentation model is adopted and the image to be processed is scaled to the first preset scale and input into the human portrait segmentation model, and image segmentation can be completed within 10 ms on the mobile phone side.

[0060] In an alternative embodiment of the present invention, in step S304, the step of inputting the image to be processed and the target object mask into the distortion model includes: inputting the image to be processed with a second preset scale and the target object mask with the second preset scale into the distortion model to obtain a deformation grid map corresponding to the image to be processed.

[0061] The image to be processed with the second preset scale and the target object mask with the second preset scale are obtained by scaling the image to be processed and the target object mask to the second preset scale respectively after obtaining the image to be processed and the target object mask and before inputting the image to be processed and the target object mask into the distortion model, thereby obtaining the image to be processed with the second preset scale and the target object mask with the second preset scale, and then inputting the image to be processed with the second preset scale and the target object mask with the second preset scale into the distortion model.

[0062] The original resolution of the image to be processed is relatively large, and the resolution of the target object mask is at the first preset scale. After scaling the image to be processed and the target object mask to the second preset scale respectively, and then inputting them into the distortion model, the computational load of the model can be greatly reduced, and the computational efficiency can be improved. Furthermore, the entire image processing method can run in real time on a terminal such as a mobile phone.

[0063] The above-mentioned second preset scale may be the same as the first preset scale or different from the first preset scale. The embodiments of the present invention do not specifically limit the above-mentioned second preset scale.

[0064] The above-mentioned distortion model can be any image generation model. The input is the image to be processed at the second preset scale and the target object mask at the second preset scale. Through a feature network with multi-layer convolution and downsampling, features of different scales of the image are extracted, and then through a decoding network that continuously upsamples, the features of different scales are fused to output a deformation grid map with the same size as the image to be processed at the second preset scale. In the embodiments of the present invention, in order to achieve fast deformation correction, a lightweight distortion model is adopted, and the image to be processed and the target object mask are scaled to the second scale and input into the distortion model, and the output of the deformation grid map can be completed within 10 ms on a terminal such as a mobile phone.

[0065] In an alternative embodiment of the present invention, in step S306, the step of performing deformation correction on the image to be processed based on the deformation grid map includes: scaling the deformation grid map to the target scale to obtain a deformation grid map at the target scale, where the target scale is equal to the scale of the image to be processed; offsetting the corresponding pixel points in the image to be processed according to the offset represented by the pixel value of each pixel point in the deformation grid map at the target scale to obtain a corrected image of the image to be processed.

[0066] Next, refer to Figure 8 for an overall introduction to the process of the image processing method of the present invention. Refer to Figure 8, obtain the image to be processed, then scale the image to be processed to the first preset scale to obtain the image to be processed at the first preset scale. Then, input the image to be processed at the first preset scale into the segmentation model to output the target object mask at the first preset scale. Next, scale the image to be processed and the target object mask at the first preset scale to the second preset scale respectively to obtain the image to be processed at the second preset scale and the target object mask at the second preset scale. Then, input the image to be processed at the second preset scale and the target object mask at the second preset scale into the distortion model to obtain the deformation grid map corresponding to the image to be processed. Scale the deformation grid map to the target scale (the target scale is equal to the scale of the image to be processed) to obtain the deformation grid map at the target scale. Finally, offset the corresponding pixel points in the image to be processed according to the offset represented by the pixel values of each pixel point in the deformation grid map at the target scale to obtain the corrected image of the image to be processed.

[0067] The above content has introduced in detail the process of the distortion model of the present invention for processing the image to be processed. Next, the training process of the distortion model will be described in detail.

[0068] In an optional embodiment of the present invention, the training process of the distortion model is as follows:

[0069] (1) Obtain a training sample set, where the training sample set includes: training object images, training object masks corresponding to the training object images, and deformation grid maps corresponding to the training object images;

[0070] The above-mentioned training object image refers to an image containing a training object, which can be an original image obtained by shooting a training object with an ultra-wide-angle lens, or an image containing a training object in an existing dataset. The above-mentioned training object mask corresponding to the training object image can specifically be generated based on the training object position marked for the training object in the training object image, or obtained by inputting the training object image into a trained segmentation model.

[0071] After obtaining the training object image, the deformation grid map required for deforming and correcting the training object image can be estimated using traditional distortion correction methods. Since the above-mentioned training sample set can be generated offline, when estimating the deformation grid map of the training object image using traditional distortion correction methods, first estimate the deformation grid map corresponding to each training object image according to a set of parameters in the traditional distortion correction method. Then, manually select the incorrect results from all the obtained deformation grid maps, and then adjust the parameters in the traditional distortion correction method. Use the method after parameter adjustment to estimate the deformation grid map of the training object image corresponding to the above incorrect results until the deformation grid maps corresponding to most of the training object images are accurate, and the deformation grid map corresponding to the training object image can be obtained.

[0072] (2) Train the original distortion model with the training sample set to obtain a distortion model.

[0073] Input the training object image and the training object mask corresponding to the training object image into the distortion model. Determine the loss based on the deformation grid map output by the distortion model and the deformation grid map corresponding to the training object image, and adjust the parameters of the distortion model according to the loss. When the training completion condition is satisfied, end the training. The training object mask helps to prompt the distortion model to pay attention to the training object mask. Compared with the method of inputting the training object image into the distortion model without inputting the training object mask, this method is more likely to obtain a deformation grid map closer to the true value.

[0074] In one example, inputting the training object image and the training object mask corresponding to the training object image into the distortion model means inputting the R layer, G layer, and B layer of the RGB training object image and the training object mask as four layers of images into the distortion model.

[0075] The image processing method of the present invention can be used for image distortion correction and also for video distortion correction. The image processing speed is fast and the image correction effect is good. By experimenting with the method of the present invention on a mobile phone with a Qualcomm 855, it can be seen that the time required to correct the image in Fig. 9(a) is 25 ms, and the corresponding corrected image is shown in Fig. 9(b).

[0076] Example 3:

[0077] The embodiment of the present invention also provides an image processing device, which is mainly used to execute the image processing method provided in the above content of the embodiment of the present invention. The following is a specific introduction to the image processing device provided in the embodiment of the present invention.

[0078] Figure 10 It is a schematic diagram of an image processing device according to an embodiment of the present invention, as Figure 10 shown. The image processing device mainly includes: an acquisition unit 10, a processing unit 20, and a deformation correction unit 30, where:

[0079] The acquisition unit 10 is used to acquire a to-be-processed image containing a target object and a target object mask corresponding to the to-be-processed image;

[0080] The processing unit 20 is used to input the to-be-processed image and the target object mask into the distortion model to obtain a deformation grid map corresponding to the to-be-processed image, where the pixel value of each pixel point of the deformation grid map represents the offset of the corresponding pixel point in the to-be-processed image;

[0081] The deformation correction unit 30 is used to perform deformation correction on the to-be-processed image based on the deformation grid map to obtain a corrected image of the to-be-processed image.

[0082] In an embodiment of the present invention, first, a to-be-processed image including a target object and a target object mask corresponding to the to-be-processed image are obtained. Then, the to-be-processed image and the target object mask are input into a distortion model to obtain a deformed mesh map corresponding to the to-be-processed image. Wherein, the pixel value of each pixel point of the deformed mesh map represents the offset of the corresponding pixel point in the to-be-processed image; the to-be-processed image is corrected based on the deformed mesh map to obtain a corrected image of the to-be-processed image. As can be seen from the above description, the present invention uses a distortion model to process the to-be-processed image and the target object mask, directly obtains a deformed mesh map corresponding to the to-be-processed image, and then corrects the to-be-processed image based on the deformed mesh map. This process does not require parameter optimization, can greatly improve the operation efficiency and enhance the correction effect; the target object mask is used in the process of training and using the distortion model, which is beneficial to strengthening the model's attention to the area where the target object is located; the deformed mesh map output by the distortion model is used to correct the entire image, and no longer distinguishes between the target object and the background, making the boundary between the target object and the background in the corrected image more natural.

[0083] Optionally, the device is further configured to: scale the to-be-processed image to a first preset scale to obtain a to-be-processed image of the first preset scale.

[0084] Optionally, the obtaining unit is further configured to: input the to-be-processed image of the first preset scale into a segmentation model to obtain a target object mask.

[0085] Optionally, the device is further configured to: scale the to-be-processed image and the target object mask to a second preset scale respectively to obtain a to-be-processed image of the second preset scale and a target object mask of the second preset scale.

[0086] Optionally, the processing unit is further configured to: input the to-be-processed image of the second preset scale and the target object mask of the second preset scale into the distortion model to obtain a deformed mesh map corresponding to the to-be-processed image.

[0087] Optionally, the deformation correction unit is further configured to: scale the deformed mesh map to a target scale to obtain a deformed mesh map of the target scale, where the target scale is equal to the scale of the to-be-processed image; offset the corresponding pixel points in the to-be-processed image according to the offset represented by the pixel value of each pixel point in the deformed mesh map of the target scale to obtain a corrected image of the to-be-processed image.

[0088] Optionally, the device is further configured to: obtain a training sample set, where the training sample set includes: a training object image, a training object mask corresponding to the training object image, and a deformed mesh map corresponding to the training object image; train the original distortion model through the training sample set to obtain a distortion model.

[0089] The image processing device provided by the embodiments of the present invention has the same implementation principle and technical effects as those of the method embodiment in the foregoing Embodiment 2. For a brief description, for the parts not mentioned in the device embodiment, reference may be made to the corresponding content in the foregoing method embodiment.

[0090] In another embodiment, a computer-readable medium having non-volatile program code executable by a processor is also provided, and the program code causes the processor to execute the steps of the method of any of the foregoing embodiments 2.

[0091] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may be a mechanical connection or an electrical connection; it may be directly connected, or indirectly connected through an intermediate medium, and it may be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0092] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0093] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments, and will not be described in detail here.

[0094] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of units is only a logical function division, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the displayed or discussed coupling or direct coupling or communication connection between each other can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be in an electrical, mechanical, or other forms.

[0095] The units described as separation components may or may not be physically separated. The components presented as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the objectives of the solution of this embodiment.

[0096] In addition, in each embodiment of the present invention, the functional units can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

[0097] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or a part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0098] Finally, it should be noted that: the above embodiments are only specific implementation manners of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting it. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: any person skilled in the technical field of the present invention can still modify the technical solutions recorded in the foregoing embodiments, or can easily think of changes, or perform equivalent replacements for some of the technical features; and these modifications, changes, or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized in that, Including: Obtain a to-be-processed image including a target object and a target object mask corresponding to the to-be-processed image; Input the to-be-processed image and the target object mask into a distortion model to obtain a deformed grid map corresponding to the to-be-processed image, where the pixel value of each pixel point in the deformed grid map represents the offset of the corresponding pixel point in the to-be-processed image, and the target object mask corresponding to the to-be-processed image is used to represent the positions of the pixel points of the target object that the distortion model needs to pay attention to; Perform distortion correction on the to-be-processed image based on the deformed grid map to obtain a corrected image of the to-be-processed image, where the position of the target pixel point in the to-be-processed image is calculated based on the deformed grid map, and the pixel value of the pixel point at the corresponding position in the corrected image is modified to the pixel value of the target pixel point in the to-be-processed image.

2. The method according to claim 1, wherein The method further includes: Scale the to-be-processed image to a first preset scale to obtain a to-be-processed image of the first preset scale.

3. The method according to claim 2, wherein Obtaining the target object mask corresponding to the to-be-processed image includes: Input the to-be-processed image of the first preset scale into a segmentation model to obtain the target object mask.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Respectively scale the to-be-processed image and the target object mask to a second preset scale to obtain a to-be-processed image of the second preset scale and a target object mask of the second preset scale.

5. The method according to claim 4, wherein Inputting the to-be-processed image and the target object mask into the distortion model includes: Input the to-be-processed image of the second preset scale and the target object mask of the second preset scale into the distortion model to obtain a deformed grid map corresponding to the to-be-processed image.

6. The method according to any one of claims 1-5, characterized in that Performing distortion correction on the to-be-processed image based on the deformed grid map includes: Scale the deformed grid map to a target scale to obtain a deformed grid map of the target scale, where the target scale is equal to the scale of the to-be-processed image; Offset the corresponding pixel points in the to-be-processed image according to the offsets represented by the pixel values of each pixel point in the deformed grid map of the target scale to obtain a corrected image of the to-be-processed image.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: Obtain a training sample set, where the training sample set includes: training object images, training object masks corresponding to the training object images, and deformed grid maps corresponding to the training object images; Train an original distortion model through the training sample set to obtain the distortion model.

8. An image processing apparatus, characterized in that, Including: An obtaining unit, configured to obtain a to-be-processed image including a target object and a target object mask corresponding to the to-be-processed image; A processing unit, configured to input the to-be-processed image and the target object mask into a distortion model to obtain a deformed grid map corresponding to the to-be-processed image, where the pixel value of each pixel point in the deformed grid map represents the offset of the corresponding pixel point in the to-be-processed image, and the target object mask corresponding to the to-be-processed image is used to represent the positions of the pixel points of the target object that the distortion model needs to pay attention to; A deformation correction unit is configured to perform deformation correction on the image to be processed based on the deformed grid map, so as to obtain a corrected image of the image to be processed. Specifically, the position of a target pixel point in the image to be processed in the corrected image is calculated based on the deformed grid map, and the pixel value of the pixel point at the corresponding position in the corrected image is modified to be the pixel value of the target pixel point in the image to be processed.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7 above.

10. A computer-readable medium having non-volatile program code executable by a processor, characterized in that, The program code causes the processor to execute the steps of the method according to any one of claims 1 to 7 above.

Images