Image processing method and device, electronic device, and computer readable medium

Abstract

The present disclosure relates to an image processing method and device, an electronic device, and a computer-readable medium. The image processing method includes: obtaining an image to be completed and a reference image; obtaining the completed 3D object information of the image to be completed based on the 3D object information of the image to be completed and the 3D object information of the reference image; obtaining optical flow information from the 3D object information of the reference image to the completed 3D object information of the image to be completed; completing the object in the image to be completed based on the optical flow information to obtain the completed image. The present disclosure solves the problem in the related art that the completed part of the completed image often has jitter or discord in timing.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to an image processing method and device, an electronic device, and a computer-readable medium. Background Art

[0002] At present, image completion is achieved through deep learning neural networks. For example, in the body dance project, users input their own pictures and specified templates (such as multiple pictures corresponding to a set of dance movements) into the deep learning neural network. After calculation by the deep learning neural network, a video can be output. Each frame of the video is an updated picture of the user, and the objects in the updated pictures are consistent with the dance movements of the objects in the specified template. In this process, if the user inputs a picture of the upper body of the human body, a deep learning neural network is required to complete the lower body of the human body. However, the training of deep learning neural networks requires a large amount of training data, and the degree of freedom of the completed picture itself is relatively high (such as the missing lower body of the human body may be in multiple postures). Therefore, the deep learning neural network is difficult to learn, and the final effect is difficult to guarantee, which often causes the completed part to jitter or violate the timing. Summary of the invention

[0003] The present disclosure provides an image processing method and device, an electronic device, and a computer-readable medium, so as to at least solve the problem in the related art that the completed part of the completed image often has jitter or violation in timing.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, comprising: acquiring an image to be completed and a reference image, wherein the reference image contains image information of a part to be completed of an object in the image to be completed; obtaining completed 3D object information of the image to be completed based on 3D object information of the image to be completed and 3D object information of the reference image; acquiring optical flow information from the 3D object information of the reference image to the completed 3D object information of the image to be completed; and completing the object in the image to be completed based on the optical flow information to obtain a completed image.

[0005] Optionally, based on the 3D object information of the image to be completed and the 3D object information of the reference image, the completed 3D object information of the image to be completed is obtained, including: acquiring the 3D object information of the image to be completed and the 3D object information of the reference image; replacing the 3D object information of the to-be-completed part in the 3D object information of the image to be completed with the 3D object information of the part corresponding to the to-be-completed part in the 3D object information of the reference image, to obtain the completed 3D object information of the image to be completed.

[0006] Optionally, obtaining optical flow information from the 3D object information of the reference image to the completed 3D object information of the image to be completed includes: rendering the completed 3D object information of the image to be completed to obtain a first rendering image; rendering the 3D object information of the reference image to obtain a second rendering image; and using the optical flow information from the second rendering image to the second rendering image as the optical flow information from the 3D object information of the reference image to the completed 3D object information of the image to be completed.

[0007] Optionally, the object in the image to be completed is completed based on the optical flow information to obtain a completed image, including: adjusting the image information of the reference image to intermediate image information aligned with the image to be completed based on the optical flow information; replacing the image information of the to-be-completed part in the image information of the image to be completed with the image information of the part corresponding to the to-be-completed part in the intermediate image information to obtain the completed image information of the image to be completed; and obtaining the completed image based on the completed image information of the image to be completed.

[0008] Optionally, the image information includes one of the following: parsing information, an RGB image, and a mask image.

[0009] Optionally, after completing the object in the image to be completed based on the optical flow information to obtain the completed image, it also includes: inputting the completed image, the rendering of the completed image and the rendering of the target image into a predetermined neural network to obtain an updated completed image, wherein the posture of the object in the updated completed image is consistent with that of the object in the target image.

[0010] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, comprising: an image acquisition unit, configured to acquire an image to be completed and a reference image, wherein the reference image contains image information of a part to be completed of an object in the image to be completed; a 3D object information completion unit, configured to obtain completed 3D object information of the image to be completed based on the 3D object information of the image to be completed and the 3D object information of the reference image; an optical flow information acquisition unit, configured to acquire optical flow information from the 3D object information of the reference image to the completed 3D object information of the image to be completed; and a processing unit, configured to complete the object in the image to be completed based on the optical flow information to obtain a completed image.

[0011] Optionally, the 3D object information completion unit is further configured to obtain 3D object information of the image to be completed and 3D object information of the reference image; replace the 3D object information of the to-be-completed part in the 3D object information of the image to be completed with the 3D object information of the reference image corresponding to the to-be-completed part in the 3D object information, to obtain the completed 3D object information of the image to be completed.

[0012] Optionally, the optical flow information acquisition unit is further configured to render the completed 3D object information of the image to be completed to obtain a first rendering image; render the 3D object information of the reference image to obtain a second rendering image; and use the optical flow information from the second rendering image to the second rendering image as the optical flow information from the 3D object information of the reference image to the completed 3D object information of the image to be completed.

[0013] Optionally, the processing unit is also configured to adjust the image information of the reference image to intermediate image information aligned with the image to be completed based on the optical flow information; replace the image information of the to-be-completed part in the image information of the image to be completed with the image information of the part corresponding to the to-be-completed part in the intermediate image information to obtain the completed image information of the image to be completed; and obtain the completed image based on the completed image information of the image to be completed.

[0014] Optionally, the image information includes one of the following: parsing information, an RGB image, and a mask image.

[0015] Optionally, the processing unit is further configured to complete the object in the image to be completed based on the optical flow information, and after obtaining the completed image, input the completed image, the rendering of the completed image and the rendering of the target image into a predetermined neural network to obtain an updated completed image, wherein the posture of the object in the updated completed image is consistent with that of the object in the target image.

[0016] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the image processing method according to the present disclosure.

[0017] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the image processing method according to the present disclosure as described above.

[0018] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which implement the image processing method according to the present disclosure when executed by a processor.

[0019] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0020] According to the image processing method and device, electronic device, and computer-readable medium disclosed in the present invention, the 3D object information of the image to be completed is completed by using the 3D object information of the reference image, and the corresponding optical flow information can be obtained based on the completed 3D object information and the 3D object information of the reference image. The object in the image to be completed can be completed based on the optical flow information without relying on a neural network for completion, and the problem of jitter or inconsistency in the timing of the completed part caused by the neural network completion method can be overcome. Therefore, the present invention solves the problem that the completed part of the completed image in the related art often jitters or inconsistencies in the timing.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0023] Figure 1 is a schematic diagram showing an implementation scenario of an image processing method according to an exemplary embodiment of the present disclosure;

[0024] Figure 2 is a flowchart of an image processing method according to an exemplary embodiment;

[0025] Figure 3 is a schematic diagram showing the result of performing image completion and matching actions using a neural network according to an exemplary embodiment;

[0026] Figure 4 is a schematic diagram showing human body analysis information and a 3D human body rendering image completed by using the present disclosure according to an exemplary embodiment;

[0027] Figure 5 is a schematic diagram showing the result of performing image completion using the present disclosure and using a neural network matching action according to an exemplary embodiment;

[0028] Figure 6 is a schematic diagram of human parsing 3 of a reference image according to an exemplary embodiment;

[0029] Figure 7 is a schematic diagram showing intermediate image information according to an exemplary embodiment;

[0030] Figure 8 is a block diagram of an image processing device according to an exemplary embodiment;

[0031] Fig. 9 is a block diagram of an electronic device 900 according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0034] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0035] At present, when a user inputs a self-image and a specified template (e.g., a picture corresponding to a set of dance moves) into a deep learning neural network, a video is obtained, in which each frame of the video is an updated self-image, and the objects in the updated self-image are respectively consistent with the dance moves of the objects in the specified template. In this process, if the user inputs a self-image of the upper body, the deep learning neural network is required to complete the lower body of the human body. However, the training of deep learning neural networks requires a large amount of training data, and the completed image itself has a high degree of freedom (e.g., the missing lower body of the human body may be in a variety of postures). Therefore, deep learning neural networks are difficult to learn, and the final effect is difficult to guarantee, which often leads to jitter or discord in the timing of the completed part.

[0036] In response to the above problems, the present disclosure provides an image processing method, which completes the 3D object information of the image to be completed through the 3D object information of the reference image, and the corresponding optical flow information can be obtained based on the completed 3D object information and the 3D object information of the reference image. The image to be completed can be completed based on the optical flow information without relying on a neural network for completion, and the problem of jitter or inconsistency in the timing of the completed part caused by the neural network completion method can be overcome. The scene of completing a character picture is taken as an example for explanation below.

[0037] Figure 1 is a schematic diagram showing an implementation scenario of an image processing method according to an exemplary embodiment of the present disclosure, such as Figure 1 The implementation scenario includes a server 100, a user terminal 110 and a user terminal 120, wherein the user terminals are not limited to two, and include but are not limited to mobile phones, personal computers and other devices. The user terminal can be equipped with a camera for acquiring images. The server can be a single server, or a server cluster composed of several servers, or a cloud computing platform or a virtualization center.

[0038] The user terminal 110 or the user terminal 120 takes a picture of the upper body of a person as the image to be completed through a camera, and sends the obtained image to be completed to the server 100. After receiving the image to be completed, the server 100 obtains a corresponding reference image, which contains image information of the part to be completed in the image to be completed. The reference image can be an image stored locally or an image sent by the user terminal 110 or the user terminal 120. Then, the server 100 generates a 3D human body based on the image to be completed and the reference image. The 3D human body information of the image to be completed is obtained by combining the 3D human body information of the reference image and the 3D human body information of the reference image to obtain the completed 3D human body information of the image to be completed, and then the optical flow information from the 3D human body information of the reference image to the completed 3D human body information of the image to be completed is obtained. Finally, the object in the image to be completed is completed based on the optical flow information to obtain the completed image, and then the completed image, the human body rendering of the completed image and the human body rendering of the target image are input into a predetermined neural network to obtain an updated completed image, in which the action of the person in the updated completed image is consistent with that of the person in the target image.

[0039] Hereinafter, an image processing method and apparatus, and an electronic device according to exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0040] Figure 2 is a flowchart of an image processing method according to an exemplary embodiment. Figure 2 As shown, the image processing method comprises the following steps:

[0041] In step S201, the image to be completed and the reference image are obtained, wherein the reference image contains image information of the part to be completed of the object in the image to be completed. The image to be completed can be a temporarily acquired image or a locally stored image, as long as it is an image that the user wants to complete. The reference image can be a temporarily acquired image or a locally stored image, as long as it meets the user's requirements, and the present disclosure does not limit the acquisition method.

[0042] In step S202, based on the 3D object information of the image to be completed and the 3D object information of the reference image, the completed 3D object information of the image to be completed is obtained. In this step, if the image to be completed is an image containing the upper body of a human body, and the purpose is to complete the lower body of the human body, then the 3D object information of this step can be the 3D human body information of the image to be completed, that is, only the 3D object information of the human body corresponding to the image is obtained, so that the calculation cost can be reduced.

[0043] According to an exemplary embodiment of the present disclosure, based on the 3D object information of the image to be completed and the 3D object information of the reference image, the completed 3D object information of the image to be completed is obtained, including: obtaining the 3D object information of the image to be completed and the 3D object information of the reference image; replacing the 3D object information of the to-be-completed part in the 3D object information of the image to be completed with the 3D object information of the part corresponding to the to-be-completed part in the 3D object information of the reference image, to obtain the completed 3D object information of the image to be completed. According to this embodiment, the 3D object information of the reference image can be used to complete the 3D object information of the image to be completed conveniently and quickly by replacement.

[0044] For example, assuming that the image to be completed is an image of the upper body of a human body, and the lower body of the human body needs to be completed, the 3D human body information of the image to be completed is smpl 11, and the 3D human body information of the reference image is smpl 3, wherein smpl 11 and smpl 3 can obtain respective 3D human body rendering images after rendering. Since each 3D human body information is an N*M vector, representing the axis angles of N M-dimensional bone points of the 3D human body, by adjusting the values ​​in smpl, the 3D human body in the human body rendering image can present different postures, which is equivalent to the person in the image making different movements, that is, by replacing the values ​​of the lower body of the human body in smpl 11 with the corresponding values ​​in smpl 3, the movements of the lower body of the human body in smpl 11 can be replaced with the corresponding movements in smpl 3. It should be noted that smpl refers to the parametric three-dimensional model of the human body constructed in an article published by Max Planck Society in 2015, "SMPL: a skinned multi-person linear model". The human body can be understood as the sum of a basic model and the deformation based on the model. PCA is performed on the deformation to obtain low-dimensional parameters that describe the shape - shape parameters (shape); at the same time, a motion tree is used to represent the posture of the human body, that is, the rotation relationship between each joint point and the parent node of the motion tree. This relationship can be expressed as a three-dimensional vector. Finally, the local rotation vector of each joint point constitutes the posture parameter (pose) of the smpl model.

[0045] return Figure 2 In step S203, optical flow information is obtained from the 3D object information of the reference image to the completed 3D object information of the image to be completed. Optical flow refers to the speed of pattern movement in a time-varying image, because when an object is moving, the brightness pattern of its corresponding point on the image is also moving. Optical flow expresses the change of the image, and because it contains information about the movement of the target, it can be used by the observer to determine the movement of the target. Optical flow information is information about the speed of pattern movement in a time-varying image.

[0046] According to an exemplary embodiment of the present disclosure, obtaining optical flow information from 3D object information of a reference image to the completed 3D object information of an image to be completed includes: rendering the completed 3D object information of the image to be completed to obtain a first rendering; rendering the 3D object information of the reference image to obtain a second rendering; and using the optical flow information from the second rendering to the second rendering as the optical flow information from the 3D object information of the reference image to the completed 3D object information of the image to be completed. According to this embodiment, relatively accurate optical flow information can be obtained based on the rendering of the 3D object information.

[0047] For example, still taking the image to be completed as the upper body image of a human body as an example, after replacing the lower body values ​​of the human body in smpl 11 with the corresponding values ​​in smpl 3, smpl 12 (i.e. the completed 3D human body information of the image to be completed) can be obtained. After smpl 12 is rendered, a 3D human body rendering image 12 can be obtained, and after smpl 3 is rendered, a 3D human body rendering image 12 can be obtained. Figure 3 , through 3D human body rendering Figure 12 and 3D human body rendering Figure 3 , you can get 3D human body rendering Figure 3 to 3 D Optical flow of human body rendering Fig. 12.

[0048] In step S204, the object in the image to be completed is completed based on the optical flow information to obtain a completed image.

[0049] According to an exemplary embodiment of the present disclosure, the object in the image to be completed is completed based on the optical flow information to obtain the completed image, including: adjusting the image information of the reference image to the intermediate image information aligned with the image to be completed based on the optical flow information; replacing the image information of the part to be completed in the image information of the image to be completed with the image information of the part corresponding to the part to be completed in the intermediate image information to obtain the completed image information of the image to be completed; and obtaining the completed image based on the completed image information of the image to be completed. According to this embodiment, the reference image is adjusted by the optical flow information so that the reference image is aligned with the image to be completed, so that the image to be completed can be completed by the replacement operation later, and a relatively accurate completed image can be obtained.

[0050] For example, still taking the image to be completed as the upper body image of a human body as an example, the above 3D human body rendering is obtained Figure 3 to 3 After the optical flow of D human body rendering Figure 12 is obtained, an optical flow wrap operation can be performed on the reference image based on the optical flow information to adjust the image information of the reference image to intermediate image information aligned with the image to be completed. Specifically, according to the coordinates of each point in the optical flow information, the pixels of the corresponding coordinate points in the reference image are assigned to the corresponding positions of the optical flow information, wherein the coordinates of each point in the optical flow information may not have an exact correspondence in the reference image, and the pixels corresponding to the coordinates in the reference image that do not have an exact correspondence in the optical flow information can be obtained by the nearest neighbor or bilinear method, and finally the intermediate image information is output, that is, the image of the reference image after the optical flow warp, this image will have an aligned feature, therefore, the lower body of the human body in the aligned image can be replaced with the image to be completed, and the missing lower body information of the human body in the image to be completed is completed, and at this time the completed image includes not only the original upper body information of the human body, but also the completed lower body information of the human body.

[0051] According to an exemplary embodiment of the present disclosure, the image information includes one of the following: parsing information, RGB map, mask map. According to this embodiment, various information of the image can be supplemented, ensuring the popularity of the method of the present disclosure.

[0052] According to an exemplary embodiment of the present disclosure, after completing the object in the image to be completed based on the optical flow information to obtain the completed image, it also includes: inputting the completed image, the human body rendering of the completed image, and the human body rendering of the target image into a predetermined neural network to obtain an image consistent with the action in the target image. According to this embodiment, directly using the completed image as the input of the neural network can ensure that the image output by the neural network can better meet the needs of the user, that is, it can allow the user to perfectly match the action of the target image, and avoids the problem of using a neural network to complete the completed image, resulting in jitter or violation in timing, and the inability of the user to perfectly match the action of the target image.

[0053] In order to verify that using the completed image disclosed in the present invention as the input of the neural network can allow the user to perfectly match the action of the target image, the following takes the image information as human parsing information as an example and combines Figure 3 , 4 , 5 are described, wherein, Figure 3 is a schematic diagram showing the result of performing image completion and matching actions using a neural network according to an exemplary embodiment, Figure 4 is a schematic diagram showing human body analysis information and a 3D human body rendering image completed by using the present disclosure according to an exemplary embodiment, Figure 5 It is a schematic diagram showing the result of performing image completion using the present disclosure and using a neural network matching action according to an exemplary embodiment.

[0054] like Figure 3 As shown, the neural network input is the human parsing 11 of the image to be completed and the 3D human rendering 11 and the 3D human rendering of the specified template Figure 2 , the output is human parsing 2 of the image consistent with the specified template action. It can be seen that the lower body of the human body is completed in human parsing 2, but the effect is not good. The calf part is marked as skin (light-colored area), the thigh part is marked as pants (dark part), and the connection is a curve, which is unnatural (theoretically it should be a nearly straight line), and there are jitter and semantic inconsistency problems in timing (in the video, the light-colored area of ​​the calf jitters, sometimes more and sometimes less). Among them, the first small picture on the left is human parsing 11 of the image to be completed, the second small picture is the 3D human body rendering 11 of the image to be completed, and the third small picture is the 3D human body rendering of the specified template (that is, the target image in the above embodiment) Figure 2,The fourth small figure is human parsing2 of the image consistent with the specified template action.

[0055] like Figure 4 As shown, the human parsing 11 of the image to be completed is completed into human parsing 12 by the method disclosed in the present invention, and the lower body of the 3D human body rendering image 11 of the image to be completed is corrected into the 3D human body rendering image 12, wherein, Figure 4 The left side is human parsing 12. Figure 4 The 3D human body rendering is shown in Figure 12 on the middle right.

[0056] like Figure 5 As shown, at this time, the input of the neural network is the completed human parsing 12, the completed 3D human body rendering 12, and the 3D human body rendering of the specified template Figure 2 , the output is human parsing 22 of the image consistent with the specified template action. It can be seen that the generated human parsing 22 has pants on the lower body, and the boundary with the upper body is flat, which is more natural overall and has stable timing. Therefore, it can significantly improve the naturalness of the final effect and significantly enhance the timing stability. Among them, the first small picture on the left is the completed human parsing 12, the second small picture is the completed 3D human body rendering 12, and the third small picture is the 3D human body rendering of the specified template Figure 2 ,The fourth small figure is the human parsing22 of the image that is consistent with the specified template action.

[0057] Still taking the human parsing information of the image information as an example, the following describes in detail how the present disclosure completes the human parsing 11 of the image to be completed into the human parsing 12 .

[0058] Assume that there are human parsing 11 and smpl 11 of the image to be completed, and human parsing 3 and smpl 3 of the reference image. Among them, smpl 11 and smpl 3 can obtain their own 3D human body renderings after rendering (such as the rendering of smpl11 is 3D human body rendering 11). Since each 3D human body information is an N*M vector, representing the axis angles of N M-dimensional bone points of the 3D human body, by adjusting the values ​​in smpl, the 3D human body in the human body rendering can present different postures, that is, the person in the picture makes different movements.

[0059] Figure 6 is a schematic diagram of human parsing 3 of a reference image according to an exemplary embodiment, as shown in Figure 6As shown, human parsing 3 of the reference image is the specified standard human parsing diagram (arms open, legs naturally separated, short sleeves on the upper body, long pants on the lower body). First, replace the lower body information of Smpl 11 with the corresponding values ​​in Smpl 3, that is, replace the lower body movements of Smpl 11 with the corresponding movements in Smpl 3, and you can get Smpl 12. Smpl 12 is rendered to get 3D human rendering 12, and Smpl 3 is rendered to get 3D human rendering Figure 3 , through 3D human body rendering Figure 12 and 3D human body rendering Figure 3 , you can get 3D human body rendering Figure 3 to 3 D The optical flow of the human body rendering image 12, based on this optical flow, humanparsing 3 can be changed into humanparsing 312 (equivalent to the above-mentioned intermediate image information), such as Figure 7 As shown, Figure 7 is a schematic diagram of an intermediate image information according to an exemplary embodiment. Then, the missing lower body information of the human body in human parsing 11 can be replaced with the corresponding information in human parsing 312 to obtain human parsing 12, that is, the image information of the completed image is obtained, which can be used as the input of the neural network to assist in obtaining an image that matches the action of the target image.

[0060] In summary, the above embodiment can complete the lower body of the human body based on the optical flow warp scheme, without the need to design and train a neural network to complete the human parsing 11 of the image to be completed, that is, without the need to use a neural network, relying on 3D human body information (smpl) and corresponding optical flow information to complete the missing lower body information of the human parsing 11, and in addition to completing the human parsing information, it can also be used to complete other related image information of the human body, such as RGB images and mask images.

[0061] Figure 8 is a block diagram of an image processing device according to an exemplary embodiment. Figure 8 , the device comprises:

[0062] The image acquisition unit 80 is configured to acquire the image to be completed and the reference image, wherein the reference image contains image information of the to-be-completed part of the object in the image to be completed; the 3D object information completion unit 82 is configured to obtain the completed 3D object information of the image to be completed based on the 3D object information of the image to be completed and the 3D object information of the reference image; the optical flow information acquisition unit 84 is configured to obtain optical flow information from the 3D object information of the reference image to the completed 3D object information of the image to be completed; the processing unit 86 is configured to complete the object in the image to be completed based on the optical flow information to obtain the completed image.

[0063] According to an exemplary embodiment of the present disclosure, the 3D object information completion unit 82 is further configured to obtain 3D object information of the image to be completed and 3D object information of the reference image; replace the 3D object information of the to-be-completed part in the 3D object information of the image to be completed with the 3D object information of the part corresponding to the to-be-completed part in the 3D object information of the reference image, and obtain the completed 3D object information of the image to be completed.

[0064] According to an exemplary embodiment of the present disclosure, the optical flow information acquisition unit 84 is further configured to render the completed 3D object information of the image to be completed to obtain a first rendering image; render the 3D object information of the reference image to obtain a second rendering image; and use the optical flow information from the second rendering image to the second rendering image as the optical flow information from the 3D object information of the reference image to the completed 3D object information of the image to be completed.

[0065] According to an exemplary embodiment of the present disclosure, the processing unit 86 is also configured to adjust the image information of the reference image to intermediate image information aligned with the image to be completed based on the optical flow information; replace the image information of the to-be-completed part in the image information of the image to be completed with the image information of the part corresponding to the to-be-completed part in the intermediate image information, to obtain the completed image information of the image to be completed; and obtain the completed image based on the completed image information of the image to be completed.

[0066] According to an exemplary embodiment of the present disclosure, the image information includes one of the following: parsing information, an RGB map, and a mask map.

[0067] According to an exemplary embodiment of the present disclosure, the processing unit 86 is further configured to complete the object in the image to be completed based on the optical flow information to obtain the completed image, and then input the completed image, the rendering of the completed image and the rendering of the target image into a predetermined neural network to obtain an updated completed image, wherein the posture of the object in the updated completed image is consistent with that of the object in the target image.

[0068] According to an embodiment of the present disclosure, an electronic device may be provided. Fig. 9 It is a block diagram of an electronic device 900 according to an embodiment of the present disclosure, and the electronic device includes at least one memory 901 and at least one processor 902, wherein a set of computer executable instructions is stored in the at least one memory, and when the computer executable instruction set is executed by the at least one processor, an image processing method according to an embodiment of the present disclosure is executed.

[0069] As an example, the electronic device 900 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 1000 is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 900 may also be part of an integrated control system or a system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0070] In the electronic device 900, the processor 902 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor 902 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0071] The processor 902 may execute instructions or codes stored in the memory, wherein the memory 901 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0072] The memory 901 may be integrated with the processor 902, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 901 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 901 and the processor 902 may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 902 can read files stored in the memory 901.

[0073] In addition, the electronic device 900 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.

[0074] According to an embodiment of the present disclosure, a computer-readable storage medium may also be provided, wherein when the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the image processing method of the embodiment of the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0075] According to an embodiment of the present disclosure, a computer program product is provided, including computer instructions, and when the computer instructions are executed by a processor, the image processing method of the embodiment of the present disclosure is implemented.

[0076] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0077] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Acquire an image to be completed and a reference image, wherein the reference image contains image information of a part to be completed of an object in the image to be completed; Obtaining completed 3D object information of the image to be completed based on the 3D object information of the image to be completed and the 3D object information of the reference image; Acquire optical flow information from the 3D object information of the reference image to the completed 3D object information of the image to be completed; The object in the image to be completed is completed based on the optical flow information to obtain a completed image.

2. The image processing method according to claim 1, characterized in that: The obtaining the completed 3D object information of the image to be completed based on the 3D object information of the image to be completed and the 3D object information of the reference image includes: Acquire 3D object information of the image to be completed and 3D object information of the reference image; The 3D object information of the to-be-completed part in the 3D object information of the to-be-completed image is replaced with the 3D object information of the reference image corresponding to the to-be-completed part in the 3D object information to obtain the completed 3D object information of the to-be-completed image.

3. The image processing method according to claim 1, characterized in that: The acquiring of optical flow information from the 3D object information of the reference image to the completed 3D object information of the image to be completed includes: Rendering the completed 3D object information of the image to be completed to obtain a first rendering image; Rendering the 3D object information of the reference image to obtain a second rendering image; The optical flow information from the second rendering image to the first rendering image is used as the optical flow information from the 3D object information of the reference image to the completed 3D object information of the image to be completed.

4. The image processing method according to claim 1, wherein: The method of completing the object in the image to be completed based on the optical flow information to obtain a completed image includes: Based on the optical flow information, adjusting the image information of the reference image to be intermediate image information aligned with the image to be completed; Replacing the image information of the part to be completed in the image information of the image to be completed with the image information of the part corresponding to the part to be completed in the intermediate image information, so as to obtain the completed image information of the image to be completed; The completed image is obtained based on the completed image information of the image to be completed.

5. The image processing method according to claim 4, characterized in that: The image information includes one of the following: analysis information, RGB image, and mask image.

6. The image processing method according to claim 1, characterized in that: After completing the object in the image to be completed based on the optical flow information to obtain the completed image, the method further includes: The completed image, the rendering of the completed image and the rendering of the target image are input into a predetermined neural network to obtain an updated completed image, wherein the posture of the object in the updated completed image is consistent with that of the object in the target image.

7. An image processing device, characterized in that: include: an image acquisition unit, configured to acquire an image to be completed and a reference image, wherein the reference image contains image information of a part to be completed of an object in the image to be completed; a 3D object information completion unit, configured to obtain the completed 3D object information of the image to be completed based on the 3D object information of the image to be completed and the 3D object information of the reference image; an optical flow information acquisition unit, configured to acquire optical flow information from the 3D object information of the reference image to the completed 3D object information of the image to be completed; The processing unit is configured to complete the object in the image to be completed based on the optical flow information to obtain a completed image.

8. The image processing device according to claim 7, wherein: The 3D object information completion unit is further configured to obtain the 3D object information of the image to be completed and the 3D object information of the reference image; replace the 3D object information of the to-be-completed part in the 3D object information of the image to be completed with the 3D object information of the part corresponding to the to-be-completed part in the 3D object information of the reference image, and obtain the completed 3D object information of the image to be completed.

9. The image processing device according to claim 7, wherein: The optical flow information acquisition unit is further configured to render the completed 3D object information of the image to be completed to obtain a first rendering image; and render the 3D object information of the reference image to obtain a second rendering image; The optical flow information from the second rendering image to the first rendering image is used as the optical flow information from the 3D object information of the reference image to the completed 3D object information of the image to be completed.

10. The image processing device according to claim 7, wherein: The processing unit is further configured to adjust the image information of the reference image to intermediate image information aligned with the image to be completed based on the optical flow information; replace the image information of the to-be-completed part in the image information of the image to be completed with the image information of the part corresponding to the to-be-completed part in the intermediate image information, to obtain the completed image information of the image to be completed; The completed image is obtained based on the completed image information of the image to be completed.

11. The image processing device according to claim 10, wherein: The image information includes one of the following: analysis information, RGB image, and mask image.

12. The image processing device according to claim 7, wherein: The processing unit is further configured to complete the object in the image to be completed based on the optical flow information to obtain a completed image, and then input the completed image, a rendering of the completed image and a rendering of the target image into a predetermined neural network to obtain an updated completed image, wherein the posture of the object in the updated completed image is consistent with that of the object in the target image.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image processing method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the image processing method according to any one of claims 1 to 6.

15. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the image processing method according to any one of claims 1 to 6 is implemented.

Images