An image processing method, apparatus, device, and storage medium

Abstract

Embodiments of the present application disclose an image processing method, apparatus, device, and storage medium. Among them, the method includes: obtaining a two-dimensional image and a three-dimensional image of an image frame including an object to be rendered; determining three-dimensional geometric data in the three-dimensional image; and adjusting the object to be rendered in the two-dimensional image frame based on the three-dimensional geometric data and the light and shadow parameters of the virtual environment to obtain a target image in which the image frame matches the virtual environment.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular, to an image processing method, apparatus, device, and storage medium. Background Art

[0002] In related technologies, when performing effect synthesis on an image, usually, partial pictures in two different images are directly stitched and fitted to obtain an effect picture; thus, due to the differences in the backgrounds and scene light sources in the two different images, the displayed picture of the effect picture is inconsistent. Summary of the Invention

[0003] In view of this, embodiments of this application provide an image processing method, apparatus, device, and storage medium.

[0004] The technical solution of this application is implemented as follows:

[0005] Embodiments of this application provide an image processing method, and the method includes:

[0006] Obtain a two-dimensional image and a three-dimensional image of an image frame including an object to be rendered;

[0007] Determine three-dimensional geometric data in the three-dimensional image;

[0008] Based on the three-dimensional geometric data and the light and shadow parameters of the virtual environment, adjust the object to be rendered in the two-dimensional image frame to obtain a target image whose image frame matches the virtual environment.

[0009] Embodiments of this application provide an image processing apparatus, and the apparatus includes:

[0010] An obtaining module, configured to obtain a two-dimensional image and a three-dimensional image of an image frame including an object to be rendered;

[0011] A determining module, configured to determine three-dimensional geometric data in the three-dimensional image;

[0012] An adjusting module, configured to adjust the object to be rendered in the two-dimensional image frame based on the three-dimensional geometric data and the light and shadow parameters of the virtual environment to obtain a target image whose frame matches the virtual environment.

[0013] Embodiments of this application further provide an electronic device, including: a processor, a memory, and a communication bus;

[0014] The communication bus is used to implement a communication connection between the processor and the memory;

[0015] The processor is configured to execute a program in the memory to implement the above-mentioned image processing method.

[0016] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement the image processing method as described above.

[0017] The image processing method, apparatus, device and storage medium provided by the embodiments of the present application first obtain a two-dimensional image and a three-dimensional image of an object to be rendered included in an image frame; secondly, determine three-dimensional geometric data in the three-dimensional image; finally, based on the three-dimensional geometric data and the light and shadow parameters of the virtual environment, adjust the object to be rendered in the two-dimensional image frame to obtain a target image in which the image frame matches the virtual environment; thus, by using the three-dimensional geometric data and the light and shadow parameters of the virtual environment to adjust the object to be rendered in the image frame, a target image in which the image frame matches the virtual environment can be obtained, which can improve the quality and efficiency of image rendering. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic flowchart of the first image processing method provided by an embodiment of the present application;

[0019] Figure 2 It is a schematic flowchart of the second image processing method provided by an embodiment of the present application;

[0020] Figure 3 It is a schematic flowchart of the third image processing method provided by an embodiment of the present application;

[0021] Figure 4 It is a schematic structural diagram of an image processing apparatus provided by an embodiment of the present application;

[0022] Figure 5 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

[0024] It should be understood that the "embodiments of the present application" or "the foregoing embodiments" mentioned throughout the specification mean that specific features, structures, or characteristics related to the embodiments are included in at least one embodiment of the present application. Therefore, the "in the embodiments of the present application" or "in the foregoing embodiments" that appear throughout the specification do not necessarily refer to the same embodiments. In addition, these specific features, structures, or characteristics can be combined in one or more embodiments in any suitable manner. In various embodiments of the present application, the magnitudes of the serial numbers of the above processes do not mean the order of execution, and the order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application. The serial numbers of the embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0025] Without special instructions, when the electronic device executes any step in the embodiments of the present application, it can be the processor of the electronic device that executes this step. It is also worth noting that the embodiments of the present application do not limit the order of execution of the following steps by the electronic device. In addition, the methods used to process data in different embodiments can be the same method or different methods. It should also be noted that any step in the embodiments of the present application can be independently executed by the electronic device, that is, when the electronic device executes any step in the following embodiments, it can be independent of the execution of other steps.

[0026] In the related art, the following two methods are usually used to implement different multi-person remote presentation systems:

[0027] 1. The video pictures including multiple videos of users are arranged in a reduced size in the same two-dimensional picture for transmission; or the background is removed by means of a machine vision algorithm, and then each separated user contour image is superimposed on a two-dimensional virtual space, and at the same time, the two-dimensional picture is rendered based on the two-dimensional virtual space and transmitted to the relevant screen.

[0028] 2. Based on the system color image and depth image data, combined with machine vision technology, the dynamic three-dimensional texture model of the user is reconstructed, and the reconstructed model of each user is input into the rendering engine. According to different virtual scene settings of the environmental parameters, the rendering engine can perform different light and shadow renderings, and then transmit the rendered picture to the user screen.

[0029] At the same time, in Method 1: Since the real scene spaces where each user is located are different, the scene light sources and their directions irradiated on each user are also different. Therefore, it is easy to cause the problem of visual inconsistency in the experience of presenting multiple users in the same two-dimensional virtual space for a video conference through Method 1; in Method 2: It is easily limited by problems such as too low sampling resolution of the three-dimensional sensor and too large amount of transmitted data, resulting in the problem of poor quality of the rendered portraits.

[0030] Based on the above problems, an embodiment of the present application provides an image processing method, which is applied to an image processing device. Refer to Figure 1 As shown, the method includes the following steps:

[0031] Step 101, obtain a two-dimensional image and a three-dimensional image of an image frame including an object to be rendered.

[0032] In an embodiment of the present application, the image processing device can be any electronic device with data processing capabilities.

[0033] Among them, the image processing device can obtain a two-dimensional image and a three-dimensional image of an image frame including an object to be rendered through an internal image acquisition module, or the image processing device can receive a two-dimensional image and a three-dimensional image of an image frame including an object to be rendered sent by other electronic devices.

[0034] In a feasible implementation manner, the image processing device can perform image acquisition on the object to be rendered at the same time to obtain a two-dimensional image and a three-dimensional image of an image frame including the object to be rendered; among them, the information presented by the object to be rendered in the two-dimensional image and the three-dimensional image, such as color, shape, etc., is exactly the same.

[0035] In another feasible implementation manner, the image processing device can perform image acquisition on the object to be rendered at two different times before and after respectively to obtain a two-dimensional image and a three-dimensional image of an image frame including the object to be rendered respectively; among them, the information presented by the object to be rendered in the two-dimensional image and the three-dimensional image may be the same or different.

[0036] It should be noted that the object to be rendered can be a person, a plant, an animal, etc. In the following other embodiments of the present application, the object to be rendered is taken as an example of a person for description; among them, the number of objects to be rendered included in the two-dimensional image and the three-dimensional image can be one, two or more.

[0037] In an embodiment of the present application, the image frames corresponding to the two-dimensional image and the three-dimensional image respectively include but are not limited to the object to be rendered. At the same time, the image sizes, presented colors, etc. of the two-dimensional image and the three-dimensional image are not limited in any way in the embodiment of the present application. And the two-dimensional image can be one image, or two or more images; correspondingly, the three-dimensional image can be one image, or two or more images.

[0038] It should be noted that the two-dimensional image is a planar image of an image frame that does not include depth information; the three-dimensional image is a stereoscopic image of an image frame with height, width, and depth.

[0039] Among them, the image screen includes the two-dimensional image and the three-dimensional image of the object to be rendered, which can be a two-dimensional color image and a three-dimensional color image. At the same time, the object to be rendered can be located in the foreground area, the middle ground area or the background area in the two-dimensional image and the three-dimensional image respectively; in the following other embodiments of the present application, it is described by taking the object to be rendered located in the foreground area of the two-dimensional image and the three-dimensional image as an example.

[0040] Step 102: Determine the three-dimensional geometric data in the three-dimensional image.

[0041] In the embodiment of the present application, the image processing device processes the three-dimensional image to obtain the three-dimensional geometric data in the three-dimensional image; among them, the three-dimensional geometric data can be the three-dimensional geometric features of the object to be rendered in the three-dimensional image.

[0042] In a feasible implementation manner, the three-dimensional geometric data can be the three-dimensional geometric triangle patch data associated with the object to be rendered in the three-dimensional image.

[0043] In another feasible implementation manner, the three-dimensional geometric data can be the three-dimensional geometric quadrilateral data obtained by performing geometric processing on the object to be rendered in the three-dimensional image.

[0044] Among them, the representation form of the three-dimensional geometric data can be data, text, code, etc., and the embodiments of the present application do not make any limitation on this.

[0045] It should be noted that before the image processing device determines the three-dimensional geometric data in the three-dimensional image, it can first perform texture feature filtering on the three-dimensional image, and then obtain the corresponding three-dimensional geometric data based on the three-dimensional image with the texture features filtered out.

[0046] Among them, the image processing device can process the three-dimensional image based on a machine vision algorithm to obtain the three-dimensional geometric data of the object to be rendered; among them, the machine vision algorithm is not limited in the embodiments of the present application.

[0047] Step 103: Based on the three-dimensional geometric data and the lighting parameters of the virtual environment, adjust the object to be rendered in the two-dimensional image screen to obtain a target image in which the image screen matches the virtual environment.

[0048] In the embodiment of the present application, the image processing device adjusts the image area where the object to be rendered is located in the two-dimensional image screen based on the three-dimensional geometric data and the lighting parameters of the virtual environment to obtain a target image in which the image screen matches the virtual environment.

[0049] Among them, the virtual environment can be set in advance or determined based on the attribute information of the object to be rendered. At the same time, the virtual environment can be the environmental information corresponding to a park venue, the environmental information corresponding to an office, or the environmental information corresponding to a classroom.

[0050] It should be noted that the light and shadow parameters of the virtual environment can refer to the light intensity, light incident direction, shadow information, etc. in the virtual environment.

[0051] In a feasible implementation, the image processing device adjusts the light and shadow parameters of each pixel in the image area where the object to be rendered is located in the two-dimensional image frame based on the determined three-dimensional geometric data and the light and shadow parameters of the virtual environment to obtain an intermediate image; and integrates and processes the intermediate image and the virtual image corresponding to the virtual environment according to a preset rule to obtain a target image; wherein, the image frame of the target image at least includes the object to be rendered after the light and shadow parameters are adjusted.

[0052] It should be noted that when both the two-dimensional image and the three-dimensional image are at least two or more images, based on the three-dimensional geometric data corresponding to each three-dimensional image and the light and shadow parameters of the virtual environment, the object to be rendered in each two-dimensional image frame is adjusted to obtain at least two target images whose image frames match the virtual environment; wherein, the light and shadow parameters of the virtual environment can change with the change of the image frame content of the three-dimensional image.

[0053] The image processing method provided by the embodiments of the present application, first, obtains a two-dimensional image and a three-dimensional image whose image frames include the object to be rendered; secondly, determines the three-dimensional geometric data in the three-dimensional image; finally, based on the three-dimensional geometric data and the light and shadow parameters of the virtual environment, adjusts the object to be rendered in the two-dimensional image frame to obtain a target image whose image frame matches the virtual environment; thus, by using the three-dimensional geometric data and the light and shadow parameters of the virtual environment to adjust the object to be rendered in the image frame to obtain a target image whose image frame matches the virtual environment, the quality and efficiency of image rendering can be improved.

[0054] Based on the foregoing embodiments, the embodiments of the present application further provide an image processing method, which is applied to an image processing device, referring to Figure 1 and Figure 2 as shown, the method includes the following steps:

[0055] Step 201: Perform region segmentation on the three-dimensional image to determine the spatial region occupied by the object to be rendered in the three-dimensional image.

[0056] In the embodiments of the present application, the image processing device performs region segmentation on the three-dimensional image to determine the spatial region occupied by the object to be rendered in the three-dimensional image.

[0057] Wherein, when the object to be rendered is a person, this spatial region is the image region occupied by the human body range of the object to be rendered in the three-dimensional image.

[0058] In a feasible implementation, the spatial region determined by the image processing device includes the head region and the limb region of the object to be rendered.

[0059] In another feasible implementation, the spatial region determined by the image processing device includes the head region and the hand region of the object to be rendered.

[0060] In yet another feasible implementation, the spatial region determined by the image processing device includes the entire body region of the object to be rendered.

[0061] It should be noted that in the embodiments of the present application, the image processing device first filters the texture features of the three-dimensional image to obtain an intermediate three-dimensional image, and then performs region segmentation on the intermediate three-dimensional image to determine the spatial region occupied by the object to be rendered in the three-dimensional image.

[0062] Step 202: Perform geometric processing on the spatial region to obtain three-dimensional geometric data.

[0063] In the embodiments of the present application, the image processing device performs geometric processing on the spatial region corresponding to the object to be rendered to obtain three-dimensional geometric data; wherein, the three-dimensional geometric data is the three-dimensional geometric feature of the object to be rendered.

[0064] It should be noted that the three-dimensional geometric data can be any data such as three-dimensional geometric triangle data, three-dimensional geometric quadrilateral data, and three-dimensional geometric polygon data.

[0065] Among them, geometric processing is to divide the spatial region according to a preset geometric shape.

[0066] It should be noted that in the embodiments of the present application, by determining the spatial region of the object to be rendered in the three-dimensional image, and then determining the three-dimensional geometric data based on this spatial region; in this way, the three-dimensional geometric features of the object to be rendered can be determined efficiently and accurately, and at the same time, only the three-dimensional geometric features are required to perform the subsequent related image rendering operations, which can reduce the amount of processing data for image rendering in the later stage, that is, the efficiency of image rendering can be further improved.

[0067] Correspondingly, the image processing device in the embodiments of the present application adjusts the object to be rendered in the two-dimensional image screen based on the three-dimensional geometric data and the light and shadow parameters of the virtual environment to obtain a target image that matches the virtual environment, that is, the image processing device executes step 103, and can also be implemented through the following steps 203 and 204:

[0068] Step 203: Based on the three-dimensional geometric data, correct the light and shadow parameters of the virtual environment to determine the light and shadow adjustment parameters.

[0069] In an embodiment of the present application, an image processing device corrects the light and shadow parameters of a virtual environment based on the determined three-dimensional geometric data, and further determines light and shadow adjustment parameters; wherein, the light and shadow adjustment parameters may be the same as or different from the light and shadow parameters of the virtual environment.

[0070] Among them, the image processing device may adjust the light intensity, light incident direction, light shadow position, length, etc. in the light and shadow parameters of the virtual environment based on the three-dimensional geometric data of the object to be rendered, that is, the three-dimensional geometric features, so as to obtain corresponding light and shadow adjustment parameters.

[0071] It should be noted that the light and shadow adjustment parameters include but are not limited to: light intensity adjustment parameters, light incident direction adjustment parameters, light shadow adjustment parameters, shadow length adjustment parameters, etc.

[0072] Step 204: Adjust the object to be rendered in the two-dimensional image frame based on the light and shadow adjustment parameters to obtain a target image.

[0073] In an embodiment of the present application, an image processing device adjusts the object to be rendered in the two-dimensional image frame based on the light and shadow adjustment parameters to obtain a target image; wherein, the target image at least includes the adjusted object to be rendered.

[0074] In a feasible implementation manner, the image processing device may adjust the light and shadow parameters of each pixel point in the area where the object to be rendered is located in the two-dimensional image frame based on the light and shadow adjustment parameters to obtain a target image.

[0075] It should be noted that in an embodiment of the present application, based on the three-dimensional geometric data of the object to be rendered, the light and shadow parameters of the virtual environment are adjusted to obtain light and shadow adjustment parameters, and then the object to be rendered in the two-dimensional image frame is adjusted based on the light and shadow adjustment parameters to obtain a target image that matches the virtual environment and includes the object to be rendered. In this way, based on the light and shadow adjustment parameters that match the three-dimensional geometric data of the object to be rendered, the object to be rendered in the two-dimensional image is adjusted to obtain a target image. In other words, on the basis of realizing the adjustment of the light and shadow parameters of the image frame in the two-dimensional image based on the three-dimensional geometric features in the three-dimensional image, the light and shadow coordination of the image frame in image rendering can be improved, and at the same time, the data processing amount in the image rendering process can be reduced, that is, the quality and efficiency of image rendering can be improved.

[0076] In a feasible implementation manner, step 204 may be implemented in the following manner of steps 204a to 204c (not shown in the figure):

[0077] Step 204a: Perform region segmentation on the two-dimensional image to determine the area to be rendered of the object to be rendered in the two-dimensional image.

[0078] In an embodiment of the present application, an image processing device performs region segmentation on a two-dimensional image to determine a region to be rendered of a rendering object in the two-dimensional image; wherein, when the rendering object is a person, portrait segmentation technology can be used to perform region segmentation on the two-dimensional image to determine the region to be rendered of the rendering object in the two-dimensional image.

[0079] Among them, the region to be rendered may be the image region occupied by the rendering object in the two-dimensional image. At the same time, the region to be rendered may be the head region, limb region or entire body region of the rendering object, etc.

[0080] It should be noted that, in the embodiments of the present application, the information presented by the rendering object in the two-dimensional image and the three-dimensional image is taken as an example for illustration.

[0081] Step 204b: Based on the light and shadow adjustment parameters, adjust the light and shadow parameters of each pixel point in the region to be rendered to obtain an intermediate image.

[0082] In an embodiment of the present application, an image processing device adjusts the light and shadow parameters of each pixel point in the region to be rendered based on the light and shadow adjustment parameters to obtain an intermediate image; wherein, the intermediate image is the image obtained by adjusting the region to be rendered corresponding to the rendering object in the two-dimensional image.

[0083] Among them, the image processing device adjusts the light and shadow parameters of each pixel point in the region to be rendered based on the light and shadow adjustment parameters, such as: light intensity, light incident direction, light shadow position and length, etc., to obtain an intermediate image corresponding to the region to be rendered.

[0084] In a feasible implementation manner, when the rendering object is a person, the image processing device weakens the light intensity of the head region of the rendering object in the region to be rendered based on the light and shadow adjustment parameters, and shifts the light shadow position of the head region to the left, etc.; at the same time, strengthens the light intensity of the limb region of the rendering object in the region to be rendered, and adjusts the angle of the light incident direction of the limb region to obtain an intermediate image corresponding to the region to be rendered.

[0085] In a feasible implementation manner, before the image processing device executes step 204b, the following step A1 (not shown in the figure) may also be executed:

[0086] Step A1: Obtain the attribute information of the rendering object.

[0087] In an embodiment of the present application, an image processing device obtains the attribute information of the rendering object; wherein, the attribute information is the characteristic information presented by the rendering object itself.

[0088] In a feasible implementation manner, the image processing device can extract information from the two-dimensional image and three-dimensional image of the image frame including the object to be rendered to obtain the attribute information of the object to be rendered.

[0089] In a feasible implementation manner, when the object to be rendered is a person, the attribute information of the object to be rendered includes but is not limited to: the gender of the object to be rendered, the age of the object to be rendered, the skin color of the object to be rendered, and the pose information of the object to be rendered in the two-dimensional image and / or three-dimensional image, etc.

[0090] In another feasible implementation manner, when the object to be rendered is a computer, the attribute information of the object to be rendered includes but is not limited to: the model of the object to be rendered, the color of the object to be rendered, etc.

[0091] Correspondingly, the image processing device adjusts the light and shadow parameters of each pixel point in the area to be rendered based on the light and shadow adjustment parameters to obtain an intermediate image. That is, when the image processing device executes step 204b, it can be implemented in the following manner of step A2:

[0092] Step A2: Based on the light and shadow adjustment parameters and the attribute information, adjust the light and shadow parameters of each pixel point in the area to be rendered to obtain an intermediate image.

[0093] In the embodiment of the present application, the image processing device adjusts the light and shadow parameters of each pixel point in the area to be rendered based on the light and shadow adjustment parameters and the attribute information to obtain an intermediate image.

[0094] Among them, the image processing device can first correct the light and shadow adjustment parameters based on the attribute information of the object to be rendered to obtain the corrected light and shadow adjustment parameters, and then adjust the light and shadow parameters of each pixel point in the area to be rendered based on the corrected light and shadow adjustment parameters to obtain an intermediate image.

[0095] It should be noted that in the embodiment of the present application, based on the three-dimensional geometric data, attribute information of the object to be rendered, and the light and shadow parameters of the virtual environment, the light and shadow parameters of each pixel point corresponding to the object to be rendered in the two-dimensional image are adjusted to obtain the target image. In this way, based on the light and shadow adjustment parameters that match both the three-dimensional geometric data and the attribute information of the object to be rendered, the object to be rendered in the two-dimensional image is adjusted to obtain the target image, which can further improve the light and shadow coordination of the image frame including the object to be rendered in image rendering.

[0096] Step 204c: Integrate the virtual image corresponding to the virtual environment and the intermediate image to obtain the target image.

[0097] In an embodiment of the present application, an image processing device integrates a virtual image corresponding to a virtual environment and an intermediate image to obtain a target image; wherein, the virtual image can be at least one virtual image obtained by the image processing device through image acquisition of the virtual environment. At the same time, the virtual image can be a three-dimensional image or a two-dimensional image. In the embodiments of the present application, the virtual image is taken as an example of a two-dimensional image for illustration.

[0098] Among them, the image processing device integrates and splices the virtual image corresponding to the virtual environment and the intermediate image according to a preset size to obtain a target image; the target image is an image capable of presenting a to-be-rendered object in the virtual environment.

[0099] It should be noted that in the embodiments of the present application, based on the three-dimensional geometric data of the to-be-rendered object and the lighting parameters of the virtual environment, the lighting parameters of each pixel point in the to-be-rendered area corresponding to the to-be-rendered object in the two-dimensional image are adjusted to obtain an intermediate image, and then a target image is obtained based on the intermediate image and the virtual image corresponding to the virtual environment. In this way, on the basis of improving the lighting coordination of the image in image rendering, the quality and efficiency of image rendering can be further improved.

[0100] In a feasible implementation manner, step 204c can be implemented in the following manner of step c1 to step c3 (not shown in the figure):

[0101] Step c1: In the virtual image, determine a preset area for fusing the to-be-rendered object.

[0102] In an embodiment of the present application, the image processing device determines a preset area for fusing the to-be-rendered object in the virtual image corresponding to the virtual scene; wherein, the preset area can be determined based on the attribute information of the to-be-rendered object or based on the environmental information of the virtual image.

[0103] In a feasible implementation manner, the preset area determined in the virtual image for fusing the to-be-rendered object can be an area having the same size and shape as the to-be-rendered area corresponding to the to-be-rendered object in the two-dimensional image.

[0104] In another feasible implementation manner, the preset area determined in the virtual image for fusing the to-be-rendered object can be similar in shape to the to-be-rendered area corresponding to the to-be-rendered object in the two-dimensional image but different in size.

[0105] Step c2: Adjust the size of the intermediate image to obtain a to-be-replaced image that matches the size of the preset area.

[0106] In an embodiment of the present application, an image processing device adjusts the size of an intermediate image to obtain a replacement image that matches the size of a preset area; wherein, the replacement image and the intermediate image present the same image content, but their sizes may be the same or different.

[0107] In a feasible implementation, the image processing device may proportionally enlarge or reduce the intermediate image based on the size of the preset area, so as to obtain a replacement image that matches the size of the preset area.

[0108] Step c3: In the virtual image, replace the preset area with the replacement image to obtain a target image.

[0109] In an embodiment of the present application, the image processing device replaces the preset area in the virtual image with the replacement image, that is, integrates the replacement image of the object to be rendered with adjusted light and shadow parameters into the virtual image, so as to obtain a target image that includes the object to be rendered with adjusted light and shadow parameters and the environmental information associated with the virtual environment in the image content.

[0110] In a feasible implementation, the image content includes a two-dimensional image and a three-dimensional image of the object to be rendered, which are respectively a two-dimensional image and a three-dimensional image of a staff member in an office; the image processing device first adjusts the light and shadow parameters of each pixel point in the portrait area of the staff member in the two-dimensional image based on the three-dimensional geometric features of the staff member in the three-dimensional image that can present the staff member in the office and present a three-dimensional effect, and the light and shadow parameters of the virtual environment (for example, the park scene at 12:00 noon), to obtain an intermediate image; secondly, obtain a virtual image corresponding to the virtual environment, that is, collect an image of the park scene at 12:00 noon to obtain a virtual image; thirdly, determine a preset area in the virtual image for fusing the object to be rendered; finally, adjust the intermediate image according to the preset size to replace the preset area in the virtual image, that is, splice and integrate the virtual image and the intermediate image to obtain a target image that can present the staff member in the park scene at 12:00 noon.

[0111] It should be noted that in an embodiment of the present application, in the virtual image corresponding to the virtual environment, a preset area for fusing the object to be rendered is determined, and then the size of the intermediate image of the object to be rendered with adjusted light and shadow parameters in the image content is adjusted based on the size of the preset area, and finally a target image is obtained based on the virtual image and the size-adjusted intermediate image. In this way, on the basis of improving the light and shadow coordination of the image content in image rendering, the accuracy of image rendering can be further improved.

[0112] The image processing method provided by the embodiment of the present application adjusts the object to be rendered in the two-dimensional image based on the light and shadow adjustment parameters matching the three-dimensional geometric data of the object to be rendered, so as to obtain a target image. In this way, on the basis of realizing the adjustment of the light and shadow parameters of the image in the two-dimensional image based on the three-dimensional geometric features in the three-dimensional image, the light and shadow coordination of the image in the image rendering can be improved, and at the same time, the data processing amount can be reduced, that is, the quality and efficiency of the image rendering can be improved.

[0113] Based on the foregoing embodiments, the embodiment of the present application further provides an image processing method. Referring to Figure 3 as shown, the method includes the following steps:

[0114] When the acquired image includes a two-dimensional image of the object to be rendered and the three-dimensional image is a video to be processed, that is, the image processing device can execute the following steps:

[0115] Step 301, acquire a video to be processed whose image includes an object to be rendered.

[0116] In the embodiment of the present application, the image processing device acquires a video to be processed whose image includes an object to be rendered; wherein, the image can refer to each frame of the video in the video to be processed.

[0117] For the related descriptions of the object to be rendered and the like, reference can be made to the descriptions in the foregoing embodiments, which will not be elaborated here.

[0118] In the embodiment of the present application, each frame of the video to be processed includes an object to be rendered; wherein, the pose information, color information, etc. of the object to be rendered in each frame of the video can be the same or different.

[0119] Correspondingly, the image processing device executes step 102 provided in the foregoing embodiment, that is, to determine the three-dimensional geometric data in the three-dimensional image, which can be implemented by the following step 302:

[0120] Step 302, determine at least one set of three-dimensional geometric data in the video to be processed.

[0121] In the embodiment of the present application, the image processing device determines at least one set of three-dimensional geometric data in the video to be processed; wherein, it can be to determine the three-dimensional geometric data in the three-dimensional image corresponding to each frame of the video in the video to be processed.

[0122] It should be noted that the related descriptions of the three-dimensional geometric data and the implementation manners of obtaining the three-dimensional geometric data can be referred to the descriptions in the foregoing embodiments, which will not be elaborated here.

[0123] Accordingly, the image processing device adjusts the object to be rendered in the two-dimensional image frame based on the three-dimensional geometric data and the light and shadow parameters of the virtual environment, and obtains a target image in which the image frame matches the virtual environment. This can be achieved through the following steps of 303:

[0124] Step 303: Based on at least one set of three-dimensional geometric data and the light and shadow parameters of the virtual environment, adjust the object to be rendered in each video frame of the video to be processed, and obtain a target video in which the video frame matches the virtual environment.

[0125] In the embodiment of the present application, the image processing device adjusts the light and shadow parameters of the object to be rendered in each video frame of the video to be processed based on at least one set of three-dimensional geometric data and the light and shadow parameters of the virtual environment, so as to obtain a target video in which the video frame matches the virtual environment; wherein, the light and shadow parameters of the virtual environment can change according to each video frame of the video to be processed.

[0126] It should be noted that in the embodiment of the present application, the light and shadow parameters of the virtual environment and the implementation manner of adjusting the object to be rendered in each video frame of the video to be processed can refer to that described in the above embodiment, and will not be elaborated here.

[0127] The image processing method provided by the embodiment of the present application includes: first, obtaining a video to be processed whose frame includes an object to be rendered; second, determining the three-dimensional geometric data of the video to be processed; finally, based on the three-dimensional geometric data and the light and shadow parameters of the virtual environment, adjusting the object to be rendered in each video frame of the video to be processed, and obtaining a target video in which the video frame matches the virtual environment; in this way, by using the three-dimensional geometric data and the light and shadow parameters of the virtual environment to adjust the object to be rendered in the video frame, a target video in which the video frame matches the virtual environment is obtained, which can improve the quality and efficiency of video frame rendering.

[0128] Based on this, the image processing method provided by the embodiment of the present application can be implemented in the actual image rendering process through the following steps:

[0129] Step 1: Collect a two-dimensional image and a three-dimensional image including the object to be rendered through a depth (Red Green Blue-Depth, RGB-D) camera.

[0130] Step 2: Remove the background from the two-dimensional image and the three-dimensional image respectively (here it is assumed that the object to be rendered is in the foreground area of the two-dimensional image and the three-dimensional image), that is, obtain a rendering area and a space area that only include the object to be rendered; wherein, the rendering area corresponds to the two-dimensional image, and the space area corresponds to the three-dimensional image; at the same time, the space area includes the three-dimensional geometric features corresponding to the object to be rendered in the three-dimensional image after removing the texture features.

[0131] Step 3: Input the data corresponding to the area to be rendered and the spatial area into a rendering engine for frame rendering. The rendering engine may include, but is not limited to, Open Graphics Library (OpenGL), Multimedia Programming Interface (DirectX), etc.

[0132] Step 4: In the rendering engine, a two-dimensional canvas may be placed in front of a virtual camera to render the image corresponding to the area to be rendered. Meanwhile, three-dimensional geometric features in the spatial area are placed between the two-dimensional canvas and the virtual camera. Then, the rendering engine processes the image corresponding to the area to be rendered and the three-dimensional geometric features in combination with pre-set virtual environment parameters (i.e., the lighting parameters of the virtual environment) to obtain a target image after rendering the image corresponding to the area to be rendered.

[0133] Step 5: Output the target image to a relevant display device for display.

[0134] It should be noted that in the embodiments of the present application, based on the superposition of a two-dimensional image and three-dimensional geometric data in front of the virtual camera of the rendering engine, frame rendering is performed in combination with the lighting parameters of the virtual scene. Among them, the two-dimensional image (which may be a two-dimensional color image) mainly provides image data including the object to be rendered, and the three-dimensional image mainly provides three-dimensional geometric data for correcting the lighting parameters of the virtual scene. Then, based on the rendering technology in the rendering engine, lighting effects under different light sources are generated and superimposed on the object to be rendered in the image frame of the two-dimensional image. In this way, the problem of inconsistent environmental lighting parameters that occurs when rendering only based on a two-dimensional image frame, and the problem of excessive data transmission caused by the three-dimensional image carrying texture feature data when rendering only based on a three-dimensional image can be solved, thereby improving the quality and efficiency of image rendering.

[0135] Meanwhile, in the case where the two-dimensional image and three-dimensional image including the object to be rendered are a video to be processed, frame rendering for the video frame can also be realized based on the above steps 1 to 5, that is, frame rendering is performed for each video frame in the video to be processed.

[0136] Based on the foregoing embodiments, the embodiments of the present application further provide an image processing apparatus 4. The image processing apparatus 4 may be applied to Figures 1 to 3 In a corresponding embodiment of an image processing method provided, refer to Figure 4 As shown, the image processing apparatus 4 includes:

[0137] An acquisition module 41, configured to acquire a two-dimensional image and a three-dimensional image of an image frame including an object to be rendered;

[0138] A determination module 42, configured to determine three-dimensional geometric data in the three-dimensional image;

[0139] An adjustment module 43, configured to adjust the object to be rendered in the two-dimensional image frame based on the three-dimensional geometric data and the lighting parameters of the virtual environment, so as to obtain a target image whose frame matches the virtual environment.

[0140] In other embodiments of the present application, the determination module 42 is further configured to perform region segmentation on the three-dimensional image to determine the spatial region occupied by the object to be rendered in the three-dimensional image; perform geometric processing on the spatial region to obtain the three-dimensional geometric data.

[0141] In other embodiments of the present application, the adjustment module 43 is further configured to correct the lighting parameters of the virtual environment based on the three-dimensional geometric data to determine lighting adjustment parameters; adjust the object to be rendered in the two-dimensional image frame based on the lighting adjustment parameters to obtain the target image.

[0142] In other embodiments of the present application, the adjustment module 43 is further configured to perform region segmentation on the two-dimensional image to determine the region to be rendered of the object to be rendered in the two-dimensional image; adjust the lighting parameters of each pixel point in the region to be rendered based on the lighting adjustment parameters to obtain an intermediate image; integrate the virtual image corresponding to the virtual environment and the intermediate image to obtain the target image.

[0143] In other embodiments of the present application, the acquisition module 41 is further configured to acquire the attribute information of the object to be rendered; the adjustment module 43 is further configured to adjust the lighting parameters of each pixel point in the region to be rendered based on the lighting adjustment parameters and the attribute information to obtain the intermediate image.

[0144] In other embodiments of the present application, the adjustment module 43 is further configured to determine a preset region for fusing the object to be rendered in the virtual image; adjust the size of the intermediate image to obtain a replacement image that matches the size of the preset region; in the virtual image, replace the preset region with the replacement image to obtain the target image.

[0145] In other embodiments of the present application, when the two-dimensional image and the three-dimensional image of the object to be rendered are a video to be processed, the determination module 42 is further configured to determine at least one set of three-dimensional geometric data in the video to be processed; the adjustment module 43 is further configured to adjust the object to be rendered in each frame of the video frame of the video to be processed based on the at least one set of three-dimensional geometric data and the lighting parameters of the virtual environment to obtain a target video whose video frame matches the virtual environment.

[0146] It should be noted that the specific implementation process of the steps executed by each module in this embodiment can be referred toFigures 1 to 3 The implementation process in the image processing method provided by the corresponding embodiment will not be elaborated here.

[0147] The image processing device provided by the embodiment of the present application adjusts the object to be rendered in the two-dimensional image based on the light and shadow adjustment parameters matching the three-dimensional geometric data of the object to be rendered, so as to obtain a target image. In this way, on the basis of realizing the adjustment of the light and shadow parameters of the image in the two-dimensional image based on the three-dimensional geometric features in the three-dimensional image, the light and shadow coordination of the image in the image rendering can be improved, and at the same time, the amount of data processing can be reduced, that is, the quality and efficiency of the image rendering can be improved.

[0148] Based on the foregoing embodiments, an embodiment of the present application further provides an electronic device 5, which can be applied to Figures 1 to 3 In an image processing method provided by the corresponding embodiment, refer to Figure 5 As shown, the electronic device 5 may include: a processor 51, a memory 52, and a communication bus 53, where:

[0149] The communication bus 53 is used to implement the communication connection between the processor 51 and the memory 52.

[0150] The processor 51 is configured to execute the program of the image processing method stored in the memory 52 to implement the image processing method provided by the corresponding embodiment with reference to Figures 1 to 3 The corresponding embodiment provides an image processing method.

[0151] The electronic device provided by the embodiment of the present application adjusts the object to be rendered in the two-dimensional image based on the light and shadow adjustment parameters matching the three-dimensional geometric data of the object to be rendered, so as to obtain a target image. In this way, on the basis of realizing the adjustment of the light and shadow parameters of the image in the two-dimensional image based on the three-dimensional geometric features in the three-dimensional image, the light and shadow coordination of the image in the image rendering can be improved, and at the same time, the amount of data processing can be reduced, that is, the quality and efficiency of the image rendering can be improved.

[0152] Based on the foregoing embodiments, an embodiment of the present application provides a computer-readable storage medium, and the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement as Figures 1 to 3 The image processing method provided by the corresponding embodiment.

[0153] It should be noted that the above computer-readable storage medium may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; it may also be various electronic devices including one or any combination of the above memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc.

[0154] It should be noted that in this text, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without more limitations, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including that element.

[0155] The serial numbers of the embodiments of the present application above are only for description and do not represent the superiority or inferiority of the embodiments.

[0156] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc) and includes several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0157] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in multiple blocks.

[0158] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in multiple blocks.

[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in multiple blocks.

[0160] The above are only the preferred embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structural or equivalent process transformation made using the content of the specification and drawings of the present application, or any direct or indirect application in other related technical fields, is equally included in the patent protection scope of the present application.

Claims

1. An image processing method, the method comprises: obtaining a two-dimensional image and a three-dimensional image of an image frame including an object to be rendered; performing region segmentation on the three-dimensional image to determine a spatial region occupied by the object to be rendered in the three-dimensional image; performing geometric processing on the spatial region to obtain three-dimensional geometric data; correcting the lighting parameters of a virtual environment based on the three-dimensional geometric data to determine lighting adjustment parameters; adjusting the object to be rendered in the two-dimensional image frame based on the lighting adjustment parameters to obtain a target image that matches the virtual environment; the target image is characterized as an image presenting the object to be rendered in the virtual environment.

2. The method according to claim 1, wherein the adjusting the object to be rendered in the two-dimensional image frame based on the lighting adjustment parameters to obtain a target image that matches the virtual environment comprises: performing region segmentation on the two-dimensional image to determine a region to be rendered of the object to be rendered in the two-dimensional image; adjusting the lighting parameters of each pixel point in the region to be rendered based on the lighting adjustment parameters to obtain an intermediate image; integrating the virtual image corresponding to the virtual environment and the intermediate image to obtain the target image.

3. The method according to claim 2, before the adjusting the lighting parameters of each pixel point in the region to be rendered based on the lighting adjustment parameters to obtain an intermediate image, the method further comprises: obtaining attribute information of the object to be rendered; the adjusting the lighting parameters of each pixel point in the region to be rendered based on the lighting adjustment parameters to obtain an intermediate image, comprising: adjusting the lighting parameters of each pixel point in the region to be rendered based on the lighting adjustment parameters and the attribute information to obtain the intermediate image.

4. The method according to claim 2, wherein the integrating the virtual image corresponding to the virtual environment and the intermediate image to obtain the target image comprises: determining a preset region in the virtual image for fusing the object to be rendered; adjusting the size of the intermediate image to obtain a replacement image that matches the size of the preset region; replacing the preset region with the replacement image in the virtual image to obtain the target image.

5. The method according to claim 1, when the two-dimensional image and the three-dimensional image of the object to be rendered are a video to be processed, the performing region segmentation on the three-dimensional image to determine a spatial region occupied by the object to be rendered in the three-dimensional image ； the performing geometric processing on the spatial region to obtain three-dimensional geometric data, comprising: determining at least one set of three-dimensional geometric data in the video to be processed; the correcting the lighting parameters of a virtual environment based on the three-dimensional geometric data to determine lighting adjustment parameters; adjusting the object to be rendered in the two-dimensional image frame based on the lighting adjustment parameters to obtain a target image that matches the virtual environment, comprising: Adjust the object to be rendered in each video frame of the video to be processed based on the at least one set of three-dimensional geometric data and the lighting parameters of the virtual environment, to obtain a target video in which the video frames match the virtual environment.

6. An image processing apparatus, the apparatus comprises: an acquisition module, configured to acquire an image frame including a two-dimensional image and a three-dimensional image of an object to be rendered; a determination module, configured to perform region segmentation on the three-dimensional image to determine a spatial region occupied by the object to be rendered in the three-dimensional image; perform geometric processing on the spatial region to obtain three-dimensional geometric data; an adjustment module, configured to correct the lighting parameters of the virtual environment based on the three-dimensional geometric data to determine lighting adjustment parameters; and adjust the object to be rendered in the two-dimensional image frame based on the lighting adjustment parameters to obtain a target image in which the image frame matches the virtual environment; the target image is characterized as an image presenting the object to be rendered in the virtual environment.

7. An electronic device, comprises: a processor, a memory, and a communication bus; the communication bus is configured to implement a communication connection between the processor and the memory; the processor is configured to execute a program in the memory to implement the image processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the image processing method according to any one of claims 1 to 5.

Images