Image processing method and device, electronic device, and computer-readable storage medium

Abstract

The embodiment of the present application provides an image processing method, including obtaining an image to be processed, constructing three-dimensional point cloud data of the image to be processed in a virtual three-dimensional space; obtaining projection parameters of a simulated light source; in the virtual three-dimensional space, based on the projection parameters of the simulated light source, performing shadow rendering processing on the three-dimensional point cloud data to obtain a target shadow image; performing fusion processing on the image to be processed and the target shadow image to obtain a final image containing shadow information. The embodiment of the present application also provides an image processing device, an electronic device, and a computer-readable storage medium.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to an image processing method and device, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the development of image processing technology, users can adjust the light and shadow effects of images through image processing devices, or superimpose different light and shadow effects on the collected images, so as to achieve the purpose of beautifying the image.

[0003] Currently, image processing devices usually adjust the brightness value of a single or local pixel in an image to simulate light and shadow effects, which cannot reflect the optical principles of the real world and has a high degree of distortion. Summary of the invention

[0004] Embodiments of the present application provide an image processing method and device, an electronic device, and a computer-readable storage medium.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, an image processing method is provided, comprising:

[0007] Acquire an image to be processed, and construct three-dimensional point cloud data of the image to be processed in a virtual three-dimensional space;

[0008] Get the projection parameters of the simulated light source;

[0009] In the virtual three-dimensional space, based on the projection parameters of the simulated light source, shadow rendering is performed on the three-dimensional point cloud data to obtain a target shadow image;

[0010] The image to be processed and the target shadow image are fused to obtain a final image containing shadow information.

[0011] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0012] A first acquisition unit, used for acquiring an image to be processed;

[0013] A point cloud construction unit, used for constructing three-dimensional point cloud data of the image to be processed in a virtual three-dimensional space;

[0014] A second acquisition unit, used to acquire projection parameters of the simulated light source;

[0015] A rendering processing unit, configured to perform shadow rendering processing on the three-dimensional point cloud data in the virtual three-dimensional space based on the projection parameters of the simulated light source to obtain a target shadow image;

[0016] The fusion processing unit is used to perform fusion processing on the image to be processed and the target shadow image to obtain a final image containing shadow information.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a processor and a memory for storing a computer program that can be run on the processor;

[0018] Wherein, the processor is used to execute the steps of the image processing method described in the first aspect when running the computer program.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the image processing method described in the first aspect.

[0020] The image processing method, device, electronic device and computer-readable storage medium provided by the embodiments of the present application can, specifically, construct three-dimensional point cloud data of the image to be processed in a virtual three-dimensional space, simulate a real lighting scene, and perform shadow rendering processing on the three-dimensional point cloud in the virtual three-dimensional space to obtain a target shadow image. Furthermore, an image with added light and shadow effects is obtained based on the shadow image and the image to be processed. In this way, the real lighting effect can be simulated, and the image content of the image to be processed can be shadow rendered in a three-dimensional scene, which reflects the real optical principle and increases the realism and layering of the light and shadow effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of an image processing method provided in an embodiment of the present application Figure 1 ;

[0022] Figure 2 A schematic diagram of an image processing method provided in an embodiment of the present application Figure 2 ;

[0023] Figure 3 A schematic diagram of an image processing method provided in an embodiment of the present application Figure 3 ;

[0024] Figure 4 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0025] Figure 5 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of the structural composition of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0029] In order to improve the authenticity of the light and shadow effects superimposed in the image, a convolutional neural network based on deep learning technology is usually used to superimpose light and shadow effects on a single static image. Specifically, the image processing device can first extract the brightness information of the image to be processed, and then input the brightness information of the image to be processed into the convolutional neural network to obtain the target brightness information of the image to be processed. Finally, the image processing device can merge the target brightness information with the color information of each pixel in the image to be processed to obtain the final image with superimposed light and shadow effects.

[0030] However, to train a convolutional neural network that can superimpose light and shadow effects, it is necessary to collect a large number of images with different light and shadow effects as sample images. In order to obtain sample images with ideal light and shadow effects, it is necessary to perform complex light and shadow arrangements on the shooting site before collecting sample images for network training. It can be seen that in the related art, in order to obtain a convolutional neural network model that can superimpose light and shadow effects, a large amount of manpower and material costs are required in the early stage, and the convolutional neural network model has large restrictions on application scenarios.

[0031] Based on this, the embodiment of the present application provides an image processing method, and the execution subject of the image processing method can be the image processing device provided by the embodiment of the present application, or an electronic device integrating the image processing device, and the image processing device here can be implemented in hardware or software. Among them, the electronic device can be a smart phone, a tablet computer, a personal calculator, a server or an industrial calculator, etc., and the implementation of the present application does not limit the type of electronic device.

[0032] Please refer to Figure 1 , Figure 1 A schematic diagram of the process of the image processing method provided in the embodiment of the present application is shown in FIG. Figure 1 As shown, the image processing method comprises the following steps:

[0033] Step 110: Acquire the image to be processed, and construct three-dimensional point cloud data of the image to be processed in a virtual three-dimensional space.

[0034] In the embodiment of the present application, the image to be processed may be an image acquired by the electronic device through a camera, an image stored in a local storage space of the electronic device, or an image downloaded from the network. The embodiment of the present application limits the source of the image to be processed.

[0035] After acquiring the image to be processed, the electronic device can perform three-dimensional reconstruction on all image contents in the image to be processed in a virtual three-dimensional space to obtain three-dimensional point cloud data of the image to be processed.

[0036] Specifically, the electronic device can obtain depth information of the image to be processed while obtaining the image to be processed, and construct three-dimensional point cloud data of the image to be processed through the two-dimensional image information and depth information of the image to be processed.

[0037] In a possible implementation, the electronic device may collect depth information of all photographed objects contained in the image to be processed based on the hardware structure. For example, when collecting the image to be processed, the electronic device may simultaneously collect depth information of the photographed object through a depth camera, and the depth camera may be a time of flight (ToF) camera or a binocular camera.

[0038] In another possible implementation, the electronic device may also obtain depth information of all photographed objects in the image to be processed through software calculation functions. For example, the electronic device may use a depth estimation model to perform depth estimation processing on the image to be processed to obtain depth information of each pixel in the image to be processed.

[0039] In the embodiment of the present application, the virtual three-dimensional space can be a virtual space created by an electronic device simulating a real three-dimensional space. The virtual three-dimensional space is used to imitate a real three-dimensional space scene, so that the user can add a virtual light source in the three-dimensional space according to the needs to simulate the lighting in the real scene to achieve a more realistic projection effect.

[0040] Step 120: Obtain projection parameters of the simulated light source.

[0041] Here, the simulated light source refers to a virtual spotlight that can be superimposed in the virtual three-dimensional space. By simulating the illumination of the virtual spotlight, the above three-dimensional point cloud data can be virtually projected.

[0042] In the embodiment of the present application, projection parameters refer to some parameters when simulating light sources to project light effects on three-dimensional point clouds. Here, projection parameters may include but are not limited to at least one of the number of simulated light sources, the light source position of the simulated light source, the light source irradiation direction of the simulated light source, and the light intensity.

[0043] The light source position may refer to the coordinate information of the simulated light source in the virtual three-dimensional space. The light source irradiation angle specifically refers to the angle between the light emitted by the simulated light source and the projection plane.

[0044] In the embodiment of the present application, the projection parameters of the simulated light source can be set by the user.

[0045] In a possible implementation, the electronic device may provide a parameter configuration interface for the user. When the user uses the electronic device to superimpose light and shadow effects on the image to be processed, the projection parameters of the simulated light source may be input through the parameter configuration interface. In this way, the electronic device may obtain the projection parameters of the simulated light source input by the user through the parameter configuration interface.

[0046] In another possible implementation, the electronic device may also provide the user with different light and shadow mode options, such as sunny light and shadow mode, cloudy light and shadow model, indoor light and shadow mode, outdoor light and shadow mode, etc. Here, the electronic device may pre-store projection parameters of simulated light sources corresponding to different light and shadow modes. In this way, the electronic device may determine the projection parameters corresponding to the light and shadow mode selected by the user.

[0047] Step 130: In the virtual three-dimensional space, based on the projection parameters of the simulated light source, shadow rendering is performed on the three-dimensional point cloud data to obtain a target shadow image.

[0048] In a real lighting scene, when light is blocked by an opaque object during propagation, a dark area is formed on the back of the blocking object. The dark area is projected onto another object to form a shadow area. Shadows can reflect the spatial relationship between objects in a real scene.

[0049] Based on this, the embodiment of the present application can restore the 3D point cloud data of the image to be processed, add a simulated light source in the virtual 3D space where the 3D point cloud data is located, and simulate the real lighting effect by simulating the light source. Based on this, the electronic device can perform shadow drawing on the 3D point cloud data to increase the realism and layering of the shadow image.

[0050] In an embodiment of the present application, the electronic device can construct a depth map of the three-dimensional point cloud data from the perspective of the simulated light source, and determine the display parameters of each pixel in the three-dimensional point cloud data based on the depth map to obtain a target shadow image.

[0051] Step 140: Fusing the image to be processed and the target shadow image to obtain a final image containing shadow information.

[0052] In the embodiment of the present application, after obtaining the target shadow image, the target shadow image can be fused with the image to be processed to obtain an image containing shadow information. Here, fusion of the target shadow image and the image to be processed can be performed by merging the display parameters of the same pixel points in the shadow image and the image to be processed.

[0053] It can be seen that in the image processing method provided by the embodiment of the present application, the electronic device can construct the three-dimensional point cloud data of the image to be processed in the virtual three-dimensional space, simulate the real lighting scene, and perform shadow rendering processing on the three-dimensional point cloud in the virtual three-dimensional space to obtain the target shadow image. Then, according to the shadow image and the image to be processed, an image with added light and shadow effects is obtained. In this way, the real lighting effect can be simulated, and the image content of the image to be processed can be shadow rendered in the three-dimensional scene, which reflects the real optical principle and increases the realism and layering of the light and shadow effects.

[0054] In one embodiment of the present application, refer to Figure 2 As shown, the above step 110 can be implemented by the following steps 1101 and 1102:

[0055] Step 1101: perform depth estimation processing on the image to be processed according to the depth estimation model to obtain depth information of each pixel in the image to be processed.

[0056] Among them, the depth estimation model is used to predict the depth between each pixel in the image and the image acquisition device.

[0057] In the embodiment of the present application, a pre-trained depth estimation model can be used to perform depth estimation processing on the image to be processed to obtain the depth information of each pixel in the image to be processed. The depth information of the pixel specifically refers to the distance between the pixel and the image acquisition device.

[0058] Here, the depth estimation model may specifically be a monocular depth estimation model. Monocular depth estimation refers to predicting the depth information of each pixel in an image using only one image, that is, the monocular depth estimation model can restore the depth results of all photographed objects in the image using only one static image.

[0059] The monocular depth estimation model is obtained by pre-training a large number of depth sample images. The training process of the monocular depth estimation model may include:

[0060] Obtain a depth sample image and label information corresponding to the depth sample image; the label information includes the depth information of each pixel in the depth sample image; use the monocular depth estimation model to be trained to perform depth prediction processing on the depth sample image to obtain a first output result; based on the first output result and the label information corresponding to the depth sample image, obtain a first difference value; based on the first difference value, train the monocular depth estimation model to be trained until the monocular depth estimation model to be trained meets the training conditions, and obtain the above-mentioned monocular depth estimation model. The training condition here can be that the similarity between the output result of the monocular depth estimation model to be trained and the label information meets a preset threshold.

[0061] It should be noted that the training data (depth sample images) of the monocular depth estimation model is easier to obtain than the training data of the convolutional neural network that can superimpose light and shadow effects. The depth sample images can be collected by a red, green, and blue (RGB) image acquisition device and a laser scanner.

[0062] Step 1102: Based on the depth information of each pixel and the image to be processed, perform three-dimensional reconstruction on the image to be processed to obtain three-dimensional point cloud data.

[0063] Specifically, the electronic device can determine the position coordinate information of each pixel point in the image to be processed in a two-dimensional plane based on the image to be processed. In this way, the electronic device can restore the three-dimensional point cloud data of the image to be processed based on the depth information of each pixel and the position coordinate information of each pixel in the two-dimensional plane.

[0064] It should be noted that the three-dimensional point cloud data includes a large number of feature points, and each feature point may correspond to a pixel point in the image to be processed.

[0065] In summary, the data processing method provided in the embodiment of the present application can obtain the depth information of the image to be processed through software-implemented calculations to avoid the hardware design cost of depth estimation. In addition, the embodiment of the present application can use a monocular depth estimation model to directly predict the depth of a single static image, avoiding the use of multiple images with different perspectives to restore the depth information, thereby expanding the application scenarios.

[0066] In one embodiment of the present application, refer to Figure 3 As shown in the flowchart, step 130 performs shadow rendering processing on the point cloud data in the virtual three-dimensional space based on the projection parameters of the simulated light source to obtain a target shadow image, which can be achieved by following the steps 1301 to 1303.

[0067] Step 1301: In a virtual three-dimensional space, based on projection parameters of a simulated light source, determine a depth map of the point cloud data under the perspective of the simulated light source.

[0068] In the embodiment of the present application, the electronic device can determine the simulated light source viewing angle based on parameters such as the light source position and the light source irradiation direction of the simulated light source, so that the electronic device constructs a depth map of the three-dimensional point cloud data from the simulated light source viewing angle.

[0069] Specifically, the electronic device can draw a depth map of the three-dimensional point cloud data from the perspective of the simulated light source, wherein each pixel in the depth map can correspond to a feature point in the three-dimensional point cloud data, and the depth value of each pixel in the depth map can represent the distance between the feature point in the three-dimensional point cloud data corresponding to the pixel and the simulated light source.

[0070] Step 1302: Based on the depth map, obtain a first feature point set and / or a second feature point set from the three-dimensional point cloud data; wherein the first feature point set is a set of unobstructed feature points in the three-dimensional point cloud data; and the second feature point set is a set of obstructed feature points in the three-dimensional point cloud data.

[0071] It should be noted that since the three-dimensional point cloud data is a set of points of a solid three-dimensional object, when simulating the light source to illuminate the three-dimensional point cloud data, there must be some feature points that are directly illuminated by the virtual light source, and some feature points cannot be illuminated by the virtual light source due to occlusion by other feature points.

[0072] In the embodiment of the present application, the depth map is a depth map obtained when the three-dimensional point cloud data is viewed from the perspective of a simulated light source. Therefore, the depth map only includes the depth values ​​of the feature points that are not blocked in the three-dimensional point cloud data.

[0073] Based on this, the electronic device can use the depth map to obtain unobstructed feature points in the three-dimensional point cloud data to obtain a first feature point set. Or the electronic device can use the depth map to obtain obstructed feature points in the three-dimensional point cloud data to obtain a second feature point set. In another possible implementation, the electronic device can also use the depth map to simultaneously obtain unobstructed feature points and obstructed feature points in the three-dimensional point cloud data to obtain a first feature point set and a second feature point set, respectively.

[0074] In an embodiment of the present application, the electronic device obtains the first feature point set and / or the second feature point set from the point cloud data based on the depth map, which can be achieved by the following steps:

[0075] From the three-dimensional point cloud data, searching for feature points included in the depth map to obtain a first feature point set;

[0076] and / or,

[0077] From the three-dimensional point cloud data, feature points not included in the depth map are searched to obtain a second feature point set.

[0078] After the above analysis, it can be known that the depth map only includes the unobstructed feature points in the 3D point cloud data. Therefore, the electronic device can determine whether the feature points in the 3D point cloud data appear in the depth map one by one. When it is determined that a feature point in the 3D point cloud appears in the depth map, it can be determined that the feature point is an unobstructed feature point. When a feature point in the 3D point cloud data does not appear in the depth map, it can be determined that the feature point is an obstructed feature point.

[0079] In this way, the electronic device traverses all feature points in the three-dimensional point cloud data and can obtain all feature points that are not blocked by other feature points and / or blocked feature points, thus obtaining the first feature point set and / or the second feature point set.

[0080] Step 1303: Generate a target shadow image based on the first feature point set and / or the second feature point set.

[0081] In the embodiment of the present application, the electronic device can obtain unobstructed feature points and obstructed feature points based on the first feature point set and / or the second feature point set.

[0082] In this way, the electronic device can set different display parameters for the feature points included in different feature point sets. For example, the electronic device can set the brightness value of the unobstructed feature points in the first feature point set to a preset brightness value, or set the color parameter of the obstructed feature points in the first feature point set to a predetermined shadow color, etc.

[0083] Exemplary, reference Figure 4 In the scene schematic diagram shown, the simulated light source 41 can be set at the upper left of the three-dimensional point cloud data 42. Therefore, the feature points of the three-dimensional point cloud data 42 facing the light source area are unobstructed feature points, and the right area 421 and area 422 of the three-dimensional point cloud data 42 are obstructed, so the feature points in area 421 and area 422 are obstructed points. In this way, the color parameters of area 421 and area 422 can be set to the predetermined shadow color.

[0084] Furthermore, after determining the display parameters based on each feature point, the electronic device generates a target shadow image based on the display parameters of each feature point.

[0085] In an embodiment of the present application, when the electronic device generates a target shadow image based on the first feature point set and / or the second feature point set, it can also determine the contour formed by the feature points to be rendered according to the positional relationship between the feature points in the feature point set. Then, the display parameters of the feature points located at the edge of the contour are blurred to avoid excessively sharp edges of the contour, so as to achieve a softer and more realistic light and shadow effect.

[0086] Here, the display parameters of the feature points of the contour edge are blurred, which can be achieved through a Gaussian blur filter.

[0087] For example, after obtaining the second feature point set, the electronic device can determine the shadow outline based on the positional relationship between the blocked feature points in the second feature point set, and perform fuzzy processing on the display parameters of the feature points at the edge of the shadow outline, so that the light effect at the shadow edge is softer.

[0088] In one embodiment of the present application, in step 130, shadow rendering processing is performed on the three-dimensional point cloud data based on the projection parameters of the simulated light source to obtain a target shadow image, including:

[0089] Based on the projection parameters of the simulated light source, the three-dimensional point cloud data is subjected to shadow rendering processing to obtain an initial shadow image;

[0090] The initial shadow image is filtered to obtain a target shadow image; the filtering process is used to eliminate interference information in the initial shadow image.

[0091] Here, the electronic device may perform shadow rendering processing on the three-dimensional point cloud data according to the method provided in the above embodiment to obtain an initial shadow image.

[0092] The initial shadow image is obtained by rendering the point cloud data, and there are gaps between feature points. Therefore, the occluded part of the real scene may be determined as unoccluded by the electronic device due to the gaps in the 3D point cloud data. Alternatively, the occluded part of the real scene may also be determined as occluded by the electronic device due to the gaps between the 3D point cloud data.

[0093] Thus, inside the initial shadow image, there may be a small number of pixel points whose pixel values ​​are greatly different from those of other pixel points.

[0094] For example, in the initial shadow image, there may be holes (a collection of large-difference pixel points in the form of points) or small cracks (a collection of large-difference pixel points in the form of strips) in the shadow part. Since these holes or small cracks do not belong to the shadow area, in order to avoid the influence of these holes and small cracks on the light and shadow effects, filtering processing can be used to eliminate the infected information such as these holes or small cracks and connect the shadow areas together.

[0095] In the embodiment of the present application, the filtering process may be to first perform a dilation operation on the initial shadow image and then perform an erosion operation. Here, the dilation operation may be used to eliminate small particle noise contained in the shadow area. In addition, the erosion operation may be used to eliminate some small holes or small cracks.

[0096] After filtering the initial shadow image, a target shadow image is obtained. At this point, interference information has been eliminated from the target shadow image through filtering, which can improve the light and shadow effect of the image.

[0097] In an embodiment of the present application, the number of simulated light sources may include multiple, that is, the user can set multiple virtual light sources in the virtual three-dimensional space to achieve a variety of light and shadow effects.

[0098] Here, when there are multiple simulated light sources, shadow rendering is performed on the three-dimensional point cloud data based on the projection parameters of the simulated light sources in step 130 to obtain a target shadow image, which can be achieved by the following steps:

[0099] Based on the projection parameters of multiple simulated light sources, shadow rendering processing is performed on the three-dimensional point cloud data respectively to obtain a shadow image to be fused corresponding to each simulated light source in the multiple simulated light sources;

[0100] A plurality of shadow images to be fused are fused to obtain a target shadow image.

[0101] It is understandable that when there are multiple superimposed virtual light sources, the electronic device can perform shadow rendering processing on the three-dimensional point cloud data according to the projection parameters of different virtual light sources, and obtain a shadow image to be fused corresponding to each virtual light source. The way in which the electronic device performs shadow rendering on each virtual light source is the same as in the above embodiment, and will not be repeated here.

[0102] Furthermore, after determining the shadow image to be fused corresponding to each simulated light source, the electronic device may perform fusion processing on the multiple shadow images to be fused to obtain a final target shadow image.

[0103] Specifically, the electronic device may determine the positional relationship between shadow areas and / or illumination areas in a plurality of shadow images to be fused, and adjust display parameters of pixels in the shadow areas or illumination areas to fuse the plurality of shadow images to be fused.

[0104] Exemplarily, if the shadow area in the shadow image to be fused 1 and the illuminated area in the shadow image to be fused 2 are the same area, the electronic device can increase the brightness value of the area to obtain the target shadow image. If the shadow area in the shadow image to be fused 1 and the shadow area in the shadow image to be fused 2 are the same area, the electronic device reduces the brightness value of the area to obtain the target shadow image.

[0105] It can be seen that in the embodiments of the present application, a variety of different simulated light sources can be superimposed on the image to be processed, thereby improving the application scenarios of light and shadow processing and the sense of hierarchy of light and shadow effects.

[0106] Based on the above embodiments, the present application provides an image processing device, which can be applied to the electronic device described above, such as Figure 5 As shown, the image processing device comprises:

[0107] A first acquisition unit 51, used to acquire an image to be processed;

[0108] A point cloud construction unit 52, used to construct three-dimensional point cloud data of the image to be processed in a virtual three-dimensional space;

[0109] A second acquisition unit 53, used to acquire projection parameters of the simulated light source;

[0110] A rendering processing unit 54 is used to perform shadow rendering processing on the three-dimensional point cloud data in the virtual three-dimensional space based on the projection parameters of the simulated light source to obtain a target shadow image;

[0111] The fusion processing unit 55 is used to perform fusion processing on the image to be processed and the target shadow image to obtain a final image containing shadow information.

[0112] In some embodiments of the present application, the point cloud construction unit 52 is specifically used to perform depth estimation processing on the image to be processed according to a depth estimation model to obtain depth information of each pixel in the image to be processed; the depth estimation model is used to predict the depth information between each pixel in the image and the image acquisition device; based on the depth information of each pixel and the image to be processed, the image to be processed is three-dimensionally reconstructed to obtain the three-dimensional point cloud data.

[0113] In some embodiments of the present application, the rendering processing unit 54 is used to determine a depth map of the three-dimensional point cloud data under the perspective of the simulated light source based on the projection parameters of the simulated light source; based on the depth map, obtain a first feature point set and / or a second feature point set from the three-dimensional point cloud data; the first feature point set is a set of unobstructed feature points in the three-dimensional point cloud data; the second feature point set is a set of obstructed feature points in the three-dimensional point cloud data; based on the first feature point set and / or the second feature point set, generate the target shadow image.

[0114] In some embodiments of the present application, the rendering processing unit 54 is further used to search for feature points included in the depth map from the three-dimensional point cloud data to obtain the first feature point set;

[0115] And / or, searching the three-dimensional point cloud data for feature points that are not included in the depth map to obtain the second feature point set.

[0116] In some embodiments of the present application, the rendering processing unit 54 is specifically used to perform shadow rendering processing on the three-dimensional point cloud data based on the projection parameters of the simulated light source to obtain an initial shadow image; perform filtering processing on the initial shadow image to obtain the target shadow image; and the filtering processing is used to eliminate interference information in the initial shadow image.

[0117] In the embodiment of the present application, the number of the simulated light sources includes a plurality;

[0118] The rendering processing unit 54 is specifically used to perform shadow rendering processing on the three-dimensional point cloud data based on the projection parameters of multiple simulated light sources, respectively, to obtain a shadow image to be fused corresponding to each of the multiple simulated light sources; and to perform fusion processing on the multiple shadow images to be fused to obtain the target shadow image.

[0119] In the embodiment of the present application, the projection parameters of the simulated light source include the light source position and / or the light source irradiation angle of the simulated light source.

[0120] It should be noted that the functional units in this embodiment may be integrated into one processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above integrated module may be implemented in the form of hardware or in the form of software functional modules.

[0121] If the integrated module is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0122] Based on the above embodiment, another embodiment of the present application further provides an electronic device, such as Figure 6 As shown, the electronic device proposed in the embodiment of the present application may include a processor 601 and a memory 602 storing instructions executable by the processor;

[0123] The processor 601 and the memory 602 are connected via a bus 603;

[0124] The processor 601, when running the computer program stored in the memory 602, may execute the following instructions:

[0125] Acquire an image to be processed, and construct three-dimensional point cloud data of the image to be processed in a virtual three-dimensional space;

[0126] Get the projection parameters of the simulated light source;

[0127] In the virtual three-dimensional space, based on the projection parameters of the simulated light source, shadow rendering is performed on the three-dimensional point cloud data to obtain a target shadow image;

[0128] The image to be processed and the target shadow image are fused to obtain a final image containing shadow information.

[0129] In some embodiments of the present application, the processor 601 may further execute the following instructions:

[0130] According to the depth estimation model, depth estimation processing is performed on the image to be processed to obtain depth information of each pixel in the image to be processed; the depth estimation model is used to predict the depth information between each pixel in the image and the image acquisition device;

[0131] Based on the depth information of each pixel and the image to be processed, three-dimensional reconstruction processing is performed on the image to be processed to obtain the three-dimensional point cloud data.

[0132] In some embodiments of the present application, the processor 601 may further execute the following instructions:

[0133] Determining a depth map of the three-dimensional point cloud data at a perspective of the simulated light source based on projection parameters of the simulated light source;

[0134] Based on the depth map, obtaining a first feature point set and / or a second feature point set from the three-dimensional point cloud data; the first feature point set is a set of unobstructed feature points in the three-dimensional point cloud data; the second feature point set is a set of obstructed feature points in the three-dimensional point cloud data;

[0135] The target shadow image is generated based on the first feature point set and / or the second feature point set.

[0136] In some embodiments of the present application, the processor 601 may further execute the following instructions:

[0137] Searching for feature points included in the depth map from the three-dimensional point cloud data to obtain the first feature point set;

[0138] and / or,

[0139] Feature points not included in the depth map are searched in the three-dimensional point cloud data to obtain the second feature point set.

[0140] In some embodiments of the present application, the processor 601 may further execute the following instructions:

[0141] Based on the projection parameters of the simulated light source, shadow rendering is performed on the three-dimensional point cloud data to obtain an initial shadow image;

[0142] The initial shadow image is subjected to filtering processing to obtain the target shadow image; the filtering processing is used to eliminate interference information in the initial shadow image.

[0143] In some embodiments of the present application, the number of the simulated light sources includes a plurality;

[0144] The processor 601 may further execute the following instructions:

[0145] Based on the projection parameters of multiple simulated light sources, shadow rendering processing is performed on the three-dimensional point cloud data respectively to obtain a shadow image to be fused corresponding to each simulated light source in the multiple simulated light sources;

[0146] A fusion process is performed on the multiple shadow images to be fused to obtain the target shadow image.

[0147] In the embodiment provided in the present application, the processor 601 may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), and a controller. It is understandable that for different devices, the electronic device used to implement the above-mentioned processor function may also be other, and the embodiment of the present application does not specifically limit it.

[0148] In practical applications, the memory 602 may be a volatile memory, such as RAM; or a non-volatile memory, such as ROM, flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or a combination of the above types of memories, and provide instructions and data to the processor 1101.

[0149] The present application also provides a computer storage medium, specifically a computer readable storage medium, on which computer instructions are stored. When the computer storage medium is located in an electronic device manufacturing device, the computer instructions are executed by a processor to implement any step of the above-mentioned image processing method in the present application.

[0150] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and the like; it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0151] It should be understood that the "one embodiment" or "an embodiment" or "an embodiment of the present application" or "the aforementioned embodiment" or "some embodiments" mentioned throughout the specification means that the target features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" or "an embodiment of the present application" or "the aforementioned embodiment" or "some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, the features, structures or characteristics of these targets can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence numbers of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0152] In the absence of special instructions, the detection device performs any step in the embodiment of the present application, and the processor of the detection device may perform the step. Unless otherwise specified, the embodiment of the present application does not limit the order in which the detection device performs the following steps. In addition, the method used for processing data in different embodiments may be the same method or different methods. It should also be noted that any step in the embodiment of the present application can be independently executed by the detection device, that is, when the detection device executes any step in the above embodiment, it can be independent of the execution of other steps.

[0153] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0154] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0155] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0156] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0157] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0158] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0159] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0160] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a detection device, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0161] In the embodiments of the present application, the descriptions of the same steps and the same contents in different embodiments can refer to each other. In the embodiments of the present application, the term "and" does not affect the sequence of the steps.

[0162] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, It is characterized in that include: Get the image to be processed; According to the monocular depth estimation model, depth estimation processing is performed on the image to be processed to obtain depth information of each pixel in the image to be processed; The monocular depth estimation model is used to predict the depth information between each pixel in the image and the image acquisition device based on an image; Based on the depth information of each pixel point and the image to be processed, performing three-dimensional reconstruction processing on the image to be processed to obtain three-dimensional point cloud data of the image to be processed in a virtual three-dimensional space; Get the projection parameters of the simulated light source; In the virtual three-dimensional space, based on the projection parameters of the simulated light source, shadow rendering is performed on the three-dimensional point cloud data to obtain a target shadow image; Performing fusion processing on the image to be processed and the target shadow image to obtain a final image containing shadow information; The step of performing shadow rendering processing on the three-dimensional point cloud data in the virtual three-dimensional space based on the projection parameters of the simulated light source to obtain a target shadow image includes: In the virtual three-dimensional space, based on the projection parameters of the simulated light source, a depth map of the three-dimensional point cloud data under the perspective of the simulated light source is determined; wherein the depth map is a depth map obtained when the three-dimensional point cloud data is viewed from the perspective of the simulated light source; and the depth map only includes depth values ​​of unobstructed feature points in the three-dimensional point cloud data; Based on the depth map, obtaining a first feature point set and / or a second feature point set from the three-dimensional point cloud data; wherein the first feature point set is a set of unobstructed feature points in the three-dimensional point cloud data; and the second feature point set is a set of obstructed feature points in the three-dimensional point cloud data; Different display parameters are set for the feature points included in the first feature point set and / or the second feature point set respectively, and the target shadow image is generated based on the display parameter of each feature point.

2. The method according to claim 1, It is characterized in that The acquiring a first feature point set and / or a second feature point set from the three-dimensional point cloud data based on the depth map includes: Searching for feature points included in the depth map from the three-dimensional point cloud data to obtain the first feature point set; and / or, Feature points not included in the depth map are searched in the three-dimensional point cloud data to obtain the second feature point set.

3. The method according to claim 1 or 2, It is characterized in that The step of performing shadow rendering processing on the three-dimensional point cloud data based on the projection parameters of the simulated light source to obtain a target shadow image further includes: Based on the projection parameters of the simulated light source, shadow rendering is performed on the three-dimensional point cloud data to obtain an initial shadow image; The initial shadow image is subjected to filtering processing to obtain the target shadow image; the filtering processing is used to eliminate interference information in the initial shadow image.

4. The method according to claim 1 or 2, It is characterized in that The number of the simulated light sources includes a plurality; The step of performing shadow rendering processing on the three-dimensional point cloud data based on the projection parameters of the simulated light source to obtain a target shadow image further includes: Based on the projection parameters of multiple simulated light sources, shadow rendering processing is performed on the three-dimensional point cloud data respectively to obtain a shadow image to be fused corresponding to each simulated light source in the multiple simulated light sources; A fusion process is performed on the multiple shadow images to be fused to obtain the target shadow image.

5. The method according to claim 1 or 2, It is characterized in that The projection parameters of the simulated light source include the light source position and / or the light source irradiation angle of the simulated light source.

6. An image processing device, It is characterized in that The device comprises: A first acquisition unit, used for acquiring an image to be processed; A point cloud construction unit is used to perform depth estimation processing on the image to be processed according to a monocular depth estimation model to obtain depth information of each pixel in the image to be processed; the monocular depth estimation model is used to predict the depth information between each pixel in the image and the image acquisition device based on one image; based on the depth information of each pixel and the image to be processed, perform three-dimensional reconstruction processing on the image to be processed to obtain three-dimensional point cloud data of the image to be processed in a virtual three-dimensional space; A second acquisition unit, used to acquire projection parameters of the simulated light source; A rendering processing unit, configured to determine, in the virtual three-dimensional space, a depth map of the three-dimensional point cloud data under the perspective of the simulated light source based on the projection parameters of the simulated light source; wherein the depth map is a depth map obtained when looking at the three-dimensional point cloud data from the perspective of the simulated light source; the depth map only includes depth values ​​of unobstructed feature points in the three-dimensional point cloud data; based on the depth map, obtain a first feature point set and / or a second feature point set from the three-dimensional point cloud data; wherein the first feature point set is a set of unobstructed feature points in the three-dimensional point cloud data; and the second feature point set is a set of obstructed feature points in the three-dimensional point cloud data; and set different display parameters for the feature points included in the first feature point set and / or the second feature point set, respectively, and generate a target shadow image based on the display parameters of each feature point; The fusion processing unit is used to perform fusion processing on the image to be processed and the target shadow image to obtain a final image containing shadow information.

7. An electronic device, It is characterized in that The electronic device comprises a processor and a memory for storing a computer program capable of running on the processor; Wherein, when the processor is used to run the computer program, it executes the steps of the image processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, It is characterized in that A computer program is stored thereon, and the computer program is executed by a processor to implement the steps of the image processing method according to any one of claims 1 to 5.

Images