Multi-weather outdoor RGBD image generation method, system and device and storage medium

Abstract

A multi-weather outdoor RGBD image generation method, system and device, and a storage medium, the method comprising: step S1: obtaining an indoor RGBD image, detecting a target object in the indoor RGBD image, and determining an illumination direction according to the target object; s2, determining the shape and position of a light spot according to the illumination direction; s3, adjusting the overall brightness of the target object and / or the background according to the weather mode; and S4, determining a brightness distribution form according to the depth value and the brightness of the target object and the shape of the light spot, intercepting a part on the brightness distribution form to generate the light spot, and superposing the light spot to the position. According to the invention, an outdoor scene image which better accords with reality can be generated.

Description

Background Art

[0002] In the training process of image recognition and machine learning models, the selection and quality of training samples are crucial factors, which are directly related to the final performance and generalization ability of the model. An ideal training dataset should contain rich and diverse samples to cover various scenarios that may be encountered in the actual application of the model. However, in actual operation, collecting training samples often faces many challenges, especially in scenarios with significant differences between indoor and outdoor environments.

[0003] For indoor scenes, it is relatively easy to collect a large amount of high-quality indoor RGBD image data because the environment is relatively stable, the lighting conditions are controllable, and the background elements are relatively simple. This convenience allows us to quickly build a large indoor RGBD image dataset to provide the model with rich indoor feature learning materials. However, when we turn our attention to outdoor scenes, the situation is quite different. The complexity of the outdoor environment far exceeds that of the indoor environment, including changeable weather conditions (such as sunny, cloudy, rainy, snowy, etc.), complex lighting changes, dynamic background elements (such as pedestrians, vehicles, animals and plants, etc.), and different seasonal changes, all of which greatly increase the difficulty of outdoor image data collection.

[0004] Due to the above reasons, obtaining sufficient quantity and quality of outdoor image samples has become a difficult task. This not only requires the collector to have professional skills and equipment, but also requires a lot of time and effort to adapt to the various uncertainties of the outdoor environment. Therefore, in practice, there are often sufficient indoor samples and scarce outdoor samples. This data imbalance directly leads to poor performance of the trained model in recognizing outdoor scenes and limited generalization ability.

[0005] To alleviate this problem, researchers have tried to use image conversion technology to convert indoor RGBD images into outdoor images, in order to expand the outdoor sample library without increasing the actual acquisition cost. However, although existing image conversion technology can achieve style transfer or scene transformation to a certain extent, it still has significant deficiencies in imitating the authenticity, detail richness and dynamic changes of outdoor scenes.

[0006] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0007] To this end, the present invention determines the lighting direction according to the target object, and then determines the shape and position of the light spot, adjusts the overall brightness according to the weather pattern, and then determines the brightness distribution form of the light spot, intercepts part of the brightness distribution form to generate the light spot, and superimposes it on the image, so that the generated image is more consistent with the image of a real outdoor scene.

[0008] In a first aspect, the present invention provides a method for generating a multi-weather outdoor RGBD image, characterized by comprising:

[0009] Step S1: obtaining an indoor RGBD image, detecting a target object in the indoor RGBD image, and determining a lighting direction according to the target object;

[0010] Step S2: determining the shape and position of the light spot according to the illumination direction;

[0011] Step S3: adjusting the overall brightness of the target object and / or background according to the weather pattern;

[0012] Step S4: determining a brightness distribution form according to the depth value, brightness and shape of the light spot of the target object, intercepting a portion of the brightness distribution form to generate the light spot, and superimposing it on the position.

[0013] Optionally, the multi-weather outdoor RGBD image generation method is characterized in that step S1 comprises:

[0014] Step S11: Acquire indoor RGBD image;

[0015] Step S12: Detecting a target object on the indoor RGBD image using a target detection model;

[0016] Step S13: reconstruct the target object in three dimensions, and determine the illumination direction according to the brightness distribution of the target object.

[0017] Optionally, the multi-weather outdoor RGBD image generation method is characterized in that step S2 comprises:

[0018] Step S21: determining the initial position of the light spot according to the shape of the target object and the illumination direction;

[0019] Step S22: determining the shape, position and number of the light spots according to the initial position and the geometric relationship of the target object.

[0020] Optionally, the multi-weather outdoor RGBD image generation method is characterized in that step S3 comprises:

[0021] Step S31: determining a weather mode; the weather mode includes at least one of sunny, cloudy and rainy days;

[0022] Step S32: determining the brightness ratio of the target object and the background according to the weather pattern;

[0023] Step S33: overall adjusting the brightness of the target object and / or the background according to the brightness ratio;

[0024] Step S34: performing brightness balance adjustment on the target object and / or the interior of the background.

[0025] Optionally, the multi-weather outdoor RGBD image generation method is characterized in that step S4 comprises:

[0026] Step S41: determining the brightness distribution shape of the light spot according to the brightness and depth value of the target object and the shape of the light spot;

[0027] Step S42: intercepting a portion in the brightness distribution form to determine the brightness distribution of the light spot;

[0028] Step S43: Generate the light spot, and superimpose the light spot on the position.

[0029] Optionally, the multi-weather outdoor RGBD image generation method is characterized in that step S42 comprises:

[0030] Step S421: determining a backup center point on the brightness distribution form according to the brightness of the target object; the backup center point includes at least two points;

[0031] Step S422: selecting at least two center points from the backup center points according to the illumination direction and the depth value of the target object;

[0032] Step S423: According to the shape of the light spot, determine the cut-off part with the central point as the center, and splice the cut-off part to obtain the brightness distribution of the light spot.

[0033] Optionally, the multi-weather outdoor RGBD image generation method is characterized in that step S43 comprises:

[0034] Step S431: generating the light spot;

[0035] Step S432: setting transparency in a gradient form according to the brightness of the light spot;

[0036] Step S433: superimposing the light spot to the position.

[0037] In a second aspect, the present invention provides a multi-weather outdoor RGBD image generation system, which is used to implement any of the multi-weather outdoor RGBD image generation methods described above, and is characterized by comprising:

[0038] An acquisition module, used to obtain an indoor RGBD image, detect a target object in the indoor RGBD image, and determine a lighting direction according to the target object;

[0039] A light spot module, used to determine the shape and position of the light spot according to the illumination direction;

[0040] An adjustment module, for adjusting the overall brightness of the target object and / or background according to weather patterns;

[0041] The superposition module is used to determine the brightness distribution form according to the depth value, brightness of the target object and the shape of the light spot, and to intercept a part of the brightness distribution form to generate the light spot, and to superimpose it on the position.

[0042] In a third aspect, the present invention provides a multi-weather outdoor RGBD image generation device, characterized in that it includes:

[0043] processor;

[0044] a memory storing executable instructions of the processor;

[0045] Wherein, the processor is configured to perform the steps of any of the aforementioned multi-weather outdoor RGBD image generation methods by executing the executable instructions.

[0046] In a fourth aspect, the present invention provides a computer-readable storage medium for storing a program, characterized in that when the program is executed, the steps of any of the aforementioned methods for generating multi-weather outdoor RGBD images are implemented.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The present invention determines the lighting direction according to the target object, and then determines the shape and position of the light spot, adjusts the overall brightness according to the weather pattern, and then determines the brightness distribution form of the light spot, intercepts part of the brightness distribution form to generate the light spot, and superimposes it on the image, so that the generated image is more consistent with the image of a real outdoor scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings in the following descriptions are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:

[0050] Figure 1 This is a flowchart of a method for generating a multi-weather outdoor RGBD image in an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of a brightness distribution form in an embodiment of the present invention;

[0052] Figure 3 A flowchart of a step of determining a lighting direction in an embodiment of the present invention;

[0053] Figure 4 This is a flow chart of steps for determining the shape and position of a light spot in an embodiment of the present invention;

[0054] Figure 5 A flow chart of the steps of adjusting according to the weather mode in an embodiment of the present invention;

[0055] Figure 6 This is a flow chart of the steps of intercepting and superimposing light spots in an embodiment of the present invention;

[0056] Figure 7 This is a flow chart of steps for determining the brightness distribution of a light spot in an embodiment of the present invention;

[0057] Figure 8 This is a flow chart of steps for superimposing light spots in an embodiment of the present invention;

[0058] Fig. 9 This is a schematic diagram of the structure of a multi-weather outdoor RGBD image generation system in an embodiment of the present invention;

[0059] Fig.10 is a schematic structural diagram of a multi-weather outdoor RGBD image generation device in an embodiment of the present invention; and

[0060] Fig.11 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0062] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0063] The embodiment of the present invention provides a multi-weather outdoor RGBD image generation method, which aims to solve the problems existing in the prior art.

[0064] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0065] The present invention determines the lighting direction according to the target object, and then determines the shape and position of the light spot, adjusts the overall brightness according to the weather pattern, and then determines the brightness distribution form of the light spot, intercepts part of the brightness distribution form to generate the light spot, and superimposes it on the image, so that the generated image is more consistent with the image of a real outdoor scene.

[0066] Figure 1 FIG. 1 is a flowchart of a method for generating a multi-weather outdoor RGBD image in an embodiment of the present invention. Figure 1 As shown, the steps of a multi-weather outdoor RGBD image generation method in an embodiment of the present invention include:

[0067] Step S1: obtaining an indoor RGBD image, detecting a target object in the indoor RGBD image, and determining a lighting direction according to the target object.

[0068] In this step, a clear indoor RGBD image is used as the starting point. This image should contain indoor scenes that need to simulate outdoor environments. In the obtained indoor RGBD image, image processing techniques (such as edge detection, contour recognition, etc.) are used to identify and locate key target objects. These objects may be furniture, people, windows, etc., which will be used to determine the lighting direction and generate light spots in subsequent steps. According to the position and orientation of the detected target objects (especially windows, light sources, etc.), the lighting direction in the indoor scene is inferred. This usually involves analyzing the shadows and brightness in the image to determine the approximate position and direction of the light source. The lighting direction can be divided into three types: backlight, front light, and side light. In this step, you can determine which type of lighting direction it belongs to, and the specific lighting angle requirements do not need to be very precise.

[0069] Step S2: determining the shape and position of the light spot according to the illumination direction.

[0070] In this step, the shape and position of the light spot are a pair of variables that affect each other. Taking the palm as an example, when the light is irradiated from the back of the palm, multiple divergent light spots will be formed in the gaps between the fingers; when the light is irradiated from the side, a circular light spot will be formed. Based on the direction of the light and the area of ​​the target object, the shape of the light spot and its position in the image can be determined.

[0071] Step S3: adjusting the overall brightness of the target object and / or background according to the weather pattern.

[0072] In this step, select the weather mode to be simulated (such as sunny, cloudy, rainy, etc.). Usually one weather is simulated at a time, and by continuously adjusting the weather mode, outdoor images of multiple weather modes and outdoor images of different brightness parameters in the same weather mode are obtained. According to the selected weather mode, adjust the overall brightness of the target object and / or background in the indoor RGBD image. For example, in sunny mode, the overall brightness may be increased to simulate a sunny environment; while in cloudy or rainy mode, the brightness may be reduced to reflect insufficient light. In the same weather mode, there are also different parameters to characterize different climate levels. For example, sunny days can include multiple levels, such as 5 levels, to characterize the light intensity on sunny days.

[0073] Step S4: determining a brightness distribution form according to the depth value, brightness and shape of the light spot of the target object, intercepting a portion of the brightness distribution form to generate the light spot, and superimposing it on the position.

[0074] In this step, the brightness distribution of different light spot types is different, and the brightness distribution of light spots at different depth values ​​is also different. The brightness of the target object directly affects the brightness value of the light spot. The brightness distribution of the light spot is usually uneven, with the center being brighter and the edge being darker. The brightness distribution of the light spot can be in various forms, Figure 2 A Gaussian distribution of brightness is shown in FIG. After the brightness distribution is determined, the brightness of a portion of the area is cut off from the brightness distribution as the brightness distribution of the light spot. Figure 2 The area [100,255] intercepted in the image is used as the brightness distribution of the light spot. The intercepted light spot area should have clear boundaries and natural brightness transitions. Finally, the generated light spot is superimposed on the previously determined image position. This usually involves pixel-level operations on the image to ensure that the light spot blends seamlessly with the surrounding environment. During the superposition process, the brightness, color and other parameters of the light spot also need to be adjusted to make it more coordinated with the photo.

[0075] Figure 3 FIG. 1 is a flow chart of a step of determining a lighting direction in an embodiment of the present invention. Figure 3 As shown, in an embodiment of the present invention, a step of determining a light direction includes:

[0076] Step S11: Acquire an indoor RGBD image.

[0077] In this step, obtain indoor images containing color information and depth information to provide basic data for subsequent target detection and 3D reconstruction. Use an RGBD camera or related equipment to shoot in an indoor environment. Ensure that the distance, angle, and lighting conditions between the camera and the target object are appropriate to obtain high-quality RGBD images.

[0078] Step S12: Detect the target object on the indoor RGBD image using a target detection model.

[0079] In this step, the target object in the image is identified and located, providing key information for subsequent 3D reconstruction and illumination direction determination. Select a suitable target detection model, such as YOLO, SSD, Faster R-CNN and other models based on deep learning. These models have been trained to recognize a variety of target objects. Input the RGBD image into the target detection model, and the model will output information such as the category, location, and bounding box of the target object. Based on the output of the model, mark the location and boundaries of the target object on the image.

[0080] Step S13: reconstruct the target object in three dimensions, and determine the illumination direction according to the brightness distribution of the target object.

[0081] In this step, the three-dimensional structure of the target object is obtained through three-dimensional reconstruction, and the illumination direction is inferred based on the brightness distribution. The target object is reconstructed into a three-dimensional point cloud using depth information. This usually involves converting the depth image into point cloud data, and performing filtering, denoising, and registration. A three-dimensional model of the target object is constructed based on the point cloud data, and techniques such as triangulation and surface reconstruction can be used. The brightness distribution of the target object in the image is analyzed. The brightness distribution is usually affected by the illumination direction and the surface material of the object. Using an illumination model or related algorithms, combined with the three-dimensional model of the target object and the brightness distribution information, the illumination direction is inferred. This may require consideration of multiple factors, such as the type of light source, the orientation of the object, and the reflective characteristics of the surface. Based on the inference results, the illumination direction is determined and used as a reference for subsequent image processing or scene rendering.

[0082] This embodiment can obtain the target object in the indoor RGBD image, and perform 3D reconstruction and determine the illumination direction thereof. Such information will provide key support for the subsequent generation of multi-weather outdoor RGBD images.

[0083] Figure 4 FIG. 1 is a flow chart of steps for determining the shape and position of a light spot in an embodiment of the present invention. Figure 4 As shown, in an embodiment of the present invention, a step of determining the shape and position of a light spot includes:

[0084] Step S21: determining an initial position of the light spot according to the shape of the target object and the illumination direction.

[0085] In this step, the shape of the target object is analyzed, especially the part of its surface facing the light source. Combined with the direction of the light, the position where the light may directly illuminate the target object is inferred. This position is used as the initial position of the light spot. Usually, this position is located on the side of the target object facing the light source and has a relatively high brightness.

[0086] Step S22: determining the shape, position and number of the light spots according to the initial position and the geometric relationship of the target object.

[0087] In this step, the possible shape of the light spot is inferred based on the surface material and reflectivity of the target object, as well as the type and intensity of the light source. For example, if the target object is a smooth plane, the light spot may be circular or elliptical; if the target object has a textured or uneven surface, the shape of the light spot may be more complex. Based on the initial position, the position of the light spot is fine-tuned according to the geometry of the target object and the projection angle of the light source. Ensure that the light spot can accurately reflect the illumination of the light on the target object. Determine the number of light spots to be generated based on the number and distribution of target objects.

[0088] This embodiment can generate a more realistic and lifelike light spot effect, providing a basis for subsequent image processing tasks.

[0089] Figure 5 FIG. 1 is a flow chart of the steps of adjusting according to the weather mode in an embodiment of the present invention. Figure 5 As shown, in an embodiment of the present invention, a step of adjusting according to the weather mode includes:

[0090] Step S31: Determine the weather mode.

[0091] In this step, the weather mode includes at least one of sunny, cloudy, and rainy. This step specifies the weather conditions to be simulated. These conditions may include sunny (sunny, high brightness), cloudy (soft light, moderate brightness but may have partial shadows), rainy (dim light, low brightness, may be accompanied by rain effect), etc.

[0092] Step S32: Determine the brightness ratio of the target object and the background according to the weather pattern.

[0093] In this step, after the weather mode is determined, the brightness ratio between the target object (such as furniture, people, etc.) and the background needs to be set based on the lighting characteristics of this weather mode. For example, in sunny mode, the target object may be brighter than the background because the sun will shine directly on them; while in cloudy mode, the brightness of the target and the background may be closer because the overall light is darker.

[0094] Step S33: overall adjusting the brightness of the target object and / or the background according to the brightness ratio.

[0095] In this step, the brightness of the target object and / or background is adjusted overall according to the brightness ratio determined in step S32. This usually involves adding and subtracting the brightness value of each pixel in the image to achieve the desired brightness effect.

[0096] Step S34: performing brightness balance adjustment on the target object and / or the interior of the background.

[0097] In this step, after the overall brightness adjustment, the target object or background needs to be adjusted for uneven brightness. For example, some parts may be too bright or too dark, resulting in poor visual effects. Therefore, in this step, a brightness equalization algorithm is needed to further adjust the brightness of the target object and / or the background to adjust the overall lighting conditions from indoors to outdoors, ensuring a more uniform and natural brightness distribution for the entire image.

[0098] This embodiment can adjust the brightness of the target object and the background in the indoor RGBD image according to the selected weather mode, so as to simulate outdoor scenes under different weather conditions.

[0099] Figure 6 FIG. 1 is a flow chart of the steps of intercepting and superimposing light spots in an embodiment of the present invention. Figure 6 As shown, in an embodiment of the present invention, a step of intercepting light spots and superimposing light spots includes:

[0100] Step S41: determining the brightness distribution shape of the light spot according to the brightness and depth value of the target object and the shape of the light spot.

[0101] In this step, the depth values ​​of the spot area and the surrounding area are analyzed. The morphology of the spot is analyzed, including its size, shape and edge features. There are many types of brightness distribution, including at least one of Gaussian distribution, exponential distribution, Poisson distribution, parabolic distribution, power law distribution, Rayleigh distribution, lognormal distribution, Bessel distribution and mixed distribution. Different spot shapes have different brightness distribution shapes, and different depth values ​​will also affect the brightness distribution shape. The brightness of the target object can determine the longitudinal position of the brightness distribution shape, so the brightness distribution shape can be determined according to the brightness, depth value and spot shape of the target object.

[0102] Step S42: intercepting a portion in the brightness distribution form to determine the brightness distribution of the light spot.

[0103] In this step, a suitable interception area is determined in the brightness distribution form according to the actual size and shape of the light spot, ensuring that the interception area can completely contain the main brightness characteristics of the light spot while avoiding excessive background noise or irrelevant information.

[0104] Step S43: Generate the light spot, and superimpose the light spot on the position.

[0105] In this step, a spot image matching the spot shape and size is generated based on the brightness distribution of the intercepted area. The generated spot image is superimposed on the specified position of the original image. During the superposition process, it may be necessary to adjust the brightness, contrast and other parameters of the spot to ensure its integration and naturalness with the surrounding environment.

[0106] Figure 7 FIG. 1 is a flow chart of the steps of determining the brightness distribution of a light spot in an embodiment of the present invention. Figure 7 As shown, in an embodiment of the present invention, a step of determining the brightness distribution of a light spot includes:

[0107] Step S421: determining a backup center point on the brightness distribution pattern according to the brightness of the target object.

[0108] In this step, the brightness distribution of the target object is analyzed, especially those areas with higher brightness. By determining the brightest point in these areas, the brightness of the backup center point can be calculated according to the conversion coefficient. The conversion coefficient is a value greater than 1 and can be set according to specific needs. The backup center point includes at least two points. Figure 2 For example, in the Gaussian distribution in Figure 1, except for the high point, any brightness has two corresponding points, and these two points are the spare center points. For some brightness distribution forms, such as polynomial distribution, there can be three or more spare center points. The spare center points make the light distribution of the light spot have more possibilities.

[0109] Step S422: selecting at least two center points from the backup center points according to the illumination direction and the depth value of the target object.

[0110] In this step, the relative position relationship between the backup center point and the light source is analyzed in combination with the illumination direction and the depth value of the target object. A more suitable center point can be selected from the backup center points. The brightness distribution around multiple different backup center points is different. Different brightness distributions indicate different illumination directions, so the center point can be determined by the illumination direction.

[0111] Step S423: According to the shape of the light spot, determine the cut-off part with the central point as the center, and splice the cut-off part to obtain the brightness distribution of the light spot.

[0112] In this step, two or more suitable interception areas are determined based on the shape of the light spot with the selected center point as the center. Multiple interception parts are intercepted from the brightness distribution form, and then the multiple interception parts are spliced ​​into a continuous whole as the brightness distribution of the light spot. Figure 2 For example, Figure 2 The intercepted part in is [100,255], which is used as the first intercepted part, and [555,666] is intercepted as the second intercepted part ( Figure 2 (not shown), the two parts [100,255] and [555,666] are spliced ​​to obtain the brightness distribution of the light spot. The deviation of the brightness values ​​of the edges of different intercepted parts does not exceed the preset ratio a. For example, the brightness range of the first intercepted area is [100,125], and the brightness range of the second intercepted area is [130,150]. The values ​​of the brightness of the two are close to 125 and 130. Then, taking 125 as the benchmark, the deviation between 125 and 130 does not exceed the preset ratio a, that is, (130-125) / 125. <a。

[0113] This embodiment can accurately extract the brightness distribution of the light spot from the brightness distribution form, providing a basis for subsequent light spot generation and superposition.

[0114] Figure 8 FIG. 1 is a flow chart of the steps of superimposing light spots in an embodiment of the present invention. Figure 8 As shown, a step of superimposing light spots in an embodiment of the present invention includes:

[0115] Step S431: generating the light spot.

[0116] In this step, create a new image layer whose size and resolution should match the original image. Draw the light spot on the new image layer according to the brightness distribution of the light spot. Brighter colors can be used for areas with higher brightness, while darker colors can be used for areas with lower brightness. Make sure that the shape, size, and position of the light spot are consistent with the parameters determined previously. When generating the light spot, ensure that the color transition is natural and avoid obvious color discontinuities or mutations.

[0117] Step S432: setting transparency in a gradient according to the brightness of the light spot.

[0118] In this step, the brightness distribution of the light spot is analyzed to determine the central area with higher brightness and the edge area with lower brightness. A higher transparency (close to opaque) is set for the central area, while a lower transparency (close to completely transparent) is set for the edge area. A smooth transition zone can be set between the central area and the edge area to ensure that the change in transparency is continuous. Use image processing techniques (such as alpha channel adjustment) to apply the transparency gradient to the light spot image. The transparency setting should take into account the overall brightness and shape of the light spot to ensure that the superimposed effect is natural and realistic.

[0119] Step S433: superimposing the light spot to the position.

[0120] In this step, the spot image is superimposed on the specified position of the original image using image processing technology (such as image synthesis). During the superposition process, ensure that the transparency gradient of the spot is correctly integrated with the pixel value of the original image to avoid obvious stitching marks or color differences.

[0121] This embodiment can generate a light spot effect that is coordinated with the original image and naturally integrated.

[0122] Fig. 9 FIG. 1 is a schematic diagram of a multi-weather outdoor RGBD image generation system according to an embodiment of the present invention. Fig. 9 As shown, a multi-weather outdoor RGBD image generation system in an embodiment of the present invention includes:

[0123] An acquisition module, used to obtain an indoor RGBD image, detect a target object in the indoor RGBD image, and determine a lighting direction according to the target object;

[0124] A light spot module, used to determine the shape and position of the light spot according to the illumination direction;

[0125] An adjustment module, for adjusting the overall brightness of the target object and / or background according to weather patterns;

[0126] The superposition module is used to determine the brightness distribution form according to the depth value, brightness of the target object and the shape of the light spot, and to intercept a part of the brightness distribution form to generate the light spot, and to superimpose it on the position.

[0127] This embodiment determines the lighting direction according to the target object, and then determines the shape and position of the light spot, adjusts the overall brightness according to the weather pattern, and then determines the brightness distribution form of the light spot, intercepts a part of the brightness distribution form to generate a light spot, and superimposes it on the image, so that the generated image is more consistent with the image of a real outdoor scene.

[0128] The embodiment of the present invention also provides a multi-weather outdoor RGBD image generation device, including a processor and a memory, wherein executable instructions of the processor are stored. The processor is configured to execute the steps of a multi-weather outdoor RGBD image generation method by executing the executable instructions.

[0129] As mentioned above, this embodiment determines the lighting direction according to the target object, then determines the shape and position of the light spot, adjusts the overall brightness according to the weather pattern, and then determines the brightness distribution form of the light spot, intercepts part of the brightness distribution form to generate a light spot, and superimposes it on the image, so that the generated image is more consistent with the image of a real outdoor scene.

[0130] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0131] Fig.10 Schematic diagram of the structure of a multi-weather outdoor RGBD image generation device in an embodiment of the present invention. Fig.10 The electronic device 600 according to this embodiment of the present invention is described. Fig.10 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0132] like Fig.10As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0133] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps of various exemplary embodiments of the present invention described in the above-mentioned multi-weather outdoor RGBD image generation method section of this specification. For example, the processing unit 610 can execute the following steps: Figure 1 Follow the steps shown in .

[0134] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0135] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a grid environment.

[0136] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0137] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more grids (e.g., a local area network (LAN), a wide area network (WAN), and / or a public grid, such as the Internet) through a grid adapter 660. The grid adapter 660 may communicate with other modules of the electronic device 600 through the bus 630. It should be understood that although Fig.10Not shown, other hardware and / or software modules may be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0138] In an embodiment of the present invention, a computer-readable storage medium is also provided for storing a program, and when the program is executed, the steps of a multi-weather outdoor RGBD image generation method are implemented. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above multi-weather outdoor RGBD image generation method section of this specification.

[0139] As shown above, this embodiment determines the lighting direction according to the target object, and then determines the shape and position of the light spot, adjusts the overall brightness according to the weather pattern, and then determines the brightness distribution form of the light spot, intercepts part of the brightness distribution form to generate a light spot, and superimposes it on the image, so that the generated image is more consistent with the image of a real outdoor scene.

[0140] Fig.11 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Fig.11 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0141] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0142] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0143] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of grid, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0144] This embodiment determines the lighting direction according to the target object, and then determines the shape and position of the light spot, adjusts the overall brightness according to the weather pattern, and then determines the brightness distribution form of the light spot, intercepts a part of the brightness distribution form to generate a light spot, and superimposes it on the image, so that the generated image is more consistent with the image of a real outdoor scene.

[0145] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in this article, but will comply with the widest range consistent with the principles and novel features disclosed herein.

[0146] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for generating multi-weather outdoor RGBD images, characterized in that: include: Step S1: obtaining an indoor RGBD image, detecting a target object in the indoor RGBD image, and determining a lighting direction according to the target object; Step S2: determining the shape and position of the light spot according to the illumination direction; Step S3: adjusting the overall brightness of the target object and / or background according to the weather pattern; Step S4: determining a brightness distribution form according to the depth value, brightness and shape of the light spot of the target object, intercepting a portion of the brightness distribution form to generate the light spot, and superimposing it on the position.

2. The method for generating multi-weather outdoor RGBD images according to claim 1, characterized in that: Step S1 includes: Step S11: Acquire indoor RGBD image; Step S12: Detecting a target object on the indoor RGBD image using a target detection model; Step S13: reconstruct the target object in three dimensions, and determine the illumination direction according to the brightness distribution of the target object.

3. The method for generating multi-weather outdoor RGBD images according to claim 1, characterized in that: Step S2 includes: Step S21: determining the initial position of the light spot according to the shape of the target object and the illumination direction; Step S22: determining the shape, position and number of the light spots according to the initial position and the geometric relationship of the target object.

4. The method for generating multi-weather outdoor RGBD images according to claim 1, characterized in that: Step S3 includes: Step S31: determining a weather mode; the weather mode includes at least one of sunny, cloudy and rainy days; Step S32: determining the brightness ratio of the target object and the background according to the weather pattern; Step S33: overall adjusting the brightness of the target object and / or the background according to the brightness ratio; Step S34: performing brightness balance adjustment on the target object and / or the interior of the background.

5. The method for generating multi-weather outdoor RGBD images according to claim 1, characterized in that: Step S4 includes: Step S41: determining the brightness distribution shape of the light spot according to the brightness and depth value of the target object and the shape of the light spot; Step S42: intercepting a portion in the brightness distribution form to determine the brightness distribution of the light spot; Step S43: Generate the light spot, and superimpose the light spot on the position.

6. The method for generating multi-weather outdoor RGBD images according to claim 5, characterized in that: Step S42 includes: Step S421: determining a backup center point on the brightness distribution form according to the brightness of the target object; the backup center point includes at least two points; Step S422: selecting at least two center points from the backup center points according to the illumination direction and the depth value of the target object; Step S423: According to the shape of the light spot, determine the cut-off part with the central point as the center, and splice the cut-off part to obtain the brightness distribution of the light spot.

7. The method for generating multi-weather outdoor RGBD images according to claim 5, characterized in that: Step S43 includes: Step S431: generating the light spot; Step S432: setting transparency in a gradient form according to the brightness of the light spot; Step S433: superimposing the light spot to the position.

8. A multi-weather outdoor RGBD image generation system, used to implement the multi-weather outdoor RGBD image generation method according to any one of claims 1 to 7, characterized in that: include: An acquisition module, used to obtain an indoor RGBD image, detect a target object in the indoor RGBD image, and determine a lighting direction according to the target object; A light spot module, used to determine the shape and position of the light spot according to the illumination direction; An adjustment module, for adjusting the overall brightness of the target object and / or background according to weather patterns; The superposition module is used to determine the brightness distribution form according to the depth value, brightness of the target object and the shape of the light spot, and to intercept a part of the brightness distribution form to generate the light spot, and to superimpose it on the position.

9. A multi-weather outdoor RGBD image generation device, characterized in that: include: processor; a memory storing executable instructions of the processor; Wherein, the processor is configured to perform the steps of the multi-weather outdoor RGBD image generation method described in any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed, the steps of the multi-weather outdoor RGBD image generation method described in any one of claims 1 to 7 are implemented.

Images