Indoor shadow removal dataset generation method based on path tracing

Abstract

The present invention discloses a method for generating an indoor shadow removal dataset based on path tracing. The method first obtains a three-dimensional scene dataset, removes the self-intersection problem of furniture in each scene in the dataset, and obtains scene layout information without object collision. Secondly, the size of each room in the scene, the objects contained and the position information of each object are traversed; each room in the scene is selected to find N camera extrinsics. Then, the light source is set according to the obtained camera position. Finally, based on the scene object layout information, camera extrinsics and light source information, the image is rendered based on the path tracing algorithm to generate a direct illumination map without shadows and an indirect illumination map without shadows to obtain a shadow-free map. The present invention can quickly generate a large amount of high-quality shadow removal training data, thereby accelerating the research and development and optimization process of the shadow removal algorithm.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method for generating an indoor shadow removal dataset based on path tracing. Background Art

[0002] Shadows are a ubiquitous phenomenon that can arise from occlusion under both direct and indirect lighting. In interior scenes, shadow handling is particularly challenging due to the complexity of lighting conditions. Indoor scenes typically contain multiple direct and indirect light sources, and the interaction of these light sources with objects produces a variety of shadow effects, including hard shadows, soft shadows, and complex shadows caused by light reflection and refraction.

[0003] In the fields of computer vision and graphics, shadow removal is of great importance as it can enhance the performance of various computer vision tasks, such as object segmentation and tracking, intrinsic decomposition, and 3D reconstruction.

[0004] Existing shadow removal techniques mainly rely on deep learning methods, which require large amounts of labeled datasets for training. However, existing datasets often have the following problems:

[0005] 1. Insufficient and noisy datasets: Public datasets may contain inaccurate or incomplete shadow labels, which will cause the trained model to perform poorly in real applications.

[0006] 2. Ambiguity of shadow labels: In indoor scenes, there may be visual similarities between shadows and non-shadow areas, which makes it difficult to automatically generate accurate shadow labels.

[0007] 3. Single lighting conditions: Most existing datasets focus on shadows under direct lighting conditions and lack consideration of shadows under indirect lighting conditions, which limits the generalization ability of the model when dealing with complex indoor lighting scenes.

[0008] 4. Scene diversity and realism: Existing 3D scene datasets may have limitations in object layout, furniture diversity, and lighting conditions, which affects the diversity and realism of the dataset, thereby limiting the quality of training data and the applicability of the model.

[0009] 5. Limitations of rendering technology: Traditional rendering technology may not be able to accurately simulate the propagation and interaction of light in indoor environments, especially when dealing with multiple light sources, complex materials and spatial layouts.

[0010] The above dataset issues lead to the following shortcomings in existing shadow removal methods:

[0011] (1) For complex outdoor scenes, the effect of indirect shadow removal is not ideal;

[0012] (2) For indoor scenes, the effects of direct and indirect shadow removal are far from ideal.

[0013] At the same time, although there are currently large open source three-dimensional scene datasets, these datasets have certain problems in some aspects, such as the room is too empty, the furniture intersects, and the lamps are suspended in the air, which will lead to poor shadow removal effects in indoor scenes. Summary of the Invention

[0014] This paper proposes a path-tracing-based definition of indirect shadows and creates a new shadow removal / detection dataset that takes indirect shadows into account. This synthetic dataset contains over 20,000 image pairs with and without shadows. Experimental results demonstrate that this method can generate high-quality shadow removal datasets suitable for various indoor environments, resolving the issues discussed in the background art.

[0015] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0016] 1. Scene analysis and furniture processing: Preprocess and clean the open-source 3D indoor scene dataset to eliminate data noise and inconsistencies;

[0017] 2. Camera parameter selection: Screen suitable rooms based on the 3D information of the scene and determine the extrinsic parameters of at least one camera for the selected room, including position and orientation;

[0018] 3. Lighting settings: Dynamically set the light source based on the intersection of the camera optical axis and the scene to simulate shadow generation under indoor lighting conditions;

[0019] 4. Path tracing algorithm: Use the path tracing algorithm and the definition of indirect shadows to render and generate image pairs with and without shadows to construct a shadow removal training dataset.

[0020] In order to achieve the above object, the present invention adopts the following technical solution, comprising the following steps:

[0021] Step 1: Obtain a 3D scene dataset and remove the self-intersection problem of furniture in each scene in the dataset to obtain scene layout information without object collision. The solution to the furniture self-intersection problem is as follows:

[0022] (1-1) Traverse each room in the 3D scene dataset and obtain the bounding box of the furniture in each room and its corresponding position information.

[0023] (1-2) To room R i All the furniture in the box are arranged in ascending order according to the volume of the bounding box to form a set F all .

[0024] (1-3) Establish the set F to be checked for collision check =F all .

[0025] (1-4) from F check Choose the smallest piece of furniture F, and then all Perform collision check on the bounding box of the xy plane. If the bounding box collides, proceed to the next step. Otherwise, start from F check Remove furniture F until F check Empty.

[0026] (1-5) Use the Polygon_mesh_processing::do_intersect() function in the public computational geometry algorithm library CGAL to perform collision detection. If no collision occurs, check Delete furniture F and go to step (1-4). If a collision occurs, go to step (1-6) to remove the collision between the furniture.

[0027] (1-6) Based on the current position information of furniture F obtained in step (1-1), move it counterclockwise to the right, up, left, and down to detect collisions. The specific implementation steps are as follows:

[0028] The initialization step size is 0.05 meters, and the number of moves is one step. Furniture F moves one step to the right first, and then undergoes collision detection. If there is a collision, it moves one step up and continues the collision detection. At this time, after half a round, the number of moves increases once every half a round. If there is a collision, move two steps to the left. If a collision occurs, continue to move two steps down. The above process completes a round of judgment. If there is still a collision, repeat the above operation until the number of moves increases to 10 times, that is, 0.5 meters is moved. If no collision occurs in the end, modify the step size to 0.025 meters, and move back step by step from the current position until a collision occurs. The final position of the object is the position of the previous step before the collision. Then from F check Delete furniture F and go to step (1-4). If the collision still occurs, go to step (1-7).

[0029] (1-7) After the above movement, first move the furniture F to its original position. Assume that the set of furniture that collides with the current furniture F is F collsion , then from F collsion Select the smallest piece of furniture F c With F all Perform the above collision detection. If F c If a collision still occurs after step (1-6), then the F all Delete the furniture F and move Fc Move back to its previous position and jump to (1-4), otherwise continue with F collsion Furniture collision detection. If F collsion After all the furniture in the game moves without collision, then check Delete furniture F and skip to step (1-4).

[0030] Through the above step 1, a scene layout information without object collision is finally obtained.

[0031] Step 2: Based on the given scenario S i ∈S, first traverse the scene S i The specific information of each room in the scene (the size of the room, the objects contained and the location information of each object, etc.). i Find N camera extrinsics C for each room in parm The camera external parameters include position, rotation, and orientation.

[0032] First, a three-dimensional position t is randomly selected within the bounding box of the room. n (x, y, z), where z ranges from 1.2 meters to 1.8 meters. The camera's roll angle is set to 0°, and the pitch and yaw angles are randomly set. The pitch angle ranges from 70° to 90°, and the yaw angle ranges from 0° to 360°.

[0033] Then select the camera external parameter C that meets the following requirements parm , those that do not meet the conditions will be discarded directly:

[0034] (2-1) Assume that the intersection of the camera optical axis and the scene is p intersection , the intersection point must be in the same room as the camera, and the camera to the intersection point p intersection The distance is between 3 and 5 meters.

[0035] (2-2) The furniture accounts for between 35% and 70% of the entire camera view.

[0036] (2-3) The area of ​​a single furniture object in the viewing angle does not exceed 30%.

[0037] Furniture does not include ceilings, walls and floors.

[0038] Step 3: Set the light source according to the camera position obtained in step 2. The steps for setting the light source in the room are as follows:

[0039] (3-1) First, find the information of the light source in each room based on the information of each room, and select a camera external parameter C in step 2. parm .

[0040] (3-2) Calculate the intersection point p between the camera's optical axis and the room intersection With each light source L in the room k The distance of the light source is less than 2.5 meters (the distance is the distance in the xy two-dimensional plane), which is set to open, otherwise the light source is turned off. The brightness of the light source is determined by the number of currently open light sources and the light source and the intersection point p. intersection The distance between them is set adaptively.

[0041] (3-3) If at the intersection point p intersection If there is no light source within 2.5 meters, the program will intersection Sample a point in a circle on the xy plane with a radius of 2.5 meters and insert a light source at the point. Set the z value of the light source to align with the ceiling, and set the brightness as described in step (3-2).

[0042] (3-4) After generating the above light sources, start using the shadow map to determine whether the shadows produced by the light sources are sufficient.

[0043] The specific steps are as follows:

[0044] ① Randomly sample about 10 positions on the light source, then use the model matrix, view matrix and perspective matrix to move the camera to one of the light source positions p_light, and then generate a depth map D from this direction l .

[0045] ② Select the camera external parameters, and then generate a depth map D based on the camera external parameters c , then the depth map D c Each pixel is converted to obtain its texture coordinates in the light space coordinates, and then from D l Get the depth value of the point, and then use D c A shadow map V is obtained by pixel-by-pixel comparison.

[0046] Where i,j are the coordinates of the pixel.

[0047] ③ The depth map D generated by sampling the position of each light source l i th and D c The shadow map V obtained after comparison i th , then calculate V i th The proportion of the shadow P_shadow i th , then take all shadow maps P_shadow ith Add up and take the average to get the final shadow ratio P_shadow final , then set a threshold. If it is greater than the threshold, keep the light source position. Otherwise, discard the light source position and jump to step (3-3) to sample a new light source position. If the light source position that meets the requirements is still not found after repeating several times, then discard the camera external parameter C parm , select a new camera extrinsic parameter.

[0048] (3-5) at the distance p intersection At a distance of 2 to 4 meters, sample a position that is outside the camera's field of view, and then randomly select a point light source or surface light source at this position. If it is a surface light source, take the intersection point p as the intersection A point p' is sampled in a circle with the center as the center. The direction of the light source is pointing to point p'. The brightness of these two light sources is the reference brightness I standerd One fifth of .

[0049] Step 4: After obtaining the scene object layout information from step 1, the camera extrinsics from step 2, and the light source information from step 3, the modified path tracing algorithm is used to render the image. This algorithm is mainly used to generate a direct light map without shadows and an indirect light map without shadows. The shadows in the indirect light map are called indirect shadows. The definition of indirect shadows is as follows:

[0050] Coloring point p shading The brightness is darker when there is occlusion than when there is no occlusion.

[0051] The following two functions are defined to describe the rendering process. The functions are as follows:

[0052] Function EstimateDirectLighting(x,ω o ,with_shadow) is used to calculate the radiance of direct lighting. The direct lighting map is generated using this function. The specific execution steps of this function are as follows:

[0053] Assume that the current coloring point is point p shading , the direction of the light is ω o When with_shadow is set to True, if the incident light comes from the light source and there is an occlusion between the light source and the shading point, the radiance L' of the light will be discarded. When with_shadow is set to False, the light source and the shading point p will not be considered. shading The occlusion between the two is eliminated, thus generating a direct lighting map without shadows. The formula for calculating the total radiance is as follows:

[0054] L emit=p shading .emittedLight(ω o )

[0055] L=(L'·bsdf·cos(θ) / pdf)+L emit ,

[0056] Where bsdf represents the colored point p shading The bidirectional scattering distribution function p shading .emittedLight(ω o ) represents the colored point p shading To ω o The emitted radiance, if the shading point p shading If it is a light source, then the value is not 0. i is the direction of the incident light sampled from the shading point p shading points to the light source. θ is ω i With the colored point p shading The angle between the normals and pdf is the probability density function of the sampling method used when sampling the incident light.

[0057] Function EstimateIndirectLighting(x,ω o ,with_shadow,ray_length,bounce) is used to calculate the radiance of indirect lighting. The specific execution steps of this function are as follows:

[0058] Assume that the current input of x is the coloring point p shading , the direction of the light is ω o , ray_length is the shading point p shading The distance to the camera, bounce is used to count the number of times the ray intersects the surface, and the initial input is 0. The direction of the sampled incident light is ω i , from the coloring point p shading Pointing to the secondary light source, the intersection with the secondary light source is p' shading , and set the variable trans_depth to calculate the number of times the light passes through the transparent surface. This function will be called recursively to trace the path of the light, and the shading point will become p' the next time the function is called shading , the direction of light emission becomes -ω i , ray_length is point p shading To point p' shading The distance, bounce will increase by 1. The recursive termination condition of this function is: in the last recursive process, the light hits the light source and is randomly terminated by Russian roulette.

[0059] Since this function calculates indirect lighting, when bounce≤1, if the path of the light hits the light source, then the radiance L of the light emit Set it directly to 0, otherwise return the radiance L of the light source emit .

[0060] When with_shadow is set to False and the bounce of the ray is 1, if it meets the following conditions: ray_length is less than the threshold r and trans_depth is an odd number.

[0061] Then the intersection point p' shading Will be considered transparent, that is, set the intersection point p' shading The BSDF at is 1. Then the indirect lighting radiance L is calculated recursively to obtain an indirect lighting map without shadows.

[0062] The formula for calculating the total radiosity L is as follows:

[0063]

[0064] where ω i is the angle of the incident light sampled, θ is ω i With the colored point p shading The angle between the normals, pdf is the probability density function of the sampling method used when sampling the incident light, and bsdf is the shading point p shading The bsdf and cpdf of the material are probabilities set by Russian roulette.

[0065] The steps for rendering an image are as follows:

[0066] (1) Generate the original image with shadows according to the path tracing algorithm S .

[0067] (2) Use the EstimateDirectLighting() function to generate a direct lighting map without shadows

[0068]

[0069] (3) Use the EstimateIndirectLighting() function to generate the image Then and I S Take the maximum value to get x is the direction of the camera launch and ω is o The intersection of the ray and the scene.

[0070]

[0071]

[0072] (4) Direct lighting map without shadows and an indirect light map without shadows Add in linear space to get the original image I S The unshadowed image.

[0073] The beneficial effects of the present invention are:

[0074] We developed a comprehensive indoor shadow removal dataset generation process that leverages existing 3D indoor scene datasets and effectively generates shadowed and unshadowed image pairs using path tracing rendering techniques. This approach not only improves the training efficiency of shadow removal algorithms but also enhances their robustness in handling complex indoor lighting conditions.

[0075] By precisely controlling the light source settings and the selection of camera extrinsic parameters, the present invention can generate high-quality shadow removal datasets that can realistically reflect the lighting changes and shadow characteristics in indoor environments, thereby providing richer and more realistic training samples for the shadow removal network.

[0076] The shadow removal network trained using the dataset generated by the method of the present invention can effectively remove direct and indirect shadows in indoor scenes. In particular, in complex indoor environments, such as those with multiple objects blocking and light reflections, the network can still perform well and provide high-quality shadow removal effects.

[0077] The proposed method is highly efficient and can quickly generate large amounts of shadow removal training data, accelerating the development and optimization of shadow removal algorithms. Furthermore, due to the use of a path tracing algorithm, the generated images are high-quality, providing a reliable foundation for training and evaluating shadow removal networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 It is a flow chart of the technical solution of the present invention;

[0079] Figure 2 Object collision map;

[0080] Figure 3 Object movement diagram;

[0081] Figure 4 Lighting model diagram;

[0082] Figure 5 Original image with shadow;

[0083] Figure 6 Direct light map without shadows;

[0084] Figure 7Indirect lighting without shadows;

[0085] Figure 8 Original image without shadows. DETAILED DESCRIPTION

[0086] like Figure 1 As shown in the figure, the indoor shadow removal dataset generation method based on path tracing rendering includes the following steps: 1. Preprocessing and data cleaning of the open source 3D indoor scene dataset to ensure the natural visual effect and the practicality of the dataset; 2. Screening suitable rooms according to the 3D information of the scene and determining the external parameters of the camera to simulate the shadow effects under different observation angles and increase the diversity of the dataset; 3. Setting the light source according to the intersection position of the camera optical axis and the scene, and generating and simulating shadows to simulate the lighting conditions in the real world; 4. Using the path tracing algorithm and the definition of indirect shadows to render, generate image pairs with and without shadows to construct a high-quality shadow removal training dataset.

[0087] The following examples and attached Figure 2-8 The present invention will be described in detail.

[0088] The four steps of this method are now described in detail:

[0089] Step 1: Use the 3D scene dataset 3DFront. Each scene in this dataset may have furniture self-intersection problems, such as Figure 2 The solution to this problem is as follows:

[0090] (1-1) Traverse each room, generate a bounding box for each piece of furniture, and record its position information. This step is used for subsequent collision detection and layout optimization to ensure that there is no physical overlap between furniture, improving the realism of the scene and the usability of the dataset.

[0091] (1-2) To room R i All the furniture in the collection are sorted according to the volume of the bounding box, with small objects in front and large objects in the back, forming a set F all Because small objects are less likely to collide when moved, small objects are moved first.

[0092] (1-3) Establish the set F to be checked for collision check =F all

[0093] (1-4) from F check Choose the smallest piece of furniture F, and then all Perform collision check on the bounding box of the xy plane. If the bounding box collides, proceed to the next step. Otherwise, start from F checkRemove furniture F until F check Empty.

[0094] (1-5) Use the Polygon_mesh_processing::do_intersect() function in the public computational geometry algorithm library CGAL for further detection. If no collision occurs, check Delete furniture F and go to step (1-4). If a collision occurs, go to step (1-6) to remove the collision between the furniture.

[0095] (1-6) The current position information of furniture F obtained in step (1) is used as the center, and the furniture F is moved outward in a counterclockwise direction to the right, up, left, and down to detect collisions. Figure 3 The specific implementation steps are as follows:

[0096] The initialization step size is 0.05 meters, and the number of moves is one step. The furniture moves one step to the right first, and then undergoes collision detection. If a collision occurs, it moves one step up and continues the collision detection. At this time, after half a round, the number of moves increases once every half round. If a collision occurs, move two steps to the left. If a collision occurs, continue to move two steps down. The above process completes a round of judgment. If there is still a collision, repeat the above behavior until the maximum step size to the right is 0.5 meters. Increasing the step size after each round of movement can increase the speed of collision judgment. If no collision occurs in the end, modify the step size to 0.025 meters, move back from the current position until a collision occurs, and the position of the object is taken from the step before the collision. Then start from F check Delete furniture F and skip to step (4). If there is still a collision, proceed to steps (1-7). Moving it back can help maintain a better layout. If a stool and a table intersect, and the overlapping area is large, then after moving the stool out, the stool may be far away from the table, which looks strange, so try to move it back to a suitable position.

[0097] (1-7) After the above movement, first move the furniture F to its original position. Assume that the set of furniture that collides with the current furniture F is F collsion , then from F collsion Select the smallest piece of furniture F c With F all Perform the above collision detection. If F c After step (1-6), a collision still occurs, which means that it is difficult for furniture F to find a suitable position. all Delete the furniture F and move F c Move back to its previous position and jump to (1-4), otherwise continue with F collsion Furniture collision detection. If Fcollsion After all the furniture in the game moves without collision, then check Delete furniture F and skip to step (1-4).

[0098] Through the above step 1, a scene layout information without object collision is finally obtained.

[0099] Step 2: Based on the given scenario S i ∈S, first traverse the scene S i The specific information of each room in the scene (the size of the room, the objects contained and the specific location information of each object, etc.). Then select the scene S i The furniture has more than five rooms, which can avoid sampling the camera's external parameters in empty rooms and improve the efficiency of the algorithm. Then find two camera external parameters C in each of these screened rooms. parm . Each camera external parameter C parm The following steps will be executed 2000 times in a loop. If no camera extrinsic parameters are found, the next camera extrinsic parameter will be found. This process is designed to simulate shadow effects under different viewing angles and increase the diversity of the dataset.

[0100] First, a three-dimensional position t is randomly selected within the bounding box of the room. n (x, y, z), where z ranges from 1.2 meters to 1.8 meters. The camera's roll angle is set to 0°, and the pitch and yaw angles are randomly set. The pitch angle ranges from 70° to 90°. This ensures that the camera's orientation matches the orientation of the human eye, making the generated data more consistent with human habits. The yaw angle ranges from 0° to 360°.

[0101] Then select the camera external parameter C that meets the following requirements parm , those that do not meet the conditions will be discarded directly:

[0102] (2-1) Assume that the intersection of the camera optical axis (the center of the camera's orientation) and the scene is p intersection , the intersection point must be in the same room as the camera, so as to avoid the camera looking through the door in another room. And the camera to the intersection point p intersection The best distance is between 3 and 5 meters to prevent the camera from getting too close to or too far away from the object.

[0103] (2-2) The furniture accounts for between 35% and 70% of the camera's field of view. This way, the scene in the resulting image will not be too empty or too crowded.

[0104] (2-3) The proportion of a single furniture object in the field of view does not exceed 30%, so as to avoid the camera only seeing 1-2 objects.

[0105] Furniture does not include ceilings, walls, and floors. The specific calculation steps for the proportion are as follows:

[0106] ①Build a BVH (bounding volume hierarchy) tree for the entire room to speed up the process of finding the intersection between light and scene.

[0107] ② Based on the camera's optical axis and intrinsic parameters (FOV, far plane, near plane), the number of rays to be cast is 40*40. Each ray is emitted from the camera cam as the starting point and the direction is from the camera cam to the center c of each pixel;

[0108] ③ Each ray traverses the entire BVH tree to obtain the final leaf node. Then determine whether the ray intersects with the triangle in the leaf node;

[0109] ④After obtaining the number of times each object is hit by light, calculate the proportion of each object in the room that is hit.

[0110] Step 3: Set the light source based on the camera position obtained in step 2 to simulate the lighting conditions in the real world. The steps for setting the light source in the room are as follows:

[0111] (3-1) First, find the information of the light source in the room based on the information of each room, and select a camera external parameter C in step 2 parm .

[0112] (3-2) Calculate the intersection of the camera's optical axis and the room's intersection cam and each light source L in the room k The distance of the light source is less than 2.5 meters (the distance is the distance in the xy two-dimensional plane), which is set to open, otherwise the light source is turned off. The brightness of the light source is determined by the number of currently open light sources and the light source and the intersection point p. intersection The distance between them is set adaptively. The brightness of the light source is set adaptively as follows:

[0113] ①First set a reference brightness I standerd , set the brightness of the light source to I standerd .

[0114] ②Calculate the area of ​​the light source. If the area of ​​the light source is less than the minimum value of Area, min Or greater than the maximum value Area max , then the brightness of the light source will be adjusted linearly according to the area of ​​the light source. Because the reference brightness refers to the intensity per unit area of ​​I standerd , so the total brightness of the lamp with a larger area is greater, and vice versa.

[0115]

[0116] ③ Get the light source to the intersection point p intersection The distance distance, according to the distance to continue linearly adjust the brightness of the light source, where distance stand This is the reference distance. The farther the distance, the higher the brightness should be, so that the scene does not become too dark.

[0117]

[0118] ④ Get the number of currently turned on light sources light_count, and then set the final brightness I according to the number of turned on light sources light_count final .

[0119]

[0120] (3-3) If at the intersection point p intersection If there is no light source within 2.5 meters, the program will intersection A point is sampled in a circle on the xy plane with a radius of 2.5 meters and a light source is inserted at the point. The z value of the light source is set to align with the ceiling. The light source described above is a volumetric lamp, such as Figure 4 As shown in the figure, the lamp model comes from the 3DFuture 3D dataset. The brightness of the light source is set as shown in step (3-2).

[0121] (3-4) After generating the above light sources, start using the shadow map to determine whether the shadows produced by the light sources are sufficient.

[0122] The specific steps are as follows:

[0123] ① Randomly sample about 10 positions on the light source, then use the model matrix, view matrix and perspective matrix to move the camera to one of the light source positions p_light, and then generate a depth map D from this direction l .

[0124] ② Select the camera external parameter, and then generate a depth map D based on this external parameter c , then the depth map D c Each pixel is converted to obtain its texture coordinates in the light space coordinates, and then from D l Get the depth value of the point, and then use D c A shadow map V is obtained by pixel-by-pixel comparison.

[0125] Where i,j are the coordinates of the pixel

[0126] ③ The depth map D generated by sampling the position of each light source li th and D c The shadow map V obtained after comparison i th , then calculate V i th The proportion of the shadow P_shadow i th , then take all shadow maps P_shadow i th Add up and take the average to get the final shadow ratio P_shadow final , then set a threshold. If it is greater than the threshold, keep the light source position, and the rendered image will have rich shadows. Otherwise, discard the light source position and jump to step (3-3) to sample a new light source position. If the appropriate light source position is still not found after 100 repetitions, then discard the camera external parameter C parm , select a new camera extrinsic parameter.

[0127] (3-5) at the distance p intersection At a distance of 2 to 4 meters, sample a position that is outside the camera's field of view, and then randomly select a point light source or surface light source and place it at that position. This step can enrich the diversity of shadows. If it is a surface light source, take the intersection point p as the intersection A point p' is sampled in a circle with the center as the center. The direction of the light source is pointing to point p'. The brightness of these two light sources is the reference brightness I standerd One fifth of .

[0128] Step 4: After obtaining the collision-free scene layout information from Step 1, the camera extrinsics from Step 2, and the light source information from Step 3, use the improved path tracing algorithm to render the image. This algorithm primarily generates shadowless direct and indirect light maps. The with_shadow variable controls the generation of shadowless images.

[0129] Function EstimateDirectLighting(x,ω o ,with_shadow) is used to calculate the radiance of direct lighting. The direct lighting map is generated using this function. The specific execution steps of this function are as follows:

[0130] Assume that the current coloring point is point p shading , the direction of the light is ω oWhen with_shadow is set to True, if the incident light comes from the light source and there is an occlusion between the light source and the shading point, the radiance L' of the light will be discarded. When with_shadow is set to False, the light source and the shading point p will not be considered. shading The occlusion between the two is eliminated, thus generating a direct lighting map without shadows. The formula for calculating the total radiance is as follows:

[0131] L emit =p shading .emittedLight(ω o ),

[0132] L=(L'·bsdf·cos(θ) / pdf)+L emit

[0133] Where bsdf represents the colored point p shading The bidirectional scattering distribution function, p shading .emittedLight(ω o ) represents the colored point p shading To ω o The emitted radiance, if the shading point p shading If it is a light source, then the value is not 0. i is the direction of the incident light sampled from the shading point p shading points to the light source. θ is ω i With the colored point p shading The angle between the normals and pdf is the probability density function of the sampling method used when sampling the incident light.

[0134] Function EstimateIndirectLighting(x,ω o ,with_shadow,ray_length,bounce) is used to calculate the radiance of indirect lighting. The specific execution steps of this function are as follows:

[0135] Assume that the current input of x is the coloring point p shading , the direction of the light is ω o , ray_length is the shading point p shading The distance to the camera, bounce is used to count the number of times the ray intersects the surface, and the initial input is 0. The direction of the sampled incident light is ω i , from the coloring point p shading Pointing to the secondary light source, the intersection with the secondary light source is p' shading , and set the variable trans_depth to calculate the number of times the light passes through the transparent surface. This function will be called recursively to trace the path of the light, and the shading point will become p' the next time the function is calledshading , the direction of light emission becomes -ω i , ray_length is point p shading To point p' shading The distance, bounce will increase by 1. The recursive termination condition of this function is:

[0136] 1. During the last recursion, the light hits the light source.

[0137] 2. Randomly terminated by Russian roulette.

[0138] Since this function calculates indirect lighting, when bounce≤1, if the path of the light hits the light source, then the radiance L of the light emit Set it directly to 0, otherwise return the radiance L of the light source emit .

[0139]

[0140] When with_shadow is set to False and the bounce of the light is 1, if the following conditions are met:

[0141] 1. ray_length is less than the threshold r, which ensures that the light only penetrates nearby objects rather than distant objects.

[0142] 2. If trans_depth is an odd number, and the object is a closed mesh, when light penetrates the object, it may only penetrate one side of the object. Then, when it penetrates the other side, ray_length may be greater than the threshold r, resulting in the light not fully penetrating the object. Therefore, when light penetrates one side of the object, trans_depth increases by 1 to become an odd number. Then, when it penetrates the other side, even if ray_length is less than the threshold r, the light continues to penetrate the other side of the object. This allows light to penetrate an object instead of being stuck inside.

[0143] Then the intersection point p' shading Will be considered transparent, that is, set the intersection point p' shading The BSDF at is 1. At the same time, the path is traced recursively to p" shading When p' shading When the point becomes a colored point, at p' shading The direction of the sampled incident light ω i and the direction of the current outgoing light ω o On the contrary. This means that the light can directly penetrate p' shading point, which is equivalent to p' shading There is no object occlusion at the point. At the same time, trans_depth increases by one. At this time, p"shading and p' shading Not on the same object, because trans_depth plus one is an odd number, so the light will pass through p' shading On the other side of the object where the point is located, trans_depth increases by one to become an even number, and the light cannot penetrate the object. Then the light hits the surface of another object, and the intersection point is p" shading . Then recursively calculate the indirect lighting radiance L to obtain an indirect lighting map without shadows.

[0144] The formula for calculating the total radiosity L is as follows:

[0145]

[0146] where ω i is the angle of the incident light sampled, θ is ω i The angle with respect to the normal of the shading point p, pdf is the probability density function of the sampling method used when sampling the incident light, bsdf is the bsdf of the material of the shading point p, and cpdf is a probability set by Russian roulette.

[0147] The steps for rendering an image are as follows:

[0148] (4-1) Generate the original image with shadows according to the path tracing algorithm I S ,like Figure 5 shown.

[0149] (4-2) Use the EstimateDirectLighting() function to generate a direct lighting map without shadows like Figure 6 shown.

[0150]

[0151] (4-3) Use the EstimateIndirectLighting() function to generate an image Then and I S Take the maximum value to get like Figure 7 As shown in the figure, x is the direction of the camera emission and ω is o The intersection of the light and the scene. The maximum value of the two is taken because there is a situation where, when the light source is on the ground, if there is a table on the ground, the bottom of the table will become a secondary light source, illuminating the shading point p on the ground. shading Secondary light sources are all objects that are directly illuminated by light and will act as light sources to illuminate other objects again. Assume that the shading point p is shadingThe light from the table penetrates the ceiling, which will become its secondary light source, which will lead to the shading point p shading When there is occlusion around, the brightness is brighter than when there is no occlusion. This violates the definition of indirect shadows, indicating that this is not an indirect shadow, so the brightness of the original image is taken.

[0152]

[0153]

[0154] (4-4) Direct lighting map without shadows and an indirect light map without shadows Add in linear space to get the original image I S The unshadowed image of Figure 8 As shown:

[0155]

Claims

1. A method for generating indoor shadow removal dataset based on path tracing, characterized in that: The following steps are involved: Step 1: Obtain a 3D scene dataset, remove the self-intersection problem of furniture in each scene in the dataset, and obtain scene layout information without object collision; The solution to the self-intersection problem of furniture is as follows: 1-1: Traverse each room in the 3D scene dataset and obtain the bounding box of the furniture in each room and its corresponding position information; 1-2: Opposite room R i All the furniture in the set are arranged in ascending order according to the volume of the bounding box to form a set F all ; 1-3: Create a set F to be checked for collision check =F all ; 1-4: From F check Choose the smallest furniture F and F all Perform collision check on the bounding box of the xy plane. If the bounding box collides, proceed to the next step of detection, otherwise proceed from F check Remove furniture F until F check is empty; 1-5: Use the Polygon_mesh_processing::do_intersect() function in the computational geometry algorithm library CGAL to perform collision detection. If no collision occurs, check Delete furniture F and skip to step 1-4; if a collision occurs, proceed to step 1-6 to remove the collision between the furniture. 1-6: With the current position of furniture F as the center, move counterclockwise to the right, up, left, and down to detect collision: the initial step size is 0.05 meters, and the number of moves is one step; furniture F moves one step to the right first, and then performs collision detection; if there is a collision, it moves one step up and continues to perform collision detection; at this time, after half a round, the number of moves increases once every half round, if there is a collision, move two steps to the left; if there is a collision, continue to move two steps down; the above process completes a round of judgment. If there is still a collision, repeat the above operation until the number of moves increases to 10 times, that is, it moves 0.5 meters; if there is no collision in the end, modify the step size to 0.025 meters, and move back step by step from the current position until a collision occurs. The final position of the object is the position of the previous step before the collision; then move back from F check Delete furniture F and skip to step 1-4; if it still collides, go to step 1-7; 1-7: First move the furniture F to its original position. Assume that the set of furniture that collides with the current furniture F is F. collsion , then from F collsion Select the smallest piece of furniture F c With F all Perform the above collision detection; if F c If a collision still occurs after steps 1-6, then the F all Delete the furniture F and move F c Move it back to its previous position and skip to steps 1-4, otherwise continue with F collsion Furniture collision detection, if F collsion All the furniture in the game moves without collision, starting from F check Delete furniture F and skip to steps 1-4; Step 2: Traverse scene S i The size of each room, the objects it contains, and the location information of each object; then in the scene S i For each room in the camera, find N camera external parameters C including camera position, rotation, and orientation parm ; Step 3: Set the light source according to the camera position obtained in step 2; Step 4: Render the image based on the path tracing algorithm according to the scene object layout information, camera external parameters and light source information, generate a direct lighting map without shadows and an indirect lighting map without shadows, and obtain a shadow-free map.

2. The method for generating an indoor shadow removal dataset based on path tracing according to claim 1, characterized in that: The second step is to find the camera external parameter C parm The process is as follows: First, a three-dimensional position t is randomly selected within the bounding box of the room. n (x, y, z), where z ranges from 1.2 meters to 1.8 meters; the camera's roll angle is set to 0°, the pitch angle and yaw angle are randomly set, with the pitch angle ranging from 70° to 90° and the yaw angle ranging from 0° to 360°; Then select the camera external parameter C that meets the following requirements parm , those that do not meet the conditions will be discarded directly: Assume that the intersection of the camera optical axis and the scene is p intersection , the intersection point must be in the same room as the camera, and the camera to the intersection point p intersection The distance is between 3 and 5 meters; Furniture takes up between 35% and 70% of the camera's field of view. No single furniture object takes up more than 30% of the viewing angle.

3. The method for generating an indoor shadow removal dataset based on path tracing according to claim 2, characterized in that: The specific calculation of the proportion is as follows: Build a bounding volume hierarchy BVH tree for the entire room; According to the optical axis and intrinsic parameters of the camera, a ray is projected into the scene. The number of ray projections is 40*40. Each ray is emitted from the camera cam as the starting point and the direction is from the camera cam to the center c of each pixel. Each ray traverses the entire BVH tree and obtains the final leaf node; Then determine whether the ray intersects with the triangle in the leaf node; After obtaining the number of times each object is hit by light, calculate the proportion of each object in the room that is hit.

4. The method for generating an indoor shadow removal dataset based on path tracing according to claim 3, characterized in that: The specific implementation process of step three is as follows: 3-1: Find the light source information in the room based on the room information, and select a camera external parameter C parm ; 3-2: Calculate the intersection point p between the camera's optical axis and the room intersection With each light source L in the room k The distance between the light source and the intersection point p is less than 2.5 meters; the light source is set to on if the distance between the xy two-dimensional plane is less than 2.5 meters, otherwise it is turned off; the brightness of the light source is determined by the number of currently turned on light sources and the light source and the intersection point p. intersection The distance between them is adaptively set; 3-3: If at the intersection point p intersection There is no light source within 2.5 meters of the intersection Sample a point in a circle on the xy plane with a radius of 2.5 meters and insert a light source at that point. Set the z value of the light source to align with the ceiling and set the brightness as described in step (3-2). 3-4: After generating the above light sources, use the shadow map to determine whether the shadows produced by the light sources are sufficient; 3-5: At distance p intersection At a distance of 2 to 4 meters, sample a position that is outside the camera's field of view, and then randomly select a point light source or surface light source at this position and place it at that position; if it is a surface light source, use the intersection point p intersection A point p' is sampled in a circle with the center as the circle. The direction of the light source is pointing to point p'; the brightness of these two light sources is the reference brightness I standerd One fifth of .

5. The method for generating an indoor shadow removal dataset based on path tracing according to claim 4, characterized in that: The brightness adaptive setting of the light source in step 3-2 is specifically as follows: Set the base brightness I standerd , set the brightness of the light source to I standerd ; Calculate the area of ​​the light source. If the area of ​​the light source is less than the minimum value of Area min Or greater than the maximum value Area max , then adjust the brightness of the light source linearly according to the area of ​​the light source: Get the light source to the intersection point p intersection The distance distance, according to the distance continues to linearly adjust the brightness of the light source, where distance stand is the set reference distance: According to the number of currently turned on light sources light_count, set the final brightness I final :

6. The method for generating an indoor shadow removal dataset based on path tracing according to claim 4, characterized in that: The specific process of steps 3-4 is as follows: Randomly sample 10 positions on the light source, then use the model matrix, view matrix and perspective matrix to move the camera to one of the light source positions p_light, and generate a depth map D from this direction l ; Select the camera external parameters and generate a depth map D based on the camera external parameters c , the depth map D c Each pixel is converted to obtain its texture coordinates in the light space coordinates, and then from D l Get the depth value of the point through D c A shadow map V is obtained by pixel-by-pixel comparison: Where i, j are the coordinates of the pixel; The depth map D is generated by sampling the position of each light source l ith and D c The shadow map V obtained after comparison ith , calculate V ith The proportion of the shadow P_shadow ith , then take all shadow maps P_shadow ith Add and average to get the final shadow ratio P_shadow final , set a threshold, if it is greater than the threshold, keep the position of the light source, otherwise discard the light source position and jump to step (3-3) to sample a new light source position; if the light source position that meets the requirements is still not found after repeating several times, discard the camera external parameter C parm , select a new camera extrinsic parameter.

7. The method for generating an indoor shadow removal dataset based on path tracing according to claim 6, characterized in that: The specific implementation process of step 4 is as follows: The following two functions are defined to describe the rendering process. The functions are as follows: Function EstimateDirectLighting(x,ω o ,with_shadow) is used to calculate the radiance of direct lighting. The direct lighting map is generated using this function. The specific execution steps of this function are as follows: Assume that the current coloring point is point p shading , the direction of the light is ω o When with_shadow is set to True, if the incident light comes from the light source and there is an occlusion between the light source and the shading point, the radiance L' of the light will be discarded; when with_shadow is set to False, the light source and the shading point p are not considered. shading The occlusion between them is used to generate a direct illumination map without shadows. The formula for calculating the total radiance is as follows: L emit =p shading .emittedLight(ω o ) L=(L'·bsdf·cos(θ) / pdf)+L emit , bsdf represents the colored point p shading The bidirectional scattering distribution function p shading .emittedLight(ω o ) represents the colored point p shading To ω o The emitted radiance, if the shading point p shading It is a light source, and the value is not 0; ω i is the direction of the incident light sampled from the shading point p shading Pointing to the light source; θ is ω i With the colored point p shading The angle between the normal and the pdf is the probability density function of the sampling method used when sampling the incident light; Function EstimateIndirectLighting(x,ω o ,with_shadow,ray_length,bounce) is used to calculate the radiance of indirect lighting. The specific execution steps of this function are as follows: Assume that the current input of x is the coloring point p shading , the direction of the light is ω o , ray_length is the shading point p shading The distance to the camera, bounce is used to count the number of times the light intersects the surface, the initial input is 0; the direction of the sampled incident light is ω i , from the coloring point p shading Pointing to the secondary light source, the intersection with the secondary light source is p' shading , and set the variable trans_depth to calculate the number of times the light passes through the transparent surface; This function will be called recursively to trace the path of the ray, and the next time the function is called the shading point will become p' shading , the direction of light emission becomes -ω i , ray_length is point p shading To point p' shading The distance, bounce will increase by 1; the recursive termination condition of this function is: in the last recursive process, the light hits the light source and is randomly terminated by Russian roulette; Since this function calculates indirect lighting, when bounce≤1, if the path of the light hits the light source, then the radiance L of the light emit Set it directly to 0, otherwise return the radiance L of the light source emit ; When with_shadow is set to False and the bounce of the light is 1, if ray_length is less than the threshold r and trans_depth is an odd number, the intersection point p' shading will be considered transparent, i.e. set the intersection point p' shading The BSDF at is 1; then the indirect lighting radiance L is recursively calculated to obtain an indirect lighting map without shadows; The formula for calculating the total radiosity L is as follows: L+=EstimateIndirectLighting(p' shading ,-ω i ,with_shadow where ω i is the angle of the incident light sampled, θ is ω i With the colored point p shading The angle between the normals, pdf is the probability density function of the sampling method used when sampling the incident light, and bsdf is the shading point p shading The bsdf of the material, cpdf is a probability set by Russian roulette; The steps for rendering an image are as follows: Generate the original image with shadows according to the path tracing algorithm I S ; Use the EstimateDirectLighting() function to generate a direct lighting map without shadows Generate an image using the EstimateIndirectLighting() function Then and I S Take the maximum value to get x is the direction of the camera launch and ω is o The intersection of the light and the scene; Direct lighting map without shadows and an indirect light map without shadows Add in linear space to get the original image I S The unshadowed image.

Images