Underground safety production simulation modeling method based on VR technology

Abstract

The invention relates to the technical field of environment modeling, in particular to an underground safety production simulation modeling method based on the VR technology, and the method comprises the steps: obtaining all RGB images of an underground scene and corresponding depth images, and the attitude information and position coordinates of a camera when each frame of RGB image is shot, so as to generate a dense point cloud, and constructing a triangular mesh model; calculating the screening value of each point cloud, and obtaining the feature point cloud of each triangular patch; and obtaining all candidate images of each triangular patch, determining a mapping deviation value, a texture distortion degree and a quality evaluation value of each triangular patch in each candidate image, screening the candidate images of each triangular patch, performing texture mapping on the triangular mesh model, and importing the texture mapping result into a VR engine to construct a virtual underground environment. According to the method, the overall effect of texture mapping of the model can be improved, so that a highly vivid virtual underground scene is constructed, and the immersion and reality of underground scene safety production virtual practical training are enhanced.

Description

Technical Field

[0001] This application relates to the field of environmental modeling technology, specifically to a method for simulating and modeling underground safety production based on VR technology. Background Technology

[0002] With the deepening of smart mine construction, using VR technology to build virtual digital models of underground production scenarios has become an important means of achieving safety training, emergency drills, and remote monitoring. The core of building a realistic underground virtual environment lies in achieving high-precision geometric reconstruction and high-quality texture mapping of the 3D model of the underground scene.

[0003] When using an RGB-D camera to perform 3D modeling of underground scenes, the complex, varied, and irregular structure of the underground scene can lead to errors in the pose of the RGB-D camera. This can easily introduce geometric errors during the 3D modeling process, resulting in a lack of effective texture information sources during texture mapping. If texture information from multiple images is mixed for texture mapping, it can easily lead to texture blurring and color distortion in the 3D model. Secondly, the extremely uneven lighting in the underground environment causes strong highlight reflections and shadows on the surfaces of objects, which can obscure the true texture details of the object surfaces and exacerbate the color distortion in the texture mapping of the 3D model. As a result, it is difficult to reflect the real underground scene, which seriously undermines the realism and immersion of safe production in the virtual underground scene and reduces the quality and effectiveness of training. Summary of the Invention

[0004] To address the aforementioned technical issues, a VR-based method for simulating and modeling downhole safety production is provided to resolve existing problems.

[0005] The solution to the technical problem in this application is to provide a method for simulating and modeling downhole safety production based on VR technology, including the following steps: Acquire all RGB images of the underground scene and their corresponding depth images, as well as the camera's pose information and position coordinates when each RGB image is captured, to generate a dense point cloud and construct a triangular mesh model; Analyze the distribution density of point clouds within each triangular facet of the triangular mesh model, as well as the distribution position of each point cloud within the triangular facet. Calculate the filtering value of each point cloud, filter the point clouds within each triangular facet, and obtain the feature point cloud of each triangular facet. Based on the camera's pose information during RGB image capture, the RGB images containing feature point clouds are filtered to obtain all candidate images for each triangular facet. Then, each point cloud in each triangular facet is mapped onto each candidate image to obtain the mapping points of each point cloud on each candidate image. The deviation between the point cloud distance within each triangular facet and the camera position coordinates when each candidate image was captured, and the depth of the mapped point on the depth image, is evaluated. Combined with the screening value, the mapping deviation of each triangular facet in each candidate image is determined. The texture distortion of each triangular facet in each candidate image is determined by the difference in brightness of the points mapped from the point cloud within each triangular facet on different candidate images, and the inconsistency between the normal vector of each triangular facet and the shooting angle of each candidate image. Based on the mapping deviation and texture distortion, the quality evaluation value of each triangular facet in each candidate image is obtained. Candidate images of each triangular facet are then selected, texture mapping is performed on the triangular mesh model, and the model is imported into the VR engine to construct a virtual underground environment.

[0006] Preferably, the calculation of the filter value for each point cloud includes: All point clouds distributed within each triangular facet are formed into a point cloud set; taking any point cloud in the point cloud set as the center, all point clouds in its neighborhood that belong to the point cloud set are selected as the neighboring point clouds of the given point cloud. The number of all neighboring point clouds of any given point cloud is counted, and the average number of all point clouds distributed within each triangular patch is calculated; the ratio of the number of any given point cloud to the average number is denoted as the relative comparison. Calculate the distance between any point cloud and the centroid of its corresponding triangular facet; The ratio of the relative comparison to the distance is used as the filtering value for any point cloud.

[0007] Preferably, the step of obtaining the feature point cloud of each triangular facet includes: taking the point cloud corresponding to the maximum screening value within each triangular facet as the feature point cloud.

[0008] Preferably, the step of acquiring all candidate images of each triangular facet and mapping each point cloud within each triangular facet onto each candidate image, and acquiring the mapping points of each point cloud on each candidate image, includes: based on the camera's pose information during the capture of each frame of RGB image, transforming the feature point cloud of each triangular facet from the world coordinate system to the camera coordinate system and mapping it onto the image plane; acquiring the pixel coordinates of each feature point cloud in each RGB image; and recording the pixel points represented by the pixel coordinates as mapping points; extracting all RGB images within the camera's field of view of the mapping points as candidate images of each triangular facet; and transforming all point clouds distributed within each triangular facet from the world coordinate system to the camera coordinate system and mapping them onto the candidate images, thereby acquiring the mapping points of each point cloud within each triangular facet on each candidate image.

[0009] Preferably, determining the mapping deviation of each triangular facet in each candidate image includes: The depth value corresponding to the point mapping point in the depth image corresponding to each candidate image is used as the mapping depth of each point cloud in each candidate image. In the world coordinate system, the distance between the position coordinates of each point cloud and the position coordinates of the camera when each candidate image was captured is calculated, and this distance is used as the geometric distance between each point cloud and each candidate image. Calculate the difference between the mapping depth and the geometric distance of each point cloud in each candidate image, as the relative deviation; Analyze the deviation between the selection value of each point cloud and the selection values ​​of all point clouds in the point cloud set corresponding to its respective triangle, and calculate the mapping weight of each point cloud. The mapping deviation is the average of the product of the mapping weights and relative deviations of all point clouds distributed within each triangular patch.

[0010] Preferably, the calculation of the mapping weight of each point cloud includes: calculating the average value of the filtering values ​​of all point clouds in the point cloud set corresponding to the triangle facet to which each point cloud belongs; if the filtering value of each point cloud is greater than the average value, then the mapping weight of each point cloud is the ratio of the filtering value of each point cloud to the average value, otherwise, the mapping weight of each point cloud is a preset value.

[0011] Preferably, determining the texture distortion of each triangular facet in each candidate image includes: Each candidate image is converted from RGB space to HSL space, and the L component of each pixel in each candidate image in HSL space is extracted as the brightness value of each pixel. The differences in brightness values ​​of the points mapped from the point cloud distributed within each triangular facet in different candidate images are analyzed, and the brightness deviation of each triangular facet in each candidate image is calculated. In the world coordinate system, the vector pointing from the feature point cloud of each triangular facet to the position of the camera when each candidate image is captured is used as the view vector of each triangular facet in each candidate image. Calculate the cosine of the angle between the normal vector of each triangular facet and the view vector, and perform a positive mapping on the cosine value; The texture distortion is the ratio between the brightness deviation and the result of the positive mapping.

[0012] Preferably, the calculation of the brightness deviation of each triangular facet in each candidate image includes: Calculate the average brightness value of the points mapped to each candidate image from all point clouds distributed within each triangular facet, and use it as the texture mapping brightness of each triangular facet in each candidate image; The brightness deviation is the average difference between the texture mapping brightness of each triangular facet in each candidate image and the texture mapping brightness of all other candidate images.

[0013] Preferably, the quality assessment value is the result of negative mapping of the product of the mapping deviation and the texture distortion.

[0014] Preferably, the step of filtering candidate images for each triangular facet and performing texture mapping on the triangular mesh model includes: selecting the candidate image corresponding to the highest quality assessment value from all candidate images corresponding to each triangular facet as the texture image; performing texture mapping on all point clouds distributed in each triangular facet on the triangular mesh model; and importing the texture-mapped triangular mesh model into the VR engine.

[0015] This application has at least the following beneficial effects: This application calculates the selection value of each point cloud by analyzing its distribution density and location within each triangular facet, and obtains the feature point cloud of each triangular facet. The advantage lies in selecting point clouds that represent the overall features of the entire triangular facet based on their distribution density and location, ensuring that the selected point clouds are located in information-rich, centrally located, and stable regions, thus reducing subsequent computational load. Secondly, using the feature point clouds, all candidate images of each triangular facet are filtered, and the mapping points of each point cloud on each candidate image are obtained. The advantage of this is that by analyzing the mapping of the feature point clouds in RGB images, the selected candidate images are ensured to contain feature point clouds. Mapping the point clouds of each triangular facet onto the candidate images allows for precise determination of... The position of each point cloud on the candidate image provides accurate mapping points for texture mapping, reducing mapping errors. Determining the mapping deviation of each triangular facet in each candidate image is beneficial because it considers the difference between the depth of the point cloud in the image and the distance between the point cloud and the camera position, as well as the important representation of each point cloud within the triangular facet. This reflects the degree of matching between the model surface corresponding to the triangular facet and the depth information of the candidate image, evaluating the deviation of the geometric mapping of the triangular facets on the candidate images. This allows for the subsequent selection of images with relatively consistent geometric mapping, avoiding blurring or artifacts in texture mapping, and improving the matching accuracy and visual clarity of texture mapping. Finally, the texture distortion of each triangular facet in each candidate image is determined. Its beneficial effect lies in considering the differences in brightness of point clouds on each triangular facet in different candidate images, as well as the inconsistency between the triangular facet and the shooting angle. It reflects the degree of specular reflection component caused by interference from unspecified light sources when the camera captures candidate images at different angles. This allows for the assessment of significant texture distortion in candidate images caused by the shooting angle and interference from unspecified light sources. It identifies candidate images with abnormal lighting such as highlights or shadows caused by strong underground lighting, enabling subsequent selection of the candidate image with the least distortion for texture mapping. This reduces overexposure or shadow areas caused by unspecified light sources, and lowers texture blur and color distortion. This method enhances the realism of texture mapping by obtaining the quality evaluation value of each triangular facet in each candidate image, filtering candidate images of each triangular facet, performing texture mapping on the triangular mesh model, and importing it into the VR engine to construct a virtual underground environment. Its beneficial effect lies in comprehensively evaluating the texture quality of candidate images of each triangular facet through multiple dimensions, addressing texture distortion caused by geometric mapping deviations and interference from unspecified light sources. This allows for the selection of candidate images with clear textures to perform texture mapping on different triangular facets on the triangular mesh model, improving the overall effect of texture mapping, thereby constructing a highly realistic virtual underground scene. This enhances the immersion and realism of virtual safety training in underground scenarios, improving the effectiveness and safety of training and simulation. Attached Figure Description

[0016] The following section provides a more detailed description of the VR-based downhole safety production simulation modeling method of this application, with reference to the accompanying drawings.

[0017] Figure 1 A flowchart illustrating the steps of the VR-based downhole safety production simulation modeling method provided in this application embodiment; Figure 2 A flowchart illustrating the steps of a method for obtaining the texture distortion of each triangular facet in each candidate image, as provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description of the VR-based downhole safety production simulation modeling method proposed in this application, in conjunction with the accompanying drawings and implementation examples, provides further illustrative information. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0020] Please see Figure 1 The diagram illustrates a flowchart of a VR-based downhole safety production simulation modeling method according to an embodiment of this application. The method includes the following steps: Step 1: Acquire all RGB images of the underground scene and their corresponding depth images, as well as the camera's pose information and position coordinates when each RGB image is captured, to generate a dense point cloud and construct a triangular mesh model.

[0021] Due to the complex and ever-changing underground environment and the presence of various safety hazards, traditional two-dimensional planar information is insufficient to fully reflect the spatial structure and dynamic risks, resulting in poor effectiveness of safety training and emergency drills. With the development of virtual reality technology, integrating it with coal mine production to implement effective and systematic safety training is crucial for improving the safety skills and accident prevention and handling capabilities of coal mine workers, enhancing their safety awareness, and improving their safe production skills. To realistically recreate the underground working environment, it is necessary to create a three-dimensional model of the underground environment, constructing a highly realistic virtual scene. This allows trainees to experience the underground environment immersively, improving their awareness and response capabilities to potential hazards, thereby effectively reducing the probability of accidents.

[0022] Based on the above analysis, by equipping a mobile vehicle with an RGB-D camera, which has a built-in high-precision inertial measurement unit (IMU), and setting the inter-frame overlap rate of the RGB-D camera to >70%, the mobile vehicle can be remotely controlled to travel at a constant speed along the centerline of the underground roadway to collect real-time full-coverage data of the underground mining scene. The RGB-D camera includes a pair of color cameras and depth cameras. Therefore, when the RGB-D camera takes a picture of the underground environment, it will acquire an RGB image and a depth image. In the depth image, the pixel value of each pixel represents the distance of each pixel to the RGB-D camera. The built-in IMU unit can acquire the camera pose information of the RGB-D camera in real time when acquiring images; thus, it can acquire RGB images and depth images, as well as camera pose information, in real time. Based on all RGB and depth images, as well as the camera's pose and position coordinates at the time of image capture, a dense point cloud is generated using the ORB-SLAM3 system. The generated dense point cloud is then used as input to a voxel-based triangulation algorithm to obtain a triangular mesh model of the underground scene and its triangular facets. It should be noted that the voxel-based triangulation algorithm divides the dense point cloud into multiple triangular meshes, i.e., triangular patches. The model formed by all the triangular patches is the triangular mesh model. The process of generating dense point clouds by the ORB-SLAM3 system and the voxel-based triangulation algorithm are well-known technologies and will not be described in detail here.

[0023] At this point, all RGB and depth images of the underground scene are obtained, as well as the camera's pose information and position coordinates at the time of shooting for each RGB image, and each triangular facet on the triangular mesh model.

[0024] Step 2: Analyze the distribution density of point clouds within each triangular facet on the triangular mesh model, as well as the distribution position of each point cloud within the triangular facet. Calculate the filtering value for each point cloud, filter the point clouds within each triangular facet, and obtain the feature point cloud of each triangular facet.

[0025] By constructing complex underground scenarios in the form of 3D models, including the direction of tunnels, cross-sectional shapes, support structures, and the placement of underground equipment, a spatial framework highly similar to real underground scenarios is provided for virtual training in underground safety production. Texture mapping applies realistic texture information to the surface of the 3D underground model, such as rock textures, tunnel wall colors, and equipment appearances, making the virtual scene visually more realistic, enhancing the immersion and realism of trainees, and deepening their understanding of the underground mining environment and equipment.

[0026] In the process of texture mapping, the point cloud on the triangular mesh model needs to be mapped with reference to the texture of the pixels in the image. Due to the complexity of the underground production scene, the spatial structures of roadways, chambers, mining faces, etc. are different and irregular, and the number of triangular meshes and point clouds is relatively dense. Direct mapping may lead to discontinuous, blurry or distorted textures, and the computational load of texture mapping is large.

[0027] Based on the above analysis, by analyzing the density of point cloud distribution within each triangular facet and the distance of the point cloud to the geometric center, the point clouds distributed within the triangular facets are filtered, selecting those that are densely distributed within the facets and close to the midpoint to reduce the computational load of subsequent texture mapping. Specifically: All point clouds distributed within each triangular facet are combined to form a point cloud set; Taking any point cloud within the point cloud set as the center, select all point clouds belonging to the point cloud set within its neighborhood as the neighboring point clouds of the given point cloud. In this embodiment, the neighborhood is a neighborhood with a radius of [missing information]. The spherical neighborhood, where, for the first A triangular facet, neighborhood radius The selection method is as follows: based on the first Calculate the position coordinates of all vertices on the i-th triangular facet, and calculate the i-th... Half the radius of the inscribed circle of each triangular facet is used as the neighborhood radius. As another implementation method, the implementer can set it according to the actual situation.

[0028] The number of all neighboring point clouds of any given point cloud is counted, and the average number of all point clouds distributed within each triangular patch is calculated; the ratio of the number of any given point cloud to the average number is denoted as the relative comparison. Calculate the distance between any point cloud and the centroid of its corresponding triangular facet; In this embodiment, the distance is calculated by measuring the Euclidean distance between any point cloud and the centroid of its corresponding triangular facet. The calculation of both the Euclidean distance and the centroid is a well-known technique and will not be described in detail here.

[0029] The ratio of the relative ratio to the distance is used as the filtering value for any point cloud. The point cloud corresponding to the maximum screening value within each triangular facet is used as the feature point cloud; It should be noted that the larger the relative ratio, the higher the distribution density of the point cloud, reflecting the higher the information value it contains and the more local detail information it can provide. The centroid is the spatial geometric center of the triangular facet. The closer the point cloud is to the centroid, the smaller the distance, the more it represents the overall characteristics of the entire triangular facet. The larger the obtained screening value, the higher the representativeness and importance of the point cloud in the triangular facet. It reflects the two important characteristics of the point cloud being close to the geometric center and having a high density of neighboring point clouds. Selecting feature point clouds with high screening values ​​for texture mapping can effectively reduce texture mapping errors and enhance the realism and immersion of the virtual scene.

[0030] At this point, the feature point cloud of each triangular facet on the triangular mesh model is obtained.

[0031] Step 3: Based on the camera's pose information when the RGB images were captured, the RGB images containing feature point clouds are filtered to obtain all candidate images for each triangular facet. Each point cloud in each triangular facet is mapped onto each candidate image to obtain the mapping points of each point cloud on each candidate image. The deviation between the point cloud in each triangular facet and the camera's position coordinates when each candidate image was captured and the depth of the mapping points on the depth image is evaluated. Combined with the filtering values, the mapping deviation of each triangular facet on each candidate image is determined.

[0032] Furthermore, RGB-D cameras can simultaneously acquire RGB images and depth images, and the generated point clouds all have corresponding RGB image sets. Based on the blending method, texture mapping technology maps the triangular facets or point clouds of the triangular mesh model onto the RGB images, and then blends the mapped parts of all RGB images to obtain color information, realizing texture mapping of the triangular mesh model in the underground scene. However, due to the narrow underground space and the different and irregular spatial structures of tunnels, chambers, and mining faces, the construction process of the triangular mesh model itself is prone to geometric errors, and the blending method is prone to high facet blurring during texture mapping.

[0033] Therefore, based on this analysis, and based on each feature point cloud, all RGB images within the visible range of the RGB-D camera are extracted, specifically: Based on the camera's pose information during the capture of each RGB image frame, the feature point cloud of each triangular facet is transformed from the world coordinate system to the camera coordinate system and mapped onto the image plane. All RGB images with mapped points within the field of view of the RGB-D camera are extracted as candidate images for each triangular facet. In this embodiment, the specific process of acquiring candidate images is as follows: During the capture of each RGB image frame, the pose information of the RGB-D camera can be obtained, namely the camera's intrinsic and extrinsic parameter matrices. The extrinsic parameter matrix of the RGB-D camera is used to transform the position coordinates of the 3D feature point cloud in the world coordinate system to the camera coordinate system. The intrinsic parameter matrix is ​​then used to map the points in the camera coordinate system onto the 2D image plane, obtaining the pixel coordinates of the mapped points on the image plane. Assuming the first... The coordinates of the feature point cloud in the camera coordinate system are: The pixel coordinates of its corresponding mapping point are Check whether the mapping point is within the visible range of the image, i.e. ,and ,in, Image width, Given the image height, in this embodiment, the color image resolution is 1920×1080, then the image width is... The image height is 1920. The value is 1080. Next, check the... The coordinates of the feature point cloud in the camera coordinate system Whether it is less than or equal to the maximum shooting distance. In this embodiment, the maximum shooting distance of the RGB-D camera is 5.5 meters. Therefore, the option that satisfies this requirement is selected. ,and ,as well as All RGB images are denoted as candidate images, where, For the first The feature point cloud in the camera coordinate system The coordinates represent the distance between the feature point cloud and the camera when the RGB image was captured. This is the maximum shooting distance.

[0034] The process of transforming the feature point cloud from the world coordinate system to the camera coordinate system, as well as the acquisition of pixel coordinates, are well-known techniques and will not be elaborated here.

[0035] Secondly, the depth information of the mapping positions of each point cloud distributed within each triangular facet of the triangular mesh model in any candidate image is analyzed. Combined with the screening values, the mapping deviation is calculated, specifically: Based on the camera's pose information and position coordinates at the time of shooting any candidate image of each triangular facet, all point clouds in each triangular facet are transformed from the world coordinate system to the camera coordinate system and mapped onto the candidate image. The pixel coordinates of each point cloud in any candidate image are obtained, and the pixel points corresponding to the pixel coordinates of each point cloud in any candidate image are recorded as the mapping points of each point cloud in any candidate image. It should be noted that obtaining pixel coordinates is a well-known technique and will not be elaborated upon here.

[0036] The depth value of the point cloud corresponding to the mapping point in the depth image corresponding to any candidate image is used as the mapping depth of each point cloud in any candidate image. It should be noted that when an RGB-D camera takes a picture, one RGB image corresponds to one depth image. Therefore, each candidate image corresponds to one depth image.

[0037] In the world coordinate system, the distance between the position coordinates of each point cloud and the position coordinates of the camera when any candidate image was captured is calculated, and this distance is taken as the geometric distance between each point cloud and any candidate image. In this embodiment, the distance is calculated by measuring the Euclidean distance between the position coordinates of each point cloud and the position coordinates of the camera when any candidate image was captured. The calculation of the Euclidean distance is a well-known technique and will not be described in detail here.

[0038] Calculate the difference between the mapping depth and the geometric distance of each point cloud in any candidate image, as the relative deviation; In this embodiment, the absolute value of the difference between the mapping depth and the geometric distance of each point cloud in any candidate image is taken as the relative deviation.

[0039] It should be noted that the mapping depth reflects the observation depth of the point cloud from the shooting perspective of the candidate image, and the geometric distance reflects the actual geometric distance of the point cloud to the RGB-D camera in the world coordinate system. The larger the relative deviation, the greater the difference between the depth value of the point cloud in the image coordinates and the geometric distance, reflecting that there is a large deviation between the representation of the point cloud in the depth image and the actual three-dimensional spatial position.

[0040] Calculate the average of the filtering values ​​of all point clouds in the point cloud set corresponding to the triangular facet of each point cloud; If the filtering value of each point cloud is greater than the average value, then the mapping weight of each point cloud is the ratio of the filtering value of each point cloud to the average value; otherwise, the mapping weight of each point cloud is a preset value. In this embodiment, the preset value is 1. In other implementation methods, the implementer can set it according to the actual situation.

[0041] For each candidate image corresponding to each triangular facet, the mean of the product of the mapping weights and relative deviations of all point clouds distributed in each triangular facet is used as the mapping deviation of each triangular facet in each candidate image. It should be noted that the mapping weight reflects the importance of the point cloud in its respective triangular facet. The larger the mapping weight, the more likely the point cloud is located in a high-value region with dense point clouds and close to the center, containing rich neighborhood information. If the screening value of each point cloud is greater than the average value, it indicates that the point cloud is highly representative in the triangular facet, and its depth information is more important for the overall texture mapping and geometric reconstruction. If the screening value of each point cloud is less than or equal to the average value, that is, the smaller the mapping weight, the less representative the point cloud is in the triangular facet, and its depth information has a smaller impact on the overall texture mapping and geometric reconstruction. Its relative deviation should also contribute less to the mapping deviation. The larger the mapping deviation, the lower the degree of matching between the model surface corresponding to the triangular facet in the underground scene and the depth information of the candidate image, and the worse the consistency of the mapping geometry of the triangular facet on the candidate image.

[0042] Thus, the mapping deviation of each triangular facet in each candidate image is obtained.

[0043] Step 4: By considering the differences in brightness of the points mapped from the point cloud within each triangular facet on different candidate images, and the inconsistency between the normal vector of each triangular facet and the shooting angle of each candidate image, determine the texture distortion of each triangular facet in each candidate image; based on the mapping deviation and texture distortion, obtain the quality evaluation value of each triangular facet in each candidate image, filter the candidate images of each triangular facet, perform texture mapping on the triangular mesh model, and import it into the VR engine to construct a virtual underground environment.

[0044] The flowchart of the method for obtaining the texture distortion of each triangular facet in each candidate image provided in the embodiments of this application is as follows: Figure 2 As shown.

[0045] Furthermore, the lighting distribution in the underground environment is extremely uneven. Under complex lighting conditions, especially with a large number of artificial point light sources, such as miners' lamps and lighting lamps, these light sources can easily cause strong high-gloss reflections and shadows on the surface of underground objects. This exacerbates the color distortion of the texture mapping on the triangular mesh model corresponding to the underground objects, making it difficult to reflect the real appearance of the underground scene and reducing the realism of virtual training for safe production in the underground scene.

[0046] Based on the above analysis, the RGB-D camera acquires an RGB image with depth information. RGB images use a linear combination of three color components to represent color. To extract areas of strong highlight reflection in the image, the RGB image is converted to an HSL image, and its brightness information is extracted. Specifically: Each candidate image is converted from RGB space to HSL space, and the L component of each pixel in each candidate image in HSL space is extracted as the brightness value of each pixel. It should be noted that the process of converting an image from RGB space to HSL space is a well-known technique and will not be elaborated here.

[0047] Calculate the average brightness value of the points mapped to each candidate image from all point clouds distributed within each triangular facet, and use it as the texture mapping brightness of each triangular facet in each candidate image; It should be noted that the greater the brightness of the texture mapping, the higher the brightness of the area where the triangular facet is located in the candidate image.

[0048] Secondly, most objects exhibit both diffuse and specular reflection. When incident light reaches the object's surface, a portion of the incident light enters the object's interior, undergoes multiple reflections via a colorant medium, and then exits the surface, resulting in diffuse reflection containing the object's inherent color information. The other portion of the incident light is directly reflected by the object's surface, resulting in specular reflection. Therefore, diffuse reflection contains the object's inherent color information, and in texture mapping, it can better reflect the true color and texture details of the object's surface. Specular reflection, on the other hand, can cause bright spots on the object's surface during texture mapping, potentially obscuring the true texture details and distorting the texture mapping result. Consequently, the reflected light component in the image varies depending on the camera's shooting angle. By considering the consistency between the camera's shooting angle and the triangular facet, combined with the texture mapping brightness, the texture distortion is calculated as follows: In the world coordinate system, the vector pointing from the feature point cloud of each triangular facet to the position of the camera when each candidate image is captured is used as the view vector of each triangular facet in each candidate image. Calculate the cosine of the angle between the normal vector of each triangular facet and the view vector, and perform a positive mapping on the cosine value; In this embodiment, the specific process of positive mapping is as follows: positive mapping is performed using an exponential function, assuming the cosine value is denoted as... ,but The result is used as the positive mapping result of the cosine value, where, For an exponential function with the natural constant as the base, the cosine value is positively mapped to a value greater than 0 through a positive mapping process.

[0049] It should be noted that obtaining the normal vector is a well-known technique, and will not be elaborated upon here.

[0050] The average difference between the texture mapping brightness of each triangular facet in each candidate image and the texture mapping brightness of all other candidate images is used as the brightness deviation of each triangular facet in each candidate image. In this embodiment, the average of the absolute values ​​of the differences between the texture mapping brightness of each triangular facet in each candidate image and the texture mapping brightness of all other candidate images is taken as the brightness deviation of each triangular facet in each candidate image.

[0051] The ratio between the brightness deviation and the result of the positive mapping is used as the texture distortion of each triangular facet in each candidate image. It should be noted that a larger brightness deviation indicates a greater difference between the texture mapping brightness of each triangular facet in each candidate image and the average brightness of the other candidate images, which may be affected by the light source, resulting in highlight reflection or shadows. The larger the result of the positive mapping of the cosine value, the smaller the angle between the normal vector and the view vector of the triangular facet. This means that the camera's view angle is closer to the front when acquiring candidate images, and the candidate images contain fewer specular reflection components. The smaller the result of the positive mapping, the larger the angle between the normal vector and the view vector of the triangular facet. This means that the view angle is closer to the side or back, and the candidate images contain more specular reflection components. The texture distortion may be more severe. The greater the texture distortion, the worse the texture quality of the triangular facet in each candidate image. This indicates that its shooting angle is poor and it is more strongly affected by unspecified light sources.

[0052] Furthermore, to avoid blurring and artifacts caused by mixed textures, and to eliminate texture mapping color distortion caused by unspecified light sources in the underground scene, a high-quality texture image with low illumination components and geometric alignment with the triangular mesh model of the underground scene needs to be selected for texture mapping. Therefore, based on the mapping deviation and texture distortion, a quality evaluation value is determined, specifically: The result of the negative mapping of the product of the mapping deviation and the texture distortion is used as the quality evaluation value of each triangular facet in each candidate image. In this embodiment, the specific process of negative mapping is as follows: the reciprocal of the product of the mapping deviation and the texture distortion is used as the quality evaluation value of each triangular facet in each candidate image.

[0053] It should be noted that the higher the quality evaluation value, the higher the quality of the candidate image as a texture source for the triangular facet, and the more accurately it reflects the texture of the object's surface.

[0054] Candidate images corresponding to the maximum quality assessment value of each triangular facet are selected as texture images. Texture mapping is performed on all point clouds distributed in each triangular facet on the triangular mesh model. The texture-mapped triangular mesh model is then imported into a VR engine, such as Unity3D or Unreal Engine, to form an interactive VR application for use in underground safety training, equipment operation simulation, emergency drills, etc.

[0055] It should be noted that texture mapping technology is a well-known technology and will not be elaborated upon here.

[0056] It should be understood that, although Figure 1The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0057] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0058] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of this application, without departing from the content of the technical solution of this application, shall fall within the protection scope of the technical solution of this application.

Claims

1. A method for simulating and modeling underground safety production based on VR technology, characterized in that, The method includes the following steps: Acquire all RGB images of the underground scene and their corresponding depth images, as well as the camera's pose information and position coordinates when each RGB image is captured, to generate a dense point cloud and construct a triangular mesh model; Analyze the distribution density of point clouds within each triangular facet of the triangular mesh model, as well as the distribution position of each point cloud within the triangular facet. Calculate the filtering value of each point cloud, filter the point clouds within each triangular facet, and obtain the feature point cloud of each triangular facet. Based on the camera's pose information during RGB image capture, the RGB images containing feature point clouds are filtered to obtain all candidate images for each triangular facet. Then, each point cloud in each triangular facet is mapped onto each candidate image to obtain the mapping points of each point cloud on each candidate image. The deviation between the point cloud distance within each triangular facet and the camera position coordinates when each candidate image was captured, and the depth of the mapped point on the depth image, is evaluated. Combined with the screening value, the mapping deviation of each triangular facet in each candidate image is determined. The texture distortion of each triangular facet in each candidate image is determined by the difference in brightness of the points mapped from the point cloud within each triangular facet on different candidate images, and the inconsistency between the normal vector of each triangular facet and the shooting angle of each candidate image. Based on the mapping deviation and texture distortion, the quality evaluation value of each triangular facet in each candidate image is obtained. Candidate images of each triangular facet are then selected, texture mapping is performed on the triangular mesh model, and the model is imported into the VR engine to construct a virtual underground environment.

2. The method for simulating and modeling downhole safety production based on VR technology as described in claim 1, characterized in that, The calculation of the filter values ​​for each point cloud includes: All point clouds distributed within each triangular facet are formed into a point cloud set; taking any point cloud in the point cloud set as the center, all point clouds in its neighborhood that belong to the point cloud set are selected as the neighboring point clouds of the given point cloud. The number of all neighboring point clouds of any given point cloud is counted, and the average number of all point clouds distributed within each triangular patch is calculated; the ratio of the number of any given point cloud to the average number is denoted as the relative comparison. Calculate the distance between any point cloud and the centroid of its corresponding triangular facet; The ratio of the relative comparison to the distance is used as the filtering value for any point cloud.

3. The method for simulating and modeling downhole safety production based on VR technology as described in claim 1, characterized in that, The step of obtaining the feature point cloud of each triangular facet includes: taking the point cloud corresponding to the maximum screening value within each triangular facet as the feature point cloud.

4. The method for simulating and modeling downhole safety production based on VR technology as described in claim 1, characterized in that, The process of acquiring all candidate images of each triangular facet and mapping each point cloud within each triangular facet onto each candidate image, and acquiring the mapping points of each point cloud on each candidate image, includes: based on the camera's pose information during the capture of each RGB image frame, transforming the feature point cloud of each triangular facet from the world coordinate system to the camera coordinate system and mapping it onto the image plane; acquiring the pixel coordinates of each feature point cloud in each RGB image; and recording the pixel points represented by the pixel coordinates as mapping points; extracting all RGB images within the camera's field of view of the mapping points as candidate images of each triangular facet; and transforming all point clouds distributed within each triangular facet from the world coordinate system to the camera coordinate system and mapping them onto the candidate images, thereby acquiring the mapping points of each point cloud within each triangular facet on each candidate image.

5. The method for simulating and modeling downhole safety production based on VR technology as described in claim 2, characterized in that, The determination of the mapping deviation of each triangular facet in each candidate image includes: The depth value corresponding to the point mapping point in the depth image corresponding to each candidate image is used as the mapping depth of each point cloud in each candidate image. In the world coordinate system, the distance between the position coordinates of each point cloud and the position coordinates of the camera when each candidate image was captured is calculated, and this distance is used as the geometric distance between each point cloud and each candidate image. Calculate the difference between the mapping depth and the geometric distance of each point cloud in each candidate image, as the relative deviation; Analyze the deviation between the selection value of each point cloud and the selection values ​​of all point clouds in the point cloud set corresponding to its respective triangle, and calculate the mapping weight of each point cloud. The mapping deviation is the average of the product of the mapping weights and relative deviations of all point clouds distributed within each triangular patch.

6. The method for simulating and modeling downhole safety production based on VR technology as described in claim 5, characterized in that, The calculation of the mapping weight of each point cloud includes: calculating the average value of the filtering values ​​of all point clouds in the point cloud set corresponding to the triangle facet to which each point cloud belongs; if the filtering value of each point cloud is greater than the average value, then the mapping weight of each point cloud is the ratio of the filtering value of each point cloud to the average value, otherwise, the mapping weight of each point cloud is a preset value.

7. The method for simulating and modeling downhole safety production based on VR technology as described in claim 1, characterized in that, Determining the texture distortion of each triangular facet in each candidate image includes: Each candidate image is converted from RGB space to HSL space, and the L component of each pixel in each candidate image in HSL space is extracted as the brightness value of each pixel. The differences in brightness values ​​of the points mapped from the point cloud distributed within each triangular facet in different candidate images are analyzed, and the brightness deviation of each triangular facet in each candidate image is calculated. In the world coordinate system, the vector pointing from the feature point cloud of each triangular facet to the position of the camera when each candidate image is captured is used as the view vector of each triangular facet in each candidate image. Calculate the cosine of the angle between the normal vector of each triangular facet and the view vector, and perform a positive mapping on the cosine value; The texture distortion is the ratio between the brightness deviation and the result of the positive mapping.

8. The method for simulating and modeling downhole safety production based on VR technology as described in claim 7, characterized in that, The calculation of the brightness deviation of each triangular facet in each candidate image includes: Calculate the average brightness value of the points mapped to each candidate image from all point clouds distributed within each triangular facet, and use it as the texture mapping brightness of each triangular facet in each candidate image; The brightness deviation is the average difference between the texture mapping brightness of each triangular facet in each candidate image and the texture mapping brightness of all other candidate images.

9. The method for simulating and modeling downhole safety production based on VR technology as described in claim 1, characterized in that, The quality assessment value is the result of negative mapping of the product of the mapping deviation and the texture distortion.

10. The method for simulating and modeling downhole safety production based on VR technology as described in claim 1, characterized in that, The process of filtering candidate images for each triangular facet and performing texture mapping on the triangular mesh model includes: selecting the candidate image with the highest quality evaluation value from all candidate images corresponding to each triangular facet as the texture image; performing texture mapping on all point clouds distributed within each triangular facet on the triangular mesh model; and importing the texture-mapped triangular mesh model into the VR engine.

Images