A structure semantic map construction method for an underground weak light and weak texture environment

Abstract

The present application relates to the technical field of computer vision three-dimensional modeling, in particular to a structural semantic map construction method for underground weak light and weak texture environment, which comprises the following steps: fusing a structural semantic parameterized line detection and verification model of a traditional method and a Transfomer; establishing a geometric primitive half-plane search method guided by structural information direction; establishing a neighborhood greedy expansion algorithm based on a geometric primitive model; optimizing a geometric primitive pose and boundary one by one; fusing a point cloud map and structural information to establish a semantic map fused with geometric structure primitives. According to the present application, a variety of underground space three-dimensional maps in weak light and weak texture environment can be established without manual intervention, and complete CAD operable models of regular objects such as pipelines, columns and stairs can be established, which has important application value in various weak light and weak texture environments such as urban underground pipe corridors and subway underground tunnels, and can solve the problems of long artificial modeling cycle, low three-dimensional modeling precision in weak texture and weak light, etc.

Description

Technical Field

[0001] This invention relates to the field of computer vision 3D modeling technology, and in particular to a method for constructing structural semantic maps in underground low-light and low-texture environments. Background Technology

[0002] 3D spatial perception technology based on one or more sensors such as LiDAR, depth cameras, and RGB cameras has significant application prospects and market value in fields such as scene modeling, robot navigation, robot target grasping, intelligent detection, and digital twins. After years of exploration by academia and industry, modeling methods based on the fusion of vision, radar, and other sensors have made great progress, but they still struggle to operate stably in complex scenarios such as underground spaces. Underground space 3D perception is a key technological support for applications such as intelligent urban governance, underground infrastructure perception in battlefield environments, and underground mine inspection. Therefore, improving scene modeling capabilities in low-light and low-texture environments is of significant practical importance.

[0003] Currently, scene modeling under low-light and low-texture conditions relies on expensive 3D scanners, resulting in high costs. Furthermore, while point clouds obtained through 3D scanning can create a 3D model of the scene after post-processing such as triangular patching, this model lacks the recognition and complete modeling of regular structures within the scene. Obtaining complete CAD models of key areas is crucial for improving model visualization, simplifying subsequent manual modeling, and facilitating the rapid implementation of related intelligent applications. Multi-sensor fusion 3D modeling is generally carried out as follows: First, a robot or human equipped with sensing devices such as cameras, radar, inertial sensors, and depth cameras moves in the target scene to collect multimodal sensing information with time synchronization in real time. After the collection is completed, different inter-frame data association algorithms are used according to the different types of sensor combinations used to obtain the translation and rotation matrices between adjacent data frames. The rotation and translation matrices can be optimized in a long sequence. Then, the scene 3D point cloud model is calculated by using the rotation and translation matrices and the intrinsic and extrinsic parameters of the sensors. Finally, based on the 3D point cloud model, professional 3D point cloud editing engineers perform triangulation, manual editing, texture mapping, and other steps to build a scene CAD 3D model.

[0004] 3D scene enhancement based on structure recognition and regular geometric contours has been researched and applied in related fields, helping to achieve 3D CAD modeling and scene enhancement in complex scenes. However, there are still many shortcomings. On the one hand, in the parameter extraction of structural semantic lines, most existing deep learning line segment detection algorithms do not distinguish between structural lines and texture lines, making them difficult to directly use for 3D structure recognition and enhancement. Furthermore, breakthroughs have not yet been achieved in how to represent and learn heterogeneous structural semantic lines within a deep learning framework. On the other hand, in the modeling of regular geometric objects, deep learning geometric primitive segmentation methods based on point clouds require significant computational resources, and the segmented point clouds still need to be fitted with geometric primitive parameters using traditional methods. In addition, these methods fail to effectively integrate and utilize image and point cloud data, while our human visual system can draw geometric contour lines from a scene and effectively infer the 3D structure of the scene. Summary of the Invention

[0005] This invention provides a method and apparatus for constructing structural semantic maps in underground low-light and low-texture environments. The technical solution is as follows:

[0006] On the one hand, a method for constructing structural semantic maps in underground low-light and low-texture environments is provided. This method is implemented by an electronic device and includes:

[0007] S1. A sensor based on vision and three-dimensional depth perception acquires multimodal perception information of the target scene, wherein the multimodal perception information includes RGB visual images and depth perception information, and the depth perception information includes ordered point clouds and unordered point clouds;

[0008] S2. Using the ELSDc line segment and elliptical arc detection method, obtain the set of regular parameterized lines of the visual image, and perform embedding encoding on the line segments and elliptical arcs in the set of regular parameterized lines to obtain an embedding vector with a length of 256 dimensions.

[0009] S3. Visual features are extracted from the visual image using the image feature extraction module;

[0010] S4. Input the embedding vector and the visual features into the trained parameterized line discrimination network model to obtain a set of structural semantic lines;

[0011] S5. Preprocess the ordered and disordered point clouds in the depth perception information. By indexing the structural semantic line segments and the pixels they pass through in the ordered and disordered point clouds, associate the two-dimensional structural lines in the set of structural semantic lines with the pixels of the visual image to obtain the association relationship between the two-dimensional structural lines and the three-dimensional points.

[0012] S6. Based on the relationship between the two-dimensional structural lines and the three-dimensional points, perform three-dimensional space fitting on the two-dimensional structural lines to obtain three-dimensional structural lines.

[0013] S7. Perform a geometric primitive half-plane search based on the three-dimensional structural lines to determine whether the point cloud on both sides of each three-dimensional structural line satisfies the constraint of the regular surface. If it does, expand the neighborhood of the geometric primitives according to the greedy algorithm to determine the largest regular region.

[0014] S8. Merge the largest regular region with the point cloud set to construct structured map data.

[0015] Optionally, step S2, using the ELSDc line segment and elliptical arc detection method, obtains a set of regularly parameterized lines from the visual image, including:

[0016] S21. Using the ELSDc line segment and elliptical arc detection method, obtain the set of regular parameterized lines {L, A} of the visual image, where {L, A} is as shown in the following formula (1):

[0017] {L, A} = ELSDc( Irgb (1)

[0018] L i ={(x 1i ,y 1i ,x 2i ,y 2i |i = 1, 2, ..., N}

[0019]

[0020] Among them, I rgb Represents the visual image, (x 1i y 1i x 2i y 2i ) represents the parameter of the i-th line segment, x 1i y 1i The two-dimensional coordinates, x, are used to represent an endpoint of the i-th line segment. 2i y 2i The two-dimensional coordinates used to represent the other endpoint of the i-th line segment, where N represents the total number of line segments. The parameter x represents the j-th elliptic arc. 1j y 1j a is used to represent the two-dimensional coordinates of the center point of the elliptical arc. j b j θ represents the semi-axis lengths of the major and minor axes, respectively. j Indicates the rotation angle. Indicates the starting angle. The endpoint angle is represented by , and M represents the total number of elliptical arcs.

[0021] Optionally, step S2 performs embedding encoding on the line segments and elliptical arcs in the set of regular parameterized lines to obtain an embedding vector of length 256, including:

[0022] S22. According to the following formula (2), two different encoders are used to embed the line segment and the elliptical arc, and the encoded output is a vector with a length of 256 dimensions:

[0023]

[0024]

[0025] Wherein, LineEncoder() represents a line segment position encoder, L i This represents the i-th line segment, and ArcEncoder() represents a circular arc position encoder. i Let i represent the i-th elliptic arc.

[0026] Optionally, the image feature extraction module is a ResNet-50;

[0027] The step S3 involves extracting visual features from the visual image using an image feature extraction module, including:

[0028] The visual image is input into the ResNet-50, and a sin position encoder and a cos position encoder are used to embed the position at the scale of the visual image. The position embedding vector is then concatenated with the visual image to obtain the visual features of the visual image.

[0029] Optionally, the parameterized line discrimination network model is a Transformer model;

[0030] The training process of the parameterized line discrimination network model includes:

[0031] By using a differentiable renderer, the discrimination probability is used as the transparency value, and the line parameters are used as the line drawing constraints. The rendered image and the real structure semantic annotation binary image are subjected to L2 loss to train the parameterized line discrimination network model.

[0032] Optionally, step S5 preprocesses the ordered and unordered point clouds in the depth-sensing information, including:

[0033] Based on the optimal accuracy range of the sensor, pass-through and conditional filters are used to preprocess the depth sensing information to extract the effective sensing area of ​​the sensor;

[0034] For the ordered point cloud in the depth sensing information, a bilateral filter is used to denoise the surface of the spatial three-dimensional model. For the disordered point cloud in the depth sensing information, a Gaussian filter is used for smoothing and denoising. For cloud outliers, statistical filtering is used for removal.

[0035] Optionally, step S5 associates the two-dimensional structural lines in the set of structural semantic lines with the pixels of the visual image by indexing the structural semantic line segments and the pixels they pass through in the ordered and unordered point clouds, thereby obtaining the association relationship between the two-dimensional structural lines and the three-dimensional points, including:

[0036] For each structural line corresponding to an ordered point cloud, the indices of all pixels passed through by the structural line segment are associated with the index of the line segment to establish a set of three-dimensional coordinate points covered by the line segment.

[0037] For disordered point clouds, the sensor is used for parameter calibration to establish a correspondence between the pixels in the image coordinate system and the three-dimensional points of the disordered point cloud. The indexes of all pixels passed through by each structural line corresponding to the disordered point cloud are associated with the index of the line segment to establish a set of three-dimensional coordinate points covered by the line segment.

[0038] Optionally, step S7, which involves performing a geometric primitive half-plane search based on the three-dimensional structural lines to determine whether the point clouds on both sides of each three-dimensional structural line satisfy the constraints of a regular surface, includes:

[0039] For a line segment in a three-dimensional structure, for the two parts on both sides of the line segment, parameter fitting is performed simultaneously using a plane and a cylindrical surface. If either part on both sides of the line segment conforms to the fitting of a plane or a cylindrical surface, then it is determined that the point cloud on both sides of the line segment satisfies the constraint of a regular surface, that is, there exists a corresponding regular geometric primitive.

[0040] For the elliptical arc in the three-dimensional structural lines, for the part inside the elliptical arc, parameter fitting is performed using a plane. If there is a three-dimensional point cloud distribution that conforms to the plane, then the point cloud inside the arc is determined to satisfy the constraint of a regular surface, that is, the geometric primitive inside the elliptical arc is a plane. For the part outside the elliptical arc, parameter fitting is performed using a cylindrical surface. If there is a three-dimensional point cloud distribution that conforms to the plane, then the point cloud outside the arc is determined to satisfy the constraint of a regular surface, that is, the geometric primitive outside the elliptical arc is a cylindrical surface.

[0041] On the other hand, a structural semantic map construction device for underground low-light and low-texture environments is provided. This device is applied to a method for constructing structural semantic maps for underground low-light and low-texture environments. The device includes:

[0042] The acquisition module is used to acquire multimodal perception information of the target scene based on vision and three-dimensional depth perception sensors. The multimodal perception information includes RGB visual images and depth perception information, and the depth perception information includes ordered point clouds and unordered point clouds.

[0043] The embedding encoding module is used to obtain a set of regular parameterized lines in the visual image using the ELSDc line segment and elliptical arc detection method, and to perform embedding encoding on the line segments and elliptical arcs in the set of regular parameterized lines to obtain an embedding vector with a length of 256 dimensions.

[0044] The feature extraction module is used to extract visual features from the visual image through the image feature extraction module;

[0045] The discrimination module is used to input the embedding vector and the visual features into the trained parameterized line discrimination network model to obtain a set of structural semantic lines;

[0046] The association module is used to preprocess the ordered and disordered point clouds in the depth perception information. By indexing the structural semantic line segments of the ordered and disordered point clouds and the pixels they pass through, the two-dimensional structural lines in the set of structural semantic lines are associated with the pixels of the visual image to obtain the association relationship between the two-dimensional structural lines and the three-dimensional points.

[0047] The fitting module is used to perform three-dimensional space fitting on the two-dimensional structural lines based on the relationship between the two-dimensional structural lines and the three-dimensional points to obtain three-dimensional structural lines.

[0048] The search module is used to perform a geometric primitive half-plane search based on the three-dimensional structural lines to determine whether the point cloud on both sides of each three-dimensional structural line satisfies the constraints of the regular surface. If it does, the geometric primitive neighborhood is expanded according to the greedy algorithm to determine the largest regular region.

[0049] The fusion module is used to merge the largest identified regular region with the point cloud set to construct structured map data.

[0050] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above-described method for constructing structural semantic maps in underground low-light and low-texture environments.

[0051] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the above-described method for constructing structural semantic maps in underground low-light and low-texture environments.

[0052] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0053] Compared to structure recognition and enhancement methods based on point cloud fitting, this invention utilizes both image and point cloud information, extracting structural semantic lines as clues to discover regular structures. On one hand, it reduces the enormous computational burden and risk of inaccuracies required for point cloud segmentation and fitting. On the other hand, the structural line recognition and determination method proposed in this invention, compared to methods based on image feature points, can build 3D maps of various underground spaces in low-light and low-texture environments without manual intervention. It can also create complete, operable CAD models for regular objects such as pipelines, columns, and stairs. Even in poorly lit and textured underground spaces such as urban underground utility tunnels and subway tunnels, it works well, solving problems such as long manual modeling cycles and low accuracy in 3D modeling under weak texture and low lighting conditions. Furthermore, through cross-frame structural line matching, this invention can also be used for robot visual localization and navigation in low-texture and low-light environments. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart of a method for constructing structural semantic maps in underground low-light and low-texture environments, provided by an embodiment of the present invention.

[0056] Figure 2 This is a flowchart of a method for constructing structural semantic maps in a low-light, low-texture underground environment, provided by an embodiment of the present invention.

[0057] Figure 3 This is an architecture diagram of a parameterized line discrimination network model provided in an embodiment of the present invention;

[0058] Figure 4 This is a schematic diagram illustrating an example of a geometric primitive half-plane search method provided in an embodiment of the present invention;

[0059] Figure 5 This is a block diagram of a structural semantic map construction device for underground low-light and low-texture environments provided in an embodiment of the present invention;

[0060] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0061] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0062] This invention provides a method for constructing structural semantic maps in underground low-light and low-texture environments. This method can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The flowchart shown illustrates the method for constructing structural semantic maps in underground low-light and low-texture environments. Figure 2 The flowchart shown illustrates that the processing flow of this method may include the following steps:

[0063] S1. Based on vision and 3D depth perception sensors, acquire multimodal perception information of the target scene.

[0064] The multimodal perception information includes RGB visual images and depth perception information, which includes ordered point clouds and unordered point clouds.

[0065] In one feasible implementation, the vision and 3D depth perception-based sensor can be constructed by one or more sensors such as LiDAR, depth camera, and RGB camera. For example, the minimum sensor solution can be a single RGBD camera, which is suitable for the combination of RGB camera with other auxiliary positioning sensors such as LiDAR. It requires that all sensors have been calibrated, including internal and external parameters.

[0066] S2. Using the ELSDc line segment and elliptical arc detection method, obtain a set of regular parameterized lines in the visual image. Embed the line segments and elliptical arcs in the set of regular parameterized lines to obtain an embedding vector with a length of 256 dimensions.

[0067] Optionally, the specific operation of S2 may include the following steps S21-S22:

[0068] S21. Using the ELSDc line segment and elliptical arc detection method, obtain the set of regular parameterized lines {L, A} of the visual image, where {L, A} is as shown in the following formula (1):

[0069] {L, A} = ELSDc(I rgb (1)

[0070] L i ={(x 1i ,y 1i ,x 2i ,y 2i |i = 1, 2, ..., N}

[0071]

[0072] Among them, Irgb Represents a visual image, (x 1i y 1i x 2i y2i) represents the parameter of the i-th line segment, x 1i y 1i The two-dimensional coordinates, x, are used to represent an endpoint of the i-th line segment. 2i y 2i The two-dimensional coordinates used to represent the other endpoint of the i-th line segment, where N represents the total number of line segments. The parameter x represents the j-th elliptic arc. 1j y 1j a is used to represent the two-dimensional coordinates of the center point of the elliptical arc. j b j θ represents the semi-axis lengths of the major and minor axes, respectively. j Indicates the rotation angle. Indicates the starting angle. The endpoint angle is represented by , and M represents the total number of elliptical arcs.

[0073] S22. According to the following formula (2), two different encoders are used to embed the line segment and the elliptical arc, and the encoded output is a vector with a length of 256 dimensions:

[0074]

[0075]

[0076] Where LineEncoder() represents a line segment position encoder, L i This represents the i-th line segment, and ArcEncoder() represents the arc position encoder. i Let i represent the i-th elliptic arc.

[0077] In one feasible implementation, for line segment position encoding, rectangular box encoding, which is often used in Transformer, is used because they are similar in dimensionality and geometric meaning; for arc encoding, the minimum enclosing rotation rectangle of the arc segment is first solved using the arc parameters, and the four coordinate points of the rotation rectangle, i.e., the eight parameters, are used as inputs to the arc encoder, which are also mapped to a 256-dimensional embedding vector.

[0078] S3. Visual features are extracted from the visual image using the image feature extraction module.

[0079] Optionally, the image feature extraction module can be ResNet-50. Based on ResNet-50, the specific operation process of S3 can be as follows:

[0080] The visual image is input into ResNet-50, and the sin position encoder and cos position encoder are used to embed the position at the scale of the visual image. The position embedding vector is concatenated with the visual image to obtain the visual features of the visual image.

[0081] S4. Train the parameterized line discrimination network model to be trained.

[0082] Optionally, in order to design an end-to-end parametric line discrimination network, the parametric line discrimination network model can be set as a transformer model. The transformer is used to distinguish line segments and elliptical arcs. The modeling, perceptual clustering and joint discrimination of parametric lines are realized through an overall "encoder-decoder", self-attention, joint query and reasoning.

[0083] Line segments and arc embeddings are encoded as queries, and image features are used as keys. A parametric line classifier is constructed using a Transformer. When training the Transformer-based parametric line discrimination network model, a differentiable renderer is used, with the discrimination probability used as the transparency value and the line parameters as drawing constraints. The rendered image is then subjected to L2 loss with the real structural semantic annotation binary image. To avoid the workload of manually annotating heterogeneous parametric lines, based on the BIPED edge detection dataset, non-structural semantic lines are removed using image editing software with an eraser tool, retaining only edges that conform to human visual perception experience and reflect the scene's geometric structure for semi-supervised training of the classifier. Specifically, during training, the classifier scores each structural line embedding, and the score is used as the transparency parameter. Combined with the line parameters, a differentiable renderer is used to render the predicted image, which is then subjected to L2 loss with the edited BIPED real structural semantic line values.

[0084] S5. Input the embedded vectors and visual features into the trained parameterized line discrimination network model to obtain a set of structural semantic lines.

[0085] In one feasible implementation, the embedding vectors obtained by embedding and encoding line segments and elliptical arcs are input together with the visual features output by ResNet-50 into the trained parameterized line discrimination network model, referencing... Figure 3 It should be noted that, unlike ordinary parametric line detection such as line segments or ellipses, the structural semantic lines in this embodiment of the invention refer to regular lines that reflect changes in the three-dimensional structure of the scene, rather than regular structural lines generated by texture changes, such as the boundary lines of a flat wall surface rather than the texture boundaries generated by graffiti on the wall surface.

[0086] S6. Preprocess the ordered and disordered point clouds in the depth perception information. By indexing the structural semantic line segments and the pixels they pass through in the ordered and disordered point clouds, associate the two-dimensional structural lines in the set of structural semantic lines with the pixels of the visual image to obtain the association relationship between the two-dimensional structural lines and the three-dimensional points.

[0087] Optionally, the specific operation of S6 may include the following steps S61-S64:

[0088] S61. Based on the optimal accuracy range of the sensor, use pass-through and conditional filters to preprocess the depth sensing information and extract the effective sensing area of ​​the sensor.

[0089] In one feasible implementation, when acquiring point cloud and depth information, some noise points will inevitably appear in the point cloud data due to the influence of equipment accuracy, operator experience, and environmental factors. The role of filtering is to utilize the low-frequency characteristics of the data to remove outliers and to smooth the data or extract specific frequency band features. For the optimal accuracy range of the sensor, pass-through and conditional filters are used for preprocessing to extract the effective sensing area of ​​the sensor.

[0090] S62. For ordered point clouds in depth-sensing information, use a bilateral filter to denoise the surface of the spatial 3D model. For disordered point clouds in depth-sensing information, use a Gaussian filter for smoothing and denoising. For cloud outliers, use statistical filtering to remove them.

[0091] In one feasible implementation, a bilateral filter is used to denoise the surface of the 3D spatial model for ordered point clouds while preserving the geometric features of the point cloud data, avoiding excessive smoothing of the 3D point cloud data. For unordered point clouds, a Gaussian filter is used for smoothing and denoising. Finally, statistical filtering is used to remove outliers from the point cloud.

[0092] S63. For each structural line corresponding to the ordered point cloud, associate the indices of all pixels passed through by the structural line segment with the index of the line segment to establish a set of three-dimensional coordinate points covered by the line segment.

[0093] S64. For disordered point clouds, use sensors to perform parameter calibration, establish a correspondence between the pixels in the image coordinate system and the three-dimensional points of the disordered point cloud, associate the indexes of all pixels passed through by each structural line corresponding to the disordered point cloud with the indexes of the line segments, and establish a set of three-dimensional coordinate points covered by the line segments.

[0094] In one feasible implementation, depth-sensing information is preprocessed and associated with image pixels to establish a one-to-one mapping between pixel spatial points and 3D spatial points. This step is performed using the calibration parameters of the sensor system. For 2D pixels where no mapping can be found, their corresponding 3D coordinate index is denoised as Nan. During depth data preprocessing, the data is first truncated according to the effective working range of the sensor. Then, bilateral and Gaussian filters are used to denoise the point cloud. If the number of 3D points and 2D pixel coordinates that can be associated is K, then the 2D point p can be established using the sensor calibration system parameters M. xy (i) and the three-dimensional point P xyz The correspondence of (i):

[0095] p xy (i)=MgP xyz (i); i = 1, 2, LK

[0096] S7. Based on the relationship between two-dimensional structural lines and three-dimensional points, three-dimensional spatial fitting is performed on the two-dimensional structural lines to obtain three-dimensional structural lines.

[0097] S8. Perform a half-plane search of geometric primitives based on the three-dimensional structural lines to determine whether the point clouds on both sides of each three-dimensional structural line satisfy the constraints of the regular surface. If they do, expand the neighborhood of the geometric primitives according to the greedy algorithm to determine the largest regular region.

[0098] Optionally, S8 performs a half-plane search of geometric primitives based on 3D structural lines to determine whether the point clouds on both sides of each 3D structural line satisfy the constraints of a regular surface. The specific operation can be as follows:

[0099] For a line segment in a three-dimensional structure, for the two parts on both sides of the line segment, parameter fitting is performed using both plane and cylindrical surfaces. If either part on both sides of the line segment conforms to the fitting of the plane or cylindrical surface, then it is determined that the point cloud on both sides of the line segment satisfies the constraint of the regular surface, that is, there exists a corresponding regular geometric primitive.

[0100] For the elliptical arc in the three-dimensional structure lines, for the part inside the elliptical arc, parameter fitting is performed using a plane. If there is a three-dimensional point cloud distribution that conforms to the plane, then the point cloud inside the arc is determined to satisfy the constraint of a regular surface, that is, the geometric primitive inside the elliptical arc is a plane. For the part outside the elliptical arc, parameter fitting is performed using a cylindrical surface. If there is a three-dimensional point cloud distribution that conforms to the plane, then the point cloud outside the arc is determined to satisfy the constraint of a regular surface, that is, the geometric primitive outside the elliptical arc is a cylindrical surface.

[0101] One feasible implementation method is, for example Figure 4As shown, the basic idea of ​​geometric primitive half-plane search is to cluster continuous point clouds belonging to regular structural surfaces by analyzing the distribution of structural semantic lines in three-dimensional space on both sides, thereby achieving the goal of fast structural primitive fitting. The basic operation involves using a multi-scale grid sampling method to initially determine the surface types on both sides. For candidate sides that conform to planar or cylindrical geometric constraints, the optimal subset of points is selected from the set of points containing gross errors based on the distance and minimum criteria to fit reliable initial values ​​of the model. A robust Z-score method is then used to iteratively eliminate gross errors. Specifically:

[0102] For two types of geometric primitives—planar and cylindrical surfaces—for a line segment, we define a line segment that divides the region into two parts: an upper half-plane and a lower half-plane. Within these two parts, starting from the dividing line, we simultaneously perform parameter fitting using both planar and cylindrical surfaces. If a geometric distribution conforms to either of these, then a corresponding regular geometric primitive is considered to exist on the corresponding line segment side. For a two-dimensional arc, we define an arc that divides the region into two parts: an inner arc and an outer arc. Within the inner arc, starting from the dividing line, we perform parameter fitting using a planar surface. If a three-dimensional point cloud distribution conforms to a planar surface, then the geometric model within the inner arc is considered to exist as a planar surface. For the outer arc, starting from the dividing line, we perform parameter fitting using a cylindrical surface. If a three-dimensional point cloud distribution conforms to a cylindrical surface, then the geometric model outside the outer arc is considered to exist as a cylindrical surface.

[0103] Optionally, a greedy algorithm can be used to expand the neighborhood of geometric primitives to determine the largest regular region. The specific operation can be as follows:

[0104] For regions determined to be regular half-planes, longitudinal expansion is performed along the line segment direction until the 3D coordinates of the point cloud of the continuous region no longer satisfy the geometric primitive model of the region.

[0105] In one feasible implementation, after determining the geometric primitive types on both sides of the geometric line, a greedy algorithm is used to expand the region. During the expansion, the process proceeds continuously from the inside out. When an outlier point is encountered that does not conform to the three-dimensional primitive model established by the "geometric primitive half-plane search", the search is terminated in that direction until all directions are searched. Thus, an index relationship belonging to the structural line is established for all three-dimensional points in the region.

[0106] It should be noted that if the point clouds on both sides of a certain 3D structural line do not meet the constraints of a regular surface, the point clouds will not be processed for the time being.

[0107] S9. Merge the largest regular region with the point cloud set to construct structured map data.

[0108] In one feasible implementation, for all 3D points belonging to the structural lines, the least squares method is used to optimize the model parameters using their corresponding geometric primitive types, such as planes or cylinders, ultimately obtaining a parametric geometric model belonging to that part. Depending on the application, appropriate structural information map construction methods can be adopted. For example, by using aggregate primitive parameters, the aggregate primitive model can be placed in a point cloud map using rotation and translation matrices, enabling structural visualization and supporting tasks such as robot operation.

[0109] In this embodiment of the invention, multimodal perception information of the target scene is acquired; a set of regular parameterized lines in the visual image is obtained using the ELSDc line segment and elliptical arc detection method; the line segments and elliptical arcs in the set of regular parameterized lines are embedded and encoded to obtain embedding vectors; visual features are extracted from the visual image using an image feature extraction module; the embedding vectors and visual features are input into a trained parameterized line discrimination network model to obtain a set of structural semantic lines; the ordered and unordered point clouds in the depth perception information are preprocessed, and the structural semantic line segments of the ordered and unordered point clouds are correlated with the pixels they pass through. The index is used to associate the two-dimensional structural lines in the set of structural semantic lines with the pixels of the visual image, thus obtaining the association relationship between the two-dimensional structural lines and the three-dimensional points. Based on the association relationship between the two-dimensional structural lines and the three-dimensional points, the two-dimensional structural lines are fitted in three-dimensional space to obtain three-dimensional structural lines. Based on the three-dimensional structural lines, a geometric primitive half-plane search is performed to determine whether the point cloud on both sides of each three-dimensional structural line satisfies the constraint of the regular surface. If it does, the neighborhood of the geometric primitive is expanded according to the greedy algorithm to determine the largest regular region. The determined largest regular region is fused with the point cloud set to construct the structured map data.

[0110] Compared to structure recognition and enhancement methods based on point cloud fitting, this invention utilizes both image and point cloud information, extracting structural semantic lines as clues to discover regular structures. On one hand, it reduces the enormous computational burden and risk of inaccuracies required for point cloud segmentation and fitting. On the other hand, the structural line recognition and determination method proposed in this invention, compared to methods based on image feature points, can build 3D maps of various underground spaces in low-light and low-texture environments without manual intervention. It can also create complete, operable CAD models for regular objects such as pipelines, columns, and stairs. Even in poorly lit and textured underground spaces such as urban underground utility tunnels and subway tunnels, it works well, solving problems such as long manual modeling cycles and low accuracy in 3D modeling under weak texture and low lighting conditions. Furthermore, through cross-frame structural line matching, this invention can also be used for robot visual localization and navigation in low-texture and low-light environments.

[0111] Figure 5This is a block diagram illustrating a structural semantic map construction apparatus for a low-light, low-texture underground environment, according to an exemplary embodiment. The apparatus is used for a method of constructing a structural semantic map for a low-light, low-texture underground environment, wherein:

[0112] The acquisition module 510 is used to acquire multimodal perception information of a target scene based on a sensor that uses vision and three-dimensional depth perception. The multimodal perception information includes RGB visual images and depth perception information, and the depth perception information includes ordered point clouds and unordered point clouds.

[0113] The embedding encoding module 520 is used to obtain a set of regular parameterized lines in the visual image through the ELSDc line segment and elliptical arc detection method, and to embed and encode the line segments and elliptical arcs in the set of regular parameterized lines to obtain an embedding vector with a length of 256 dimensions.

[0114] The feature extraction module 530 is used to extract visual features from the visual image through the image feature extraction module;

[0115] The discrimination module 540 is used to input the embedding vector and the visual features into the trained parameterized line discrimination network model to obtain a set of structural semantic lines;

[0116] The association module 550 is used to preprocess the ordered point cloud and the disordered point cloud in the depth perception information. By indexing the structural semantic line segments of the ordered point cloud and the pixel points they pass through, the two-dimensional structural lines in the set of structural semantic lines are associated with the pixels of the visual image to obtain the association relationship between the two-dimensional structural lines and the three-dimensional points.

[0117] The fitting module 560 is used to perform three-dimensional space fitting on the two-dimensional structural lines based on the correlation between the two-dimensional structural lines and the three-dimensional points to obtain three-dimensional structural lines.

[0118] The search module 570 is used to perform a geometric primitive half-plane search based on the three-dimensional structural lines to determine whether the point cloud on both sides of each three-dimensional structural line satisfies the constraints of the regular surface. If it does, the geometric primitive neighborhood is expanded according to the greedy algorithm to determine the largest regular region.

[0119] The fusion module 580 is used to fuse the largest determined regular region with the point cloud set to construct structured map data.

[0120] In this embodiment of the invention, multimodal perception information of the target scene is acquired; a set of regular parameterized lines in the visual image is obtained using the ELSDc line segment and elliptical arc detection method; the line segments and elliptical arcs in the set of regular parameterized lines are embedded and encoded to obtain embedding vectors; visual features are extracted from the visual image using an image feature extraction module; the embedding vectors and visual features are input into a trained parameterized line discrimination network model to obtain a set of structural semantic lines; the ordered and unordered point clouds in the depth perception information are preprocessed, and the structural semantic line segments of the ordered and unordered point clouds are correlated with the pixels they pass through. The index is used to associate the two-dimensional structural lines in the set of structural semantic lines with the pixels of the visual image, thus obtaining the association relationship between the two-dimensional structural lines and the three-dimensional points. Based on the association relationship between the two-dimensional structural lines and the three-dimensional points, the two-dimensional structural lines are fitted in three-dimensional space to obtain three-dimensional structural lines. Based on the three-dimensional structural lines, a geometric primitive half-plane search is performed to determine whether the point cloud on both sides of each three-dimensional structural line satisfies the constraint of the regular surface. If it does, the neighborhood of the geometric primitive is expanded according to the greedy algorithm to determine the largest regular region. The determined largest regular region is fused with the point cloud set to construct the structured map data.

[0121] Compared to structure recognition and enhancement methods based on point cloud fitting, this invention utilizes both image and point cloud information, extracting structural semantic lines as clues to discover regular structures. On one hand, it reduces the enormous computational burden and risk of inaccuracies required for point cloud segmentation and fitting. On the other hand, the structural line recognition and determination method proposed in this invention, compared to methods based on image feature points, can build 3D maps of various underground spaces in low-light and low-texture environments without manual intervention. It can also create complete, operable CAD models for regular objects such as pipelines, columns, and stairs. Even in poorly lit and textured underground spaces such as urban underground utility tunnels and subway tunnels, it works well, solving problems such as long manual modeling cycles and low accuracy in 3D modeling under weak texture and low lighting conditions. Furthermore, through cross-frame structural line matching, this invention can also be used for robot visual localization and navigation in low-texture and low-light environments.

[0122] Figure 6 This is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present invention. The electronic device 600 may vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 601 and one or more memories 602. The memory 602 stores at least one instruction, which is loaded and executed by the processor 601 to implement the steps of the above-mentioned Chinese text spelling check method.

[0123] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the aforementioned Chinese text spelling check method. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0124] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0125] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

Claims

1. A method for constructing structural semantic maps in underground low-light and low-texture environments, characterized in that, The method includes: S1. A sensor based on vision and three-dimensional depth perception acquires multimodal perception information of the target scene, wherein the multimodal perception information includes RGB visual images and depth perception information, and the depth perception information includes ordered point clouds and unordered point clouds; S2. Using the ELSDc line segment and elliptical arc detection method, obtain the set of regular parameterized lines of the visual image, and perform embedding encoding on the line segments and elliptical arcs in the set of regular parameterized lines to obtain an embedding vector with a length of 256 dimensions. S3. Visual features are extracted from the visual image using the image feature extraction module; S4. Input the embedding vector and the visual features into the trained parameterized line discrimination network model to obtain a set of structural semantic lines; S5. Preprocess the ordered and disordered point clouds in the depth perception information. By indexing the structural semantic line segments and the pixels they pass through in the ordered and disordered point clouds, associate the two-dimensional structural lines in the set of structural semantic lines with the pixels of the visual image to obtain the association relationship between the two-dimensional structural lines and the three-dimensional points. S6. Based on the relationship between the two-dimensional structural lines and the three-dimensional points, perform three-dimensional space fitting on the two-dimensional structural lines to obtain three-dimensional structural lines. S7. Perform a geometric primitive half-plane search based on the three-dimensional structural lines to determine whether the point cloud on both sides of each three-dimensional structural line satisfies the constraint of the regular surface. If it does, expand the neighborhood of the geometric primitives according to the greedy algorithm to determine the largest regular region. S8. Merge the largest regular region with the point cloud set to construct structured map data.

2. The method according to claim 1, characterized in that, The S2 method, using the ELSDc line segment and elliptical arc detection method, obtains a set of regular parameterized lines from the visual image, including: S21. Using the ELSDc line segment and elliptical arc detection method, obtain the set of regular parameterized lines {L,A} of the visual image, where {L,A} is as shown in the following formula (1): {L,A}=ELSDc(I rgb ) (1) L i ={(x 1i ,y 1i ,x 2i ,y 2i )|i=1,2,…N} Among them, I rgb Represents the visual image, (x 1i ,y 1i ,x 2i ,y 2i ) represents the parameter of the i-th line segment, x 1i ,y 1i The two-dimensional coordinates, x, are used to represent an endpoint of the i-th line segment. 2i ,y 2i The two-dimensional coordinates used to represent the other endpoint of the i-th line segment, where N represents the total number of line segments. The parameter x represents the j-th elliptic arc. 1j ,y 1j a is used to represent the two-dimensional coordinates of the center point of the elliptical arc. j ,b j θ represents the semi-axis lengths of the major and minor axes, respectively. j Indicates the rotation angle. Indicates the starting angle. The endpoint angle is represented by , and M represents the total number of elliptical arcs.

3. The method according to claim 2, characterized in that, S2 performs embedding encoding on the line segments and elliptical arcs in the set of regular parameterized lines to obtain an embedding vector of length 256, including: S22. According to the following formula (2), two different encoders are used to embed the line segment and the elliptical arc, and the encoded output is a vector with a length of 256 dimensions: Wherein, LineEncoder() represents a line segment position encoder, L i This represents the i-th line segment, and ArcEncoder() represents a circular arc position encoder. i Let i represent the i-th elliptic arc.

4. The method according to claim 1, characterized in that, The image feature extraction module is ResNet-50; The step S3 involves extracting visual features from the visual image using an image feature extraction module, including: The visual image is input into the ResNet-50, and a sin position encoder and a cos position encoder are used to embed the position at the scale of the visual image. The position embedding vector is then concatenated with the visual image to obtain the visual features of the visual image.

5. The method according to claim 1, characterized in that, The parameterized line discrimination network model is a Transformer model; The training process of the parameterized line discrimination network model includes: By using a differentiable renderer, the discrimination probability is used as the transparency value, and the line parameters are used as the line drawing constraints. The rendered image and the real structure semantic annotation binary image are subjected to L2 loss to train the parameterized line discrimination network model.

6. The method according to claim 4, characterized in that, The preprocessing of ordered and disordered point clouds in the depth-sensing information in step S5 includes: Based on the optimal accuracy range of the sensor, pass-through and conditional filters are used to preprocess the depth sensing information to extract the effective sensing area of ​​the sensor; For the ordered point cloud in the depth sensing information, a bilateral filter is used to denoise the surface of the spatial three-dimensional model. For the disordered point cloud in the depth sensing information, a Gaussian filter is used for smoothing and denoising. For cloud outliers, statistical filtering is used for removal.

7. The method according to claim 1, characterized in that, S5, by indexing the structural semantic line segments and traversed pixels of ordered and unordered point clouds, associates the two-dimensional structural lines in the structural semantic line set with the pixels of the visual image, obtaining the association relationship between the two-dimensional structural lines and the three-dimensional points, including: For each structural line corresponding to an ordered point cloud, the indices of all pixels passed through by the structural line segment are associated with the index of the line segment to establish a set of three-dimensional coordinate points covered by the line segment. For disordered point clouds, the sensor is used for parameter calibration to establish a correspondence between the pixels in the image coordinate system and the three-dimensional points of the disordered point cloud. The indexes of all pixels passed through by each structural line corresponding to the disordered point cloud are associated with the index of the line segment to establish a set of three-dimensional coordinate points covered by the line segment.

8. The method according to claim 1, characterized in that, The step S7, which involves performing a geometric primitive half-plane search based on the three-dimensional structural lines to determine whether the point clouds on both sides of each three-dimensional structural line satisfy the constraints of a regular surface, includes: For a line segment in a three-dimensional structure, for the two parts on both sides of the line segment, parameter fitting is performed simultaneously using a plane and a cylindrical surface. If either part on both sides of the line segment conforms to the fitting of a plane or a cylindrical surface, then it is determined that the point cloud on both sides of the line segment satisfies the constraint of a regular surface, that is, there exists a corresponding regular geometric primitive. For the elliptical arc in the three-dimensional structural lines, for the part inside the elliptical arc, parameter fitting is performed using a plane. If there is a three-dimensional point cloud distribution that conforms to the plane, then the point cloud inside the arc is determined to satisfy the constraint of a regular surface, that is, the geometric primitive inside the elliptical arc is a plane. For the part outside the elliptical arc, parameter fitting is performed using a cylindrical surface. If there is a three-dimensional point cloud distribution that conforms to the plane, then the point cloud outside the arc is determined to satisfy the constraint of a regular surface, that is, the geometric primitive outside the elliptical arc is a cylindrical surface.

9. A structural semantic map construction device for underground low-light and low-texture environments, characterized in that, The device includes: The acquisition module is used to acquire multimodal perception information of the target scene based on vision and three-dimensional depth perception sensors. The multimodal perception information includes RGB visual images and depth perception information, and the depth perception information includes ordered point clouds and unordered point clouds. The embedding encoding module is used to obtain a set of regular parameterized lines in the visual image using the ELSDc line segment and elliptical arc detection method, and to perform embedding encoding on the line segments and elliptical arcs in the set of regular parameterized lines to obtain an embedding vector with a length of 256 dimensions. The feature extraction module is used to extract visual features from the visual image through the image feature extraction module; The discrimination module is used to input the embedding vector and the visual features into the trained parameterized line discrimination network model to obtain a set of structural semantic lines; The association module is used to preprocess the ordered and disordered point clouds in the depth perception information. By indexing the structural semantic line segments of the ordered and disordered point clouds and the pixels they pass through, the two-dimensional structural lines in the set of structural semantic lines are associated with the pixels of the visual image to obtain the association relationship between the two-dimensional structural lines and the three-dimensional points. The fitting module is used to perform three-dimensional space fitting on the two-dimensional structural lines based on the relationship between the two-dimensional structural lines and the three-dimensional points to obtain three-dimensional structural lines. The search module is used to perform a geometric primitive half-plane search based on the three-dimensional structural lines to determine whether the point cloud on both sides of each three-dimensional structural line satisfies the constraints of the regular surface. If it does, the geometric primitive neighborhood is expanded according to the greedy algorithm to determine the largest regular region. The fusion module is used to merge the largest identified regular region with the point cloud set to construct structured map data.

10. The apparatus according to claim 9, characterized in that, The embedding encoding module is used for: S21. Using the ELSDc line segment and elliptical arc detection method, obtain the set of regular parameterized lines {L,A} of the visual image, where {L,A} is as shown in the following formula (1): {L,A}=ELSDc(I rgb ) (1) L i ={(x 1i ,y 1i ,x 2i ,y 2i )|i=1,2,…N} Among them, I rgb Represents the visual image, (x 1i ,y 1i ,x 2i ,y 2i ) represents the parameter of the i-th line segment, x 1i ,y 1i The two-dimensional coordinates, x, are used to represent an endpoint of the i-th line segment. 2i ,y 2i The two-dimensional coordinates used to represent the other endpoint of the i-th line segment, where N represents the total number of line segments. The parameter x represents the j-th elliptic arc. 1j ,y 1j a is used to represent the two-dimensional coordinates of the center point of the elliptical arc. j ,b j θ represents the semi-axis lengths of the major and minor axes, respectively. j Indicates the rotation angle. Indicates the starting angle. The endpoint angle is represented by , and M represents the total number of elliptical arcs.

Images