Tunnel space perception method and device based on multi-modal 3D instance segmentation and medium

Abstract

The application discloses a tunnel space perception method and device based on multi-modal 3D instance segmentation and a medium, wherein the method comprises: acquiring sensor data; sampling the sensor data to generate environment point cloud data and RGB image data; dividing the environment point cloud data and the RGB image data based on a tunnel scene target to obtain segmented point cloud data and a point cloud segmentation result of the tunnel scene target; and calibrating and screening the segmented point cloud data based on the point cloud segmentation result of the tunnel scene target to determine key point clouds representing target object contours. When data is divided, the weight of image and point cloud features is dynamically adjusted through a channel attention mechanism, cross attention of the image and the point cloud is calculated, and the region of interest pays more attention to the information of one mode, so that the feature extraction capability in a complex tunnel shield environment is strengthened, and accurate segmentation of target data in a complex tunnel environment is realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision perception technology, and more specifically, to a method, apparatus and medium for spatial perception of tunnels based on multimodal 3D instance segmentation. Background Technology

[0002] In underground tunnel construction, the shield tunneling machine (TBM) method has become the primary choice. A TBM is a large-scale underground engineering construction equipment integrating mechanical, electrical, hydraulic, optical, and computer technologies; its technological level is an important indicator of the manufacturing level of underground construction equipment. The segment assembly machine is a crucial component of the TBM, installing prefabricated segments as permanent support during tunnel excavation. However, due to the large size of the segments and the narrow and complex working space, traditional manual segment assembly operations struggle to guarantee construction quality and speed, becoming a bottleneck limiting the efficiency of TBM construction.

[0003] Therefore, there is an urgent need to improve the automation level of tunnel segment assembly to enhance the quality and efficiency of large-scale tunnel boring machine (TBM) projects. Automated tunnel segment assembly is a fine-grained, high-precision construction task, requiring millimeter-level control of construction errors. However, automated tunnel segment assembly faces the following challenges: enclosed and complex environments, narrow construction spaces with complex equipment, and mechanical vibration and positional misalignment of the equipment. These challenges place extremely high demands on the precision and efficiency of automated tunnel segment assembly technology.

[0004] In recent years, with the development of technologies such as the Internet of Things, 5G, and artificial intelligence, sensor technology has been significantly improved. For example, sensors capable of sensing 3D space, such as LiDAR, depth cameras, and stereo cameras, have been continuously developed, possessing the ability to generate high-density point cloud data and perform 360-degree omnidirectional detection. These advanced environmental sensing sensors provide strong support for automated segment assembly.

[0005] With the continuous development of tunnel construction and shield tunneling technology, accurate perception of the tunnel's internal conditions has become an urgent need. Current research lacks a general, high-precision perception method based on deep learning, resulting in poorly accurate outputs that cannot be applied to real-world scenarios. Tunnel tube assembly is a crucial part of tunnel excavation engineering. The current tunnel environment faces multiple challenges, including confined space, complex lighting conditions, and a scarcity of multimodal target data. Summary of the Invention

[0006] This invention addresses the aforementioned problems in the prior art. Therefore, there is a need for a method, device, and medium for tunnel spatial perception based on multimodal 3D instance segmentation. This multimodal 3D spatial perception instance segmentation scheme improves the target perception capability in complex spaces, enhances the environmental perception capability of tunnel boring machine (TBM) grabbers in tunnels, and provides auxiliary decision-making for segment grabbing, specific area avoidance during segment grabbing and movement, customized route planning, and precise segment assembly.

[0007] According to a first aspect of the present invention, a method for spatial perception of tunnels based on multimodal 3D instance segmentation is provided, the method comprising:

[0008] Acquire sensor data;

[0009] The sensor data is sampled to generate environmental point cloud data and RGB image data;

[0010] Based on the tunnel scene target, the environmental point cloud data and RGB image data are divided to obtain the segmented point cloud data and the point cloud segmentation result of the tunnel scene target.

[0011] Based on the point cloud segmentation results of the target in the tunnel scene, the segmented point cloud data is calibrated and filtered to determine the key point cloud representing the outline of the target object.

[0012] Furthermore, the sensor data includes sensor parameters, pose, calibration information, and image data.

[0013] Furthermore, after acquiring the sensor data, the method further includes:

[0014] A database is constructed and data tables are set in the database to store and access the sensor parameters, pose, and calibration information in the data tables;

[0015] The system acquires control commands, which include sensor configuration parameters and the sensor state after configuring the sensor using the sensor configuration parameters. The system determines the current sensor state based on real-time sensor parameters, pose, and calibration information. If the current sensor state is not the same as the sensor state after configuring the sensor using the sensor configuration parameters, the system configures the sensor using the sensor configuration parameters.

[0016] Furthermore, the sensor data is sampled to generate environmental point cloud data and RGB image data, specifically including:

[0017] The coordinate space of each image data is transformed according to the calibration information to achieve spatial unification of the image data and obtain a unified coordinate data acquisition point cloud and RGB image;

[0018] The unified coordinate data collection point cloud is sequentially subjected to dimensional expansion, sampling and point cloud normalization processing to obtain environmental point cloud data. The steps of sampling and point cloud normalization processing of the dimensionally expanded point cloud include sampling the farthest point of the dimensionally expanded point cloud to obtain the number of input point clouds, and scaling and normalizing the dimensionally expanded point cloud.

[0019] The RGB image is sequentially resized and pixel normalized to obtain RGB image data. The steps for resizing the RGB image include: resizing the RGB image to a set size, and performing cubic interpolation using the gray values ​​of a set number of points around the sampling point to preserve image details, texture features, and process sharp edges and high-frequency information. The steps for pixel normalization of the resized RGB image include 0-1 normalization and mean-variance normalization. The 0-1 normalization scales the pixel values ​​of the image to the range of 0 to 1, and the mean-variance normalization adjusts the pixel values ​​of the image to a distribution with a mean of 0 and a standard deviation of 1.

[0020] Furthermore, based on the tunnel scene target, the environmental point cloud data and RGB image data are segmented to obtain segmented point cloud data and point cloud segmentation results of the tunnel scene target, specifically including:

[0021] A multimodal 3D instance segmentation model is constructed. The multimodal 3D instance segmentation model takes the environmental point cloud data and RGB image data as input, extracts point cloud features and image features, and uses a channel attention mechanism to fuse point cloud features and image features. The point cloud features are enhanced based on the high-dimensional features of the RGB image, and the segmented point cloud data and the point cloud segmentation results of the tunnel scene target are output.

[0022] Furthermore, based on the tunnel scene target, the environmental point cloud data and RGB image data are segmented to obtain segmented point cloud data and point cloud segmentation results of the tunnel scene target, specifically including:

[0023] A multimodal 3D instance segmentation model is constructed, which includes an RGB image feature extraction module, a point cloud data feature extraction module, a feature fusion module, and an instance segmentation network module.

[0024] The RGB image feature extraction module includes a Steam module. The RGB image feature extraction module takes the RGB image data as input and learns features at different scales and levels by capturing global and local features based on the two convolutional pooling modules in the Steam module, so as to output image features.

[0025] The point cloud data feature extraction module includes a network sampling module, a grouping module, and a PointNet network. The point cloud data feature extraction module takes the environmental point cloud data as input and samples key points P from the environmental point cloud data through the network sampling module. k The grouping module is based on sampling key points P k Select P from the surrounding point cloud N The PointNet network combines local and global point cloud features based on residual connections to extract point cloud features and output point cloud features.

[0026] The feature fusion module takes the outputs of the RGB image feature extraction module and the point cloud data feature extraction module as input, extracts point cloud features through an MLP network to obtain point cloud features P(N x 64), and copies the RGB high-dimensional features N times column-wise according to the number of point clouds N to obtain image features V(N x 1024). The point cloud features P(N x 64) and image features V(N x 1024) are then combined. The feature matrix PV is obtained by concatenating x1024. The feature matrix PV is then processed by a multilayer perceptron layer to learn the nonlinear relationships between the features contained in the feature matrix PV. The multilayer perceptron layer consists of multiple fully connected layers and nonlinear activation functions. The output range of the multilayer perceptron layer is normalized to the [0,1] interval by a normalization function. The normalized output is fed into an attention learning network to obtain an attention mask. The attention mask is multiplied element-wise with the original point cloud features P output by the point cloud data feature extraction module to obtain a weighted feature. The weighted feature is added to the original point cloud features P through residual connections to finally output the fused feature O(Nx 64).

[0027] The instance segmentation network module takes the fused features output by the feature fusion module as input, extracts the global and local features of the fused features, and divides them into three branches: a similarity matrix branch, a confidence branch, and a semantic prediction branch. The similarity matrix branch is used to generate point-by-point grouping suggestions, the confidence branch is used to filter the point-by-point grouping suggestions, and the semantic prediction branch is used to predict a category label for each group. The final instance segmentation prediction is generated through the three branches, resulting in the segmented point cloud data and the point cloud segmentation result of the tunnel scene target.

[0028] Furthermore, based on the point cloud segmentation results of the target in the tunnel scene, the point cloud data is calibrated and filtered to determine the key point cloud representing the contour of the target object, specifically including:

[0029] The segmented point cloud data is filtered according to the scene, outliers and points that do not conform to the rules are removed, and key point clouds representing the outline of the target object are determined.

[0030] Furthermore, the segmented point cloud data is filtered according to the scene, outliers and points that do not conform to the rules are removed, and key point clouds representing the contour of the target object are determined, specifically including:

[0031] The segmented point cloud data is judged and filtered based on spatial location, color, shape, scene features, and prior knowledge to remove point cloud data that is irrelevant to the current scene.

[0032] Outlier points in the segmented point cloud data are identified using outlier detection algorithms and iterative nearest neighbor algorithms.

[0033] Based on the constraints, point cloud data that is irrelevant to the current scene and outliers are filtered to determine the key point cloud representing the outline of the target object. The constraints are determined based on the shape, size, and relative position of the target object.

[0034] According to a second aspect of the present invention, a multimodal 3D instance segmentation tunnel spatial sensing device is provided, the device comprising:

[0035] The device access module is configured to acquire sensor data;

[0036] The data preprocessing module is configured to sample the sensor data to generate environmental point cloud data and RGB image data;

[0037] The 3D instance segmentation module is configured to divide the environmental point cloud data and RGB image data based on the tunnel scene target, and obtain the segmented point cloud data and the point cloud segmentation result of the tunnel scene target.

[0038] The point cloud calibration and filtering module is configured to calibrate and filter the segmented point cloud data based on the point cloud segmentation results of the target in the tunnel scene, so as to determine the key point cloud representing the outline of the target object.

[0039] According to a third aspect of the present invention, a readable storage medium is provided, the readable storage medium storing one or more programs, the one or more programs being executable by one or more processors to implement the method described above.

[0040] The present invention has at least the following beneficial effects:

[0041] 1. This invention is based on tunnel shield environment data collected by the equipment access module. It segments tunnel segments and background targets in multiple scenarios such as segment grabbing and segment assembly in the tunnel environment. Based on the segmentation results and sensor coordinate pose calculation, it generates tunnel spatial perception information. The generated information is based on the coordinate system of the visual perception tool. It outputs the pose relationship of the key point cloud of each target in the tunnel environment after screening in the sensor cluster coordinate system. The information is output to the shield machine assembly machine control system and the assembly machine control interface for automated control of the assembly machine and to be displayed to the assembly machine operator.

[0042] 2. The core module of this invention is the multimodal 3D instance segmentation model. The multimodal 3D instance segmentation model proposes and adopts an RGB and point cloud multimodal fusion instance segmentation algorithm model. It studies the fusion of image and point cloud data features based on the attention mechanism. The weights of image and point cloud features are dynamically adjusted through the channel attention mechanism, and the cross attention of image and point cloud is calculated. This makes the model pay more attention to the information of one modality in the region of interest, which enhances the model's feature extraction capability in the complex environment of tunnel shield tunneling and realizes accurate segmentation of target data in the complex environment of tunnel. Attached Figure Description

[0043] Figure 1 A schematic diagram of the module configuration of a multimodal 3D instance segmentation tunnel spatial perception tool according to an embodiment of the present invention is shown.

[0044] Figure 2 A flowchart of a tunnel spatial perception method based on multimodal 3D instance segmentation according to an embodiment of the present invention is shown.

[0045] Figure 3 A structural diagram of a multimodal 3D instance segmentation model according to an embodiment of the present invention is shown.

[0046] Figure 4 A structural diagram of a feature fusion module according to an embodiment of the present invention is shown.

[0047] Figure 5 A flowchart of the point cloud calibration and filtering module according to an embodiment of the present invention is shown.

[0048] Figure 6 A structural diagram of a multimodal 3D instance segmentation tunnel spatial perception system according to an embodiment of the present invention is shown. Detailed Implementation

[0049] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and specific examples, but this is not intended to limit the present invention. If there is no necessary sequential relationship between the various steps described herein, the order in which they are described as examples should not be considered a limitation. Those skilled in the art should understand that the order can be adjusted, as long as it does not disrupt the logical consistency between them and render the entire process impossible.

[0050] This invention provides a method for spatial perception of tunnels based on multimodal 3D instance segmentation. Please refer to [link / reference]. Figure 1 This is a schematic diagram of the module structure of a tunnel spatial perception tool based on multimodal 3D instance segmentation. The tunnel spatial perception method based on multimodal 3D instance segmentation is based on, for example... Figure 1 The sensing tool shown is implemented. This tool includes a device access module, a data preprocessing module, a 3D instance segmentation module, and a point cloud calibration and filtering module. The device access module manages the sensor devices, responsible for sensor parameter settings, status detection, and data acquisition. The data preprocessing module integrates sensor data to generate point cloud and RGB image data, unifies the coordinate space, and acquires point cloud and RGB image data according to scene requirements. The 3D instance segmentation module segments the data to obtain the point cloud segmentation results of the tunnel scene targets. The point cloud calibration and filtering module performs point cloud calibration and filtering to obtain the key point cloud of the targets. This tool, through a multimodal 3D instance segmentation method combined with device access, data preprocessing, and point cloud calibration and filtering modules, achieves perception and positioning of the tunnel space, improving the accuracy and efficiency of perception, and adapting to different scenarios and device access requirements.

[0051] Please see Figure 2 This is a flowchart of a tunnel spatial perception method based on multimodal 3D instance segmentation. The method includes steps S100 to S400, wherein steps S100 to S400 are respectively performed by… Figure 1 The sensing tool shown is implemented using the device access module, data preprocessing module, 3D instance segmentation module, and point cloud calibration and filtering module. Steps S100 to S400 will be described in detail below.

[0052] Step S100: Acquire sensor data.

[0053] Step S100 is implemented by the device access module, which is responsible for managing the sensor devices, including configuration databases and design data tables, and storing sensor parameters, poses, and calibration information. It interacts with the control interface and assembly machine control tools to control and modify the sensor configuration. In this embodiment, the sensor data is image data, and the corresponding sensor is an image sensor, such as a camera or camcorder. The sensor data collected by the sensor is also acquired by the device access module. Figure 1 S1, S2...Sn represent the first, second...nth image data respectively.

[0054] In some embodiments, the device access module uses a MySQL database to store sensor configuration information and calibration results, and sets up four data tables to store relevant data: sensor deployment scheme table, sensor information table, sensor eye-to-eye calibration table, and sensor hand-to-eye calibration table.

[0055] The sensor deployment scheme table records a set of sensor deployment schemes, including the scheme's unique identifier, deployment purpose, sensor eye-to-eye calibration scheme ID, hand-to-eye calibration ID, deployment description, start time, and modification time. The sensor information table records basic sensor information, including the sensor's unique identifier, deployment scheme ID, deployment information, calibration information, sensor model, and configuration parameters. The sensor eye-to-eye calibration table stores the calibration results of multiple sensors, including the calibration unique identifier, calibration coordinate system origin, pose information of each sensor based on the origin, sensor group calibration scheme description, and sensor calibration board description. The sensor hand-to-eye calibration table stores the sensor hand-to-eye calibration results, including the calibration unique identifier, calibration pose information, and calibration scheme description.

[0056] In some embodiments, after acquiring sensor data, the method further includes a sensor control step, which is implemented by the device access module, and the control step is as follows:

[0057] The system acquires control commands, which include sensor configuration parameters and the sensor state after configuring the sensor using the sensor configuration parameters. The system determines the current sensor state based on real-time sensor parameters, pose, and calibration information. If the current sensor state is not the same as the sensor state after configuring the sensor using the sensor configuration parameters, the system configures the sensor using the sensor configuration parameters.

[0058] The control commands can be sent by the user to the device access module through the control interface. After receiving the control command, the information of the control command is extracted to determine the target state that the sensor needs to achieve. Based on the relevant data fed by the sensor, it is determined whether the target state is reached. If the target state is not reached, the configuration parameters of the sensor are changed. If the target state is reached, the configuration parameters of the sensor are not changed.

[0059] For example, the device access module communicates with the sensor to receive sensor data and transmit it to other modules for further processing. Various sensor commands can be defined in the device access module. The format of each sensor command consists of a command type, sensor identifier, command operation, operation parameters, and whether data is returned. Sensor commands are categorized into setting commands and control commands, each with different purposes and functions. Setting commands configure sensor parameters and operating modes, allowing adjustment of various sensor parameters such as resolution, frame rate, exposure time, and sampling rate. Control commands control the sensor's behavior and execute specific operations. Control commands can trigger the sensor to perform operations such as taking pictures or recording video to obtain required data or achieve specific functions. Control commands may also include heartbeat commands and sensor status check commands to detect whether the sensor is working properly and obtain its current status information.

[0060] Step S200: Sample the sensor data to generate environmental point cloud data and RGB image data.

[0061] Step S200 is implemented by the data preprocessing module, which can be implemented as a processing chip or circuit module, etc. The specific steps can be implemented as long as they can achieve step S200. This embodiment does not impose specific restrictions on it.

[0062] Specifically, the data preprocessing module is used to process the sensor data acquired by the device access module into environmental point cloud data and RGB image data, which includes RGB image preprocessing steps and point cloud data preprocessing steps.

[0063] RGB image processing steps include image resizing and pixel normalization. For resizing, bicubic interpolation is used to preserve image texture and edge features, resizing the image to 299*299 pixels. Bicubic interpolation uses the grayscale values ​​of 16 points surrounding the sampled point for cubic interpolation, preserving image details, texture features, sharp edges, and high-frequency information. After resizing, RGB image normalization mainly consists of 0-1 normalization and mean-variance normalization. 0-1 normalization scales the image pixel values ​​to the range of 0 to 1, while mean-variance normalization adjusts the image pixel values ​​to a distribution with a mean of 0 and a standard deviation of 1. This completes the RGB image processing, yielding the RGB image data.

[0064] Point cloud data preprocessing includes point cloud data dimensional expansion, sampling, and point cloud normalization. The original feature dimensions of the input point cloud only include the x, y, and z coordinates in space. Additional normal vectors and curvature features are introduced based on the original sensor data. The number of input point clouds is obtained by sampling the farthest points, and the sampled point clouds are scaled and normalized to complete the data preprocessing.

[0065] Step S300: Based on the tunnel scene target, the environmental point cloud data and RGB image data are divided to obtain the segmented point cloud data and the point cloud segmentation result of the tunnel scene target.

[0066] Step S300 is the core step of this application, which is implemented by a 3D instance segmentation module. As an example only, the 3D instance segmentation module can be implemented as a processing chip.

[0067] The 3D instance segmentation module can segment environmental point cloud data and RGB image data by constructing a multimodal 3D instance segmentation model. Please refer to [link / reference]. Figure 3 This diagram illustrates the structure of a multimodal 3D instance segmentation model. Based on a multi-scale point cloud feature extraction network, point cloud features are extracted. An instance segmentation network integrating RGB and point cloud features is studied and designed using an attention mechanism. Compared to a single-point cloud modality model, this multimodal 3D instance segmentation model adds RGB channel features to enhance point cloud instance segmentation. While the network structure is more complex and has more parameters, its feature extraction capability is improved through the fusion of point cloud and RGB features, resulting in higher segmentation accuracy. The multimodal 3D instance segmentation model includes an RGB image feature extraction module, a point cloud data feature extraction module, a feature fusion module, and an instance segmentation network module.

[0068] The RGB image feature extraction module includes a Steam module. The RGB image feature extraction module takes the RGB image data as input and learns features at different scales and levels by capturing global and local features based on the two convolutional pooling modules in the Steam module, so as to output image features.

[0069] For example, the RGB image feature extraction module can be built based on InceptionNet V4, which mainly includes Stem, Inception, Reduction, an auxiliary classifier, and a global pooling fully connected module. The three basic Inception structures of InceptionV4 are basically the same as those used in InceptionV3, but InceptionV4 introduces some new module shapes and connection designs. The main feature is the introduction of the "Stem" module in the early stages of the network, which is used to quickly reduce the resolution of the feature maps, thereby reducing the computational cost of subsequent Inception modules.

[0070] The point cloud data feature extraction module includes a network sampling module, a grouping module, and a PointNet network. The point cloud data feature extraction module takes the environmental point cloud data as input and samples key points P from the environmental point cloud data through the network sampling module. k The grouping module is based on sampling key points P k Select P from the surrounding point cloud N The PointNet network combines local and global point cloud features based on residual connections to extract point cloud features and output point cloud features.

[0071] Compared to PointNet, this design has a larger number of parameters. Although the inference speed is lower than that of PointNet, the feature extraction capability is greatly enhanced. It realizes an end-to-end segmentation model based on a multi-scale feature extraction network improved from PointNet. The main improvements include the ResPointBlock residual point cloud feature extraction module, the MaxAttention module, and the U-Net multi-scale feature extraction network.

[0072] The feature fusion module takes the outputs of the RGB image feature extraction module and the point cloud data feature extraction module as input, and outputs fused features. Please refer to [link to relevant documentation]. Figure 4 This is a structural diagram of the feature fusion module, which includes multiple attention embedding fusion modules. The data processing flow of this feature fusion module is as follows:

[0073] First, a point cloud feature P (N x 64) is extracted from the input raw point cloud features using a multilayer perceptron layer. This point cloud feature P (N x 64) is a representation containing both local and global information of the point cloud. Then, to further extract abstract features, the point cloud feature P (N x 64) is convolved through an edge convolution layer and a non-linear activation function is applied to obtain a higher-level feature representation. For the fusion of point cloud and RGB features, the embedded RGB features are copied N times column-wise according to the number of points N in the point cloud, resulting in image features V (N x 1024). This is done to preserve the spatial information of the RGB features and fuse them element-wise with the point cloud features. Next, the point cloud feature P (N x 64) and image features V (N x 1024) are concatenated to form a feature matrix PV of size N x 1088. This is done to associate the point cloud features and image features so that the network can learn the relationships between them. Finally, the feature matrix PV is processed by a multilayer perceptron (MLP) layer. The MLP layer consists of multiple fully connected layers and non-linear activation functions, enabling it to learn complex non-linear relationships between features. This allows for the extraction of abstract representations of the features. Next, a normalization function (such as the sigmoid function) normalizes the output range of the MLP layer to the [0,1] interval. This output is interpreted as the network's attention level to each point cloud region; a larger value indicates greater importance the network places on that region, i.e., a soft attention mask (N x 64). This attention mask automatically learns the importance of different regions in the point cloud. Finally, the attention mask is multiplied element-wise with the original point cloud features P to obtain a weighted feature. This allows the network to pay more attention to important point cloud regions and assign them higher weights. The weighted feature is then added to the original point cloud features P via residual connections to obtain the final fused output feature O (N x 64). Such residual connections help the network better propagate gradients, thereby improving the network's training effectiveness and performance. This attention-embedded fusion module, through the learned attention mask, can automatically learn the importance of different regions and weight the features of important regions to improve the network's performance and robustness. This attention mechanism enables the network to focus more on important point cloud regions, thereby improving the processing effect of point cloud data and achieving better results in point cloud processing tasks.

[0074] Step S400: Based on the point cloud segmentation results of the target in the tunnel scene, the segmented point cloud data is calibrated and filtered to determine the key point cloud representing the outline of the target object.

[0075] This step is implemented by the point cloud calibration and filtering module. Please refer to [link / reference]. Figure 5The flowchart below shows the workflow of the point cloud calibration and filtering module. Specifically, based on the point cloud segmentation results of the target in the tunnel scene, the module calibrates and filters the segmented point cloud data to determine the key point cloud representing the outline of the target object. The specific implementation steps include:

[0076] Step S401: Segmentation results from the 3D instance segmentation module (segmented point cloud data). Figure 5 In the diagram, P1, P2...Pn represent point cloud points in the segmented point cloud data, where n represents the index of the point cloud point (e.g., Pn represents the nth point cloud point in the segmented point cloud data). The target model is used as input for scene feature filtering: First, based on spatial location, color, shape scene features and prior knowledge, the point cloud data is judged and filtered to remove point clouds that are not related to the current scene.

[0077] Step S402: Based on the screening results of step S401, perform RANSAC (Nearest Neighbor Algorithm) outlier screening: Use an outlier detection algorithm and iterate the nearest neighbor algorithm (RANSAC) to identify and remove outliers in the point cloud.

[0078] Step S403: Based on the filtering results of step S402, filter the data according to rules to output the data: Further filter the point cloud data according to predefined rules and constraints. These rules can be based on features such as the shape, size, and relative position of specific target objects in a specific scene. Generate filtered and calibrated point cloud data as the output data.

[0079] In one exemplary embodiment, a multimodal 3D instance segmentation tunnel spatial perception system is constructed based on the multimodal 3D instance segmentation tunnel spatial perception method described above. Please refer to [link to relevant documentation]. Figure 6 The diagram shown is a structural diagram of the system. This multimodal 3D instance segmentation tunnel spatial perception system consists of five layers, from top to bottom: system interface layer, system module layer, system foundation layer, software foundation layer, and hardware layer.

[0080] The system interface layer serves as the interface through which users interact with the tools, including a graphical user interface (GUI) and an assembly machine interface. The GUI provides a user-friendly visual interface for configuring tool parameters, monitoring tool status, and viewing processing results. The assembly machine interface is used to communicate and control the robotic assembly machine, enabling automated assembly.

[0081] The system module layer is a collection of core functional modules for the tool. These include a 3D instance segmentation module, a point cloud calibration and filtering module, a control module, and a security module. The 3D instance segmentation module extracts point cloud segmentation results from raw data acquired by sensors. The point cloud calibration and filtering module outputs the calibrated and filtered point cloud results. The control module is responsible for the overall control logic and process management of the tool. The security module monitors and protects the tool's security, including fault detection and preventative measures.

[0082] The system's foundational layer provides basic functional support for the tool. It includes a data preprocessing module, a RANSAC algorithm module, a multimodal instance segmentation algorithm module, and a sensor management module. The data preprocessing module is responsible for preprocessing and cleaning the input data. The RANSAC algorithm module is used for outlier removal calculations. The multimodal instance segmentation algorithm module is used to segment different object instances in the point cloud. The sensor management module is responsible for managing and controlling the sensor devices within the tool.

[0083] The software infrastructure layer includes a MySQL database, a Hadoop database, an RGB processing library, a point cloud processing library, a CUDA library, and a PyTorch library. Each library is used to store the corresponding sensor data and the algorithm data required to process the sensor data, or to store intermediate data and final data during and after sensor data processing, providing the data foundation for algorithm implementation in the system module layer and the system infrastructure layer.

[0084] The hardware layer is the lowest level, including hard drive storage devices, environmental sensors, and GPUs (graphics cards). Hard drive storage devices are used to store the data and files required by the tool. Environmental sensors are used to collect environmental data, such as point cloud data and image data. GPUs are used to accelerate computationally intensive tasks, improving the tool's performance and efficiency.

[0085] The system architecture described above enables a user-friendly interface, facilitates the extraction and analysis of point cloud data, ensures tool security, and provides basic functional support. The hardware layer provides the necessary physical device support, allowing the tool to operate efficiently.

[0086] This invention also provides a multimodal 3D instance segmentation tunnel spatial sensing device, the device comprising:

[0087] The device access module is configured to acquire sensor data;

[0088] The data preprocessing module is configured to sample the sensor data to generate environmental point cloud data and RGB image data;

[0089] The 3D instance segmentation module is configured to divide the environmental point cloud data and RGB image data based on the tunnel scene target, and obtain the segmented point cloud data and the point cloud segmentation result of the tunnel scene target.

[0090] The point cloud calibration and filtering module is configured to calibrate and filter the segmented point cloud data based on the point cloud segmentation results of the target in the tunnel scene, so as to determine the key point cloud representing the outline of the target object.

[0091] It should be noted that the various device structures described in this embodiment and the previously described methods belong to the same technical concept and achieve the same technical effect through the same principle, which will not be repeated here.

[0092] This invention also provides a readable storage medium storing one or more programs that can be executed by one or more processors to implement the methods described in any of the above embodiments.

[0093] Furthermore, although exemplary embodiments have been described herein, their scope includes any and all embodiments based on the invention that have equivalent elements, modifications, omissions, combinations (e.g., schemes involving intersections of various embodiments), adaptations, or alterations. Elements in the claims will be interpreted broadly based on the language used in the claims and are not limited to the examples described in this specification or during the implementation of this application, and such examples will be interpreted as non-exclusive. Therefore, this specification and examples are intended to be considered illustrative only, and the true scope and spirit are indicated by the following claims and the full scope of their equivalents.

[0094] The above description is intended to be illustrative and not restrictive. For example, the above examples (or one or more of them) can be used in combination with each other. Other embodiments can be used by those skilled in the art when reading the above description. Furthermore, in the above detailed description, various features may be grouped together to simplify the invention. This should not be construed as an intention that a feature of an unclaimed invention is necessary for any claim. Rather, the subject matter of the invention may be less than all the features of a particular embodiment of the invention. Thus, the following claims are incorporated herein by reference as examples or embodiments, wherein each claim is an independent, separate embodiment, and these embodiments are contemplated to be combined with each other in various combinations or arrangements. The scope of the invention should be determined by reference to the appended claims and the full scope of their equivalents.

Claims

1. A spatial perception method for tunnel segmentation based on multimodal 3D instance segmentation, characterized in that, The method includes: Acquire sensor data; The sensor data is sampled to generate environmental point cloud data and RGB image data; Based on the tunnel scene target, the environmental point cloud data and RGB image data are divided to obtain the segmented point cloud data and the point cloud segmentation result of the tunnel scene target. Based on the point cloud segmentation results of the target in the tunnel scene, the segmented point cloud data is calibrated and filtered to determine the key point cloud representing the outline of the target object. Based on the tunnel scene target, the environmental point cloud data and RGB image data are segmented to obtain segmented point cloud data and point cloud segmentation results of the tunnel scene target, specifically including: A multimodal 3D instance segmentation model is constructed, which includes an RGB image feature extraction module, a point cloud data feature extraction module, a feature fusion module, and an instance segmentation network module. The RGB image feature extraction module is built based on InceptionNet V4. The RGB image feature extraction module includes a Stem module. The RGB image feature extraction module takes the RGB image data as input and learns features at different scales and levels by capturing global and local features based on the two convolutional pooling modules in the Stem module, so as to output image features. The point cloud data feature extraction module includes a network sampling module, a grouping module, and a PointNet network. The point cloud data feature extraction module takes the environmental point cloud data as input and samples key points from the environmental point cloud data through the network sampling module. The grouping module is based on sampling key points. Selection of surrounding point clouds The PointNet network combines local and global point cloud features based on residual connections to extract point cloud features and output point cloud features. The feature fusion module takes the outputs of the RGB image feature extraction module and the point cloud data feature extraction module as input. It extracts point cloud features P through an MLP network. Based on the number of points N, the RGB high-dimensional features are copied N times column-wise to obtain image features V. Point cloud features P and image features V are concatenated to obtain a feature matrix PV. A multilayer perceptron layer processes the feature matrix PV to learn the nonlinear relationships between the features contained in the feature matrix PV. The multilayer perceptron layer consists of multiple fully connected layers and nonlinear activation functions. A normalization function normalizes the output range of the multilayer perceptron layer to the [0, 1] interval. The normalized output is fed into an attention learning network to obtain an attention mask. The attention mask is multiplied element-wise with the original point cloud features output by the point cloud data feature extraction module to obtain a weighted feature. The weighted feature is added to the original point cloud features through residual connections, finally outputting a fused feature O. The size of point cloud features P and fused feature O is N x 64, and the size of image features V is N x 64. 1024; The instance segmentation network module takes the fused features output by the feature fusion module as input, extracts the global and local features of the fused features, and divides them into three branches: a similarity matrix branch, a confidence branch, and a semantic prediction branch. The similarity matrix branch is used to generate point-by-point grouping suggestions, the confidence branch is used to filter the point-by-point grouping suggestions, and the semantic prediction branch is used to predict a category label for each group. The final instance segmentation prediction is generated through the three branches, resulting in the segmented point cloud data and the point cloud segmentation result of the tunnel scene target.

2. The method according to claim 1, characterized in that, The sensor data includes sensor parameters, pose, calibration information, and image data.

3. The method according to claim 2, characterized in that, After acquiring sensor data, the method further includes: A database is constructed and data tables are set in the database to store and access the sensor parameters, pose, and calibration information in the data tables; The system acquires control commands, which include sensor configuration parameters and the sensor state after configuring the sensor using the sensor configuration parameters. The system determines the current sensor state based on real-time sensor parameters, pose, and calibration information. If the current sensor state is not the same as the sensor state after configuring the sensor using the sensor configuration parameters, the system configures the sensor using the sensor configuration parameters.

4. The method according to claim 2, characterized in that, The sensor data is sampled to generate environmental point cloud data and RGB image data, specifically including: The coordinate space of each image data is transformed according to the calibration information to achieve spatial unification of the image data and obtain a unified coordinate data acquisition point cloud and RGB image; The unified coordinate data collection point cloud is sequentially subjected to dimensional expansion, sampling and point cloud normalization processing to obtain environmental point cloud data. The steps of sampling and point cloud normalization processing of the dimensionally expanded point cloud include sampling the farthest point of the dimensionally expanded point cloud to obtain the number of input point clouds, and scaling and normalizing the dimensionally expanded point cloud. The RGB image is sequentially resized and pixel normalized to obtain RGB image data. The steps for resizing the RGB image include: resizing the RGB image to a set size, and performing cubic interpolation using the gray values ​​of a set number of points around the sampling point to preserve image details, texture features, and process sharp edges and high-frequency information. The steps for pixel normalization of the resized RGB image include 0-1 normalization and mean-variance normalization. The 0-1 normalization scales the pixel values ​​of the image to the range of 0 to 1, and the mean-variance normalization adjusts the pixel values ​​of the image to a distribution with a mean of 0 and a standard deviation of 1.

5. The method according to claim 1, characterized in that, Based on the tunnel scene target, the environmental point cloud data and RGB image data are segmented to obtain segmented point cloud data and point cloud segmentation results of the tunnel scene target, specifically including: A multimodal 3D instance segmentation model is constructed. The multimodal 3D instance segmentation model takes the environmental point cloud data and RGB image data as input, extracts point cloud features and image features, and uses a channel attention mechanism to fuse point cloud features and image features. The point cloud features are enhanced based on the high-dimensional features of the RGB image, and the segmented point cloud data and the point cloud segmentation results of the tunnel scene target are output.

6. The method according to claim 1, characterized in that, Based on the point cloud segmentation results of the target in the tunnel scene, the point cloud data is calibrated and filtered to determine the key point cloud representing the contour of the target object, specifically including: The segmented point cloud data is filtered according to the scene to remove outliers and points that do not conform to the rules, and the key point cloud representing the outline of the target object is determined.

7. The method according to claim 6, characterized in that, The segmented point cloud data is filtered according to the scene, outliers and points that do not conform to the rules are removed, and key point clouds representing the contour of the target object are determined, specifically including: The segmented point cloud data is judged and filtered based on spatial location, color, shape, scene features, and prior knowledge to remove point cloud data that is irrelevant to the current scene. Outlier points in the segmented point cloud data are identified using outlier detection algorithms and iterative nearest neighbor algorithms. Based on the constraints, point cloud data that is irrelevant to the current scene and outliers are filtered to determine the key point cloud representing the outline of the target object. The constraints are determined based on the shape, size, and relative position of the target object.

8. A multimodal 3D instance segmentation tunnel spatial sensing device, used to implement the method as described in any one of claims 1 to 7, characterized in that, The device includes: The device access module is configured to acquire sensor data; The data preprocessing module is configured to sample the sensor data to generate environmental point cloud data and RGB image data; The 3D instance segmentation module is configured to divide the environmental point cloud data and RGB image data based on the tunnel scene target, and obtain the segmented point cloud data and the point cloud segmentation result of the tunnel scene target. The point cloud calibration and filtering module is configured to calibrate and filter the segmented point cloud data based on the point cloud segmentation results of the target in the tunnel scene, so as to determine the key point cloud representing the outline of the target object.

9. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, perform the method according to any one of claims 1 to 7.

Images