Semantic map construction method and control system of robot

Abstract

The application provides a semantic map construction method and a control system of a robot, wherein the method comprises: acquiring a panoramic image and point cloud data; inputting the panoramic image into a panoramic image encoder for multi-scale encoding processing to generate panoramic feature maps at multiple scales; determining a reference point set of the point cloud data in a camera coordinate system of the panoramic image; performing cross-modal feature fusion processing according to a prior semantic network, the reference point set and the panoramic feature maps at the multiple scales to obtain a target feature map; and generating a semantic map according to the target feature map and a decoder. The application can guide feature sampling offset through geometric constraints, reduce spherical projection distortion, and fuse the advantages of panoramic cameras and laser point clouds, realize geometric compatibility alignment of panoramic vision and laser radar, solve the perception bottleneck of robots in complex environments, convert physical limitations into perception advantages, and significantly improve the perception ability of robots in complex environments.

Description

Technical Field

[0001] This application relates to the field of visual perception technology, and more specifically, to a semantic map construction method and a robot control system. Background Technology

[0002] Robot environmental perception is key to its interaction with the outside world. By fusing multiple sensors, robots can accurately identify the environment, objects, and their own state, enabling intelligent interaction and action.

[0003] Existing robotic environment perception processes can achieve environmental perception by stitching together image features and point cloud features acquired by panoramic cameras and LiDAR. Specifically, a panoramic camera acquires a panoramic image of the environment, and LiDAR obtains point cloud data of the surrounding environment. After extracting point cloud features, the extracted point cloud features are projected onto the coordinate system of the panoramic image and then stitched together. This integrates the feature information from two different modalities, thereby achieving environmental perception.

[0004] However, this method cannot handle dynamic occlusion caused by the robot's biomimetic structures (such as joint movements), resulting in poor performance in dynamic environments. In addition, although panoramic images can provide a 360° field of view, they suffer from feature degradation problems under spherical projection (such as linear structure breakage, scale inconsistency, and depth perception distortion). Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a semantic map construction method and a robot control system, so as to solve the problems of poor performance in dynamic environments and feature degradation under spherical projection in the prior art.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0007] In a first aspect, one embodiment of this application provides a semantic map construction method, the method comprising:

[0008] Acquire panoramic images and point cloud data, wherein the point cloud data includes multiple point clouds;

[0009] The panoramic image is input into a pre-trained panoramic image encoder for multi-scale encoding processing to generate panoramic feature maps at multiple scales.

[0010] Determine the set of reference points for the point cloud data in the camera coordinate system of the panoramic image;

[0011] Cross-modal feature fusion processing is performed based on the pre-trained prior semantic network, the reference point set, and the panoramic feature maps at multiple scales to obtain a target feature map. The target feature map is a feature map in the BEV space, including geometric and semantic information. The prior semantic network corresponds to the target spatial range, which is determined based on the field of view of the panoramic camera and the LiDAR. The prior semantic network includes multiple grids, each grid corresponding to a spatial sub-range, and each grid has semantic information. The semantic information is used to characterize the object of interest of the grid.

[0012] A semantic map is generated based on the target feature map and the pre-trained decoder.

[0013] Optionally, determining the set of reference points for the point cloud data in the camera coordinate system of the panoramic image includes:

[0014] The point cloud data is mapped to the camera coordinate system to obtain the target point corresponding to each point cloud in the camera coordinate system.

[0015] Project the target points corresponding to each point cloud in the camera coordinate system onto the depth image plane to generate a depth map;

[0016] The depth map is back-projected to obtain multiple reference points, and the set of multiple reference points is taken as the reference point set.

[0017] Optionally, the step of performing cross-modal feature fusion processing based on the pre-trained prior semantic network, the reference point set, and the panoramic feature maps at multiple scales to obtain the target feature map includes:

[0018] Semantic geometry is dynamically bound based on the prior semantic network and the set of reference points to generate a set of sampling indexes. The set of sampling indexes includes multiple sampling indexes, each of which corresponds to a reference point, and each sampling index is used to characterize the position of the corresponding reference point in the panoramic feature map.

[0019] Based on the sampling index set, adaptive feature sampling and aggregation processing are performed on panoramic feature maps at multiple scales to obtain the target feature map.

[0020] Optionally, the step of dynamically binding semantic geometry based on the prior semantic network and the reference point set to generate a sampling index set includes:

[0021] Each reference point in the set of reference points is transformed to obtain the spherical angle corresponding to each reference point. The spherical angle includes: azimuth angle and elevation angle.

[0022] The linear layer obtained through pre-training offsets the spherical angles corresponding to each reference point based on the prior semantic network to obtain the corrected spherical coordinates of each reference point.

[0023] Calculate the sampling index corresponding to each reference point based on the corrected spherical coordinates of each reference point.

[0024] Optionally, the step of adaptively sampling and aggregating panoramic feature maps at multiple scales according to the sampling index set to obtain the target feature map includes:

[0025] Based on the sampling index set, generate sampling index sets at multiple scales;

[0026] Based on the prior semantic network, determine the offset information corresponding to the panoramic feature maps at each scale;

[0027] Based on the sampling index set at multiple scales and the offset information corresponding to the panoramic feature map at each scale, sampling features at each scale are obtained.

[0028] Based on the sampling features at each scale, the target feature map is obtained by aggregation.

[0029] Optionally, generating a set of sampling indexes at multiple scales based on the sampling index set includes:

[0030] Obtain the height and width of the panoramic image;

[0031] Obtain the height and width of panoramic feature maps at multiple scales;

[0032] Based on the height of the panoramic image, the height of the panoramic feature maps at multiple scales, and the sampling index set, determine the row components of the sampling index corresponding to each reference point at multiple scales.

[0033] Based on the width of the panoramic image, the width of the panoramic feature maps at multiple scales, and the sampling index set, determine the column components of the sampling index corresponding to each reference point at multiple scales;

[0034] A set of sampling indices for multiple scales is generated based on the row components of the sampling indices corresponding to each reference point at multiple scales and the column components of the sampling indices corresponding to each reference point at multiple scales.

[0035] Optionally, determining the offset information corresponding to the panoramic feature maps at each scale based on the prior semantic network includes:

[0036] The prior semantic network is input into the pre-trained linear layer corresponding to the first scale to generate offset information corresponding to the panoramic feature map at the first scale, where the first scale can be any scale.

[0037] Optionally, the step of aggregating the target feature map based on the sampling features at each scale includes:

[0038] Based on the sampling features at each scale and the prior semantic network, feature projection is performed to generate projection matrices, first projected features, and second projected features at each scale.

[0039] Based on the projection matrix at each scale and the first projected features at each scale, the attention weights at each scale are calculated.

[0040] The target feature map is obtained by aggregating the attention weights at each scale and the second projected features at each scale.

[0041] Optionally, the decoder includes: a convolutional layer, multiple cascaded upsampling modules, and a classification head; the step of generating a semantic map based on the target feature map and the pre-trained decoder includes:

[0042] The target feature map is reconstructed into a grid to generate a spatial grid feature map;

[0043] The spatial grid feature map is input into the convolutional layer for aggregation processing to generate aggregated features;

[0044] The aggregated features are input into the cascaded multiple upsampling modules for progressive upsampling processing to generate target size features;

[0045] A semantic map is generated based on the target size features and the classification head.

[0046] Secondly, another embodiment of this application provides a robot control system, the robot control system including: a processor and a memory, the memory storing machine-readable instructions executable by the processor, the processor executing the machine-readable instructions when the robot control system is running to perform the steps of the method described in the first aspect.

[0047] The beneficial effects of this application are as follows: By acquiring panoramic images and point cloud data, and inputting the panoramic images into a pre-trained panoramic image encoder for multi-scale encoding processing, panoramic feature maps at multiple scales are generated, and a set of reference points for the point cloud data in the camera coordinate system of the panoramic image is determined. This enables cross-modal feature fusion processing based on a pre-trained prior semantic network, the set of reference points, and panoramic feature maps at multiple scales to obtain a target feature map. Furthermore, based on the target feature map and a pre-trained decoder, a semantic map is generated. This leverages the ray characteristics of the panoramic camera, guiding feature sampling offset through geometric constraints to reduce spherical projection distortion. It also integrates the advantages of panoramic cameras and laser point clouds, achieving geometric compatibility alignment between panoramic vision and LiDAR, overcoming the limitations of a single sensor, efficiently aggregating hierarchical 3D features from panoramic images and point cloud data, capturing the complete representation of distorted objects in real 3D space, and achieving cross-modal alignment of geometric perception. This enables real-time, high-precision semantic mapping for robots in dynamic environments, solving the perception bottleneck of robots in complex environments, transforming physical limitations into perception advantages, and significantly improving the robot's perception capabilities in complex environments. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart of a semantic map construction method provided in an embodiment of this application;

[0050] Figure 2 A flowchart illustrating the process of determining the set of reference points for point cloud data in the camera coordinate system of a panoramic image in the semantic map construction method provided in this application embodiment;

[0051] Figure 3 A flowchart illustrating the process of obtaining a target feature map in the semantic map construction method provided in this application embodiment;

[0052] Figure 4 This is a flowchart illustrating the generation of a sampling index set in the semantic map construction method provided in this application embodiment;

[0053] Figure 5 Another flowchart for obtaining the target feature map in the semantic map construction method provided in the embodiments of this application;

[0054] Figure 6A flowchart illustrating the generation of a set of sampling indexes at multiple scales in the semantic map construction method provided in this application embodiment;

[0055] Figure 7 This is a flowchart illustrating the aggregation process for obtaining a target feature map in the semantic map construction method provided in this application embodiment.

[0056] Figure 8 This is a schematic diagram of the decoder structure in the semantic map construction method provided in the embodiments of this application;

[0057] Figure 9 A flowchart illustrating the generation of a semantic map in the semantic map construction method provided in this application embodiment;

[0058] Figure 10 This is a schematic diagram of the control system of the robot provided in an embodiment of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0060] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0061] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0062] Existing robotic environment perception processes can achieve environmental perception by stitching together image features and point cloud features acquired by panoramic cameras and LiDAR. Specifically, a panoramic camera acquires a panoramic image of the environment, and LiDAR obtains point cloud data of the surrounding environment. After feature extraction, the extracted point cloud features are projected onto the coordinate system of the panoramic image. Then, the point cloud features are aligned with the image features, and the image features and point cloud features are stitched together along the channel dimension. This integrates the feature information from two different modalities, thereby achieving environmental perception.

[0063] However, this method cannot handle dynamic occlusion caused by the robot's biomimetic structures (such as joint movements), resulting in poor performance in dynamic environments. In addition, although panoramic images can provide a 360° field of view, they suffer from feature degradation problems under spherical projection (such as linear structure breakage, scale inconsistency, and depth perception distortion), that is, they ignore the geometric mismatch between modalities.

[0064] In summary, existing technologies for robot environmental perception suffer from severe self-occlusion and limited field of view (FOV) problems.

[0065] Based on the aforementioned problems, this application proposes a semantic map construction method. It acquires panoramic images and point cloud data, inputs the panoramic images into a pre-trained panoramic image encoder for multi-scale encoding processing, generates panoramic feature maps at multiple scales, and determines the reference point set of the point cloud data in the camera coordinate system of the panoramic image. This allows for cross-modal feature fusion processing based on a pre-trained prior semantic network, the reference point set, and the panoramic feature maps at multiple scales to obtain a target feature map. A semantic map is then generated based on the target feature map and a pre-trained decoder. This method leverages the ray characteristics of the panoramic camera, guides feature sampling offset through geometric constraints to reduce spherical projection distortion, and integrates the advantages of panoramic cameras and laser point clouds. It achieves geometric compatibility alignment between panoramic vision and LiDAR, overcoming the limitations of a single sensor. It efficiently aggregates hierarchical 3D features from panoramic images and point cloud data, capturing the complete representation of distorted objects in real 3D space, and achieving cross-modal alignment of geometric perception. This enables real-time, high-precision semantic mapping for robots in dynamic environments, solving the perception bottleneck of robots in complex environments, transforming physical limitations into perception advantages, and significantly improving the robot's perception capabilities in complex environments.

[0066] First, the relevant background of the semantic map construction method provided in the embodiments of this application will be explained.

[0067] It should be understood that the relevant background involved in the semantic map construction method provided in the embodiments of this application can be deployed in any device that needs to perform dynamic environmental perception, such as a robot, specifically a humanoid robot or a humanoid robot.

[0068] For example, by executing the semantic map construction method provided in the embodiments of this application, the robot can acquire a semantic map in real time, thereby enabling it to understand the environment more intelligently, interact with objects, and perform complex tasks based on the semantic map.

[0069] For example, the robot can be a humanoid robot. Specifically, a humanoid robot is a type of robot that imitates the appearance, movement, or behavior patterns of a human. It typically possesses human-like structures such as a head, torso, arms, and legs, and can perform human-like functions such as walking, grasping, and facial expression interaction. Furthermore, each arm is composed of multiple joint mechanisms, enabling it to perform different actions depending on the scenario.

[0070] Specifically, a robot may include a mechanical structure, a drive system, a sensing system, a control system, and a power supply system.

[0071] For example, the mechanical structure includes at least one robotic arm, an end effector corresponding to each robotic arm, two legs, and two feet, and the mechanical structure is used to perform specific actions.

[0072] The drive system includes servo motors, etc., which are used to control the rotation or linear motion of each joint in the robotic arm and provide precise power output.

[0073] The sensing system includes a panoramic camera and a LiDAR sensor. Optionally, the sensing system may also include a torque sensor, a vision camera, an inertial measurement unit, and a tactile sensor. The sensing system is used to detect environmental information in real time to obtain real-time environmental information in order to control the robot.

[0074] Optionally, a panoramic camera can be mounted on the robot's head or top torso, ensuring 360° horizontal and wide vertical coverage (e.g., ±90°) to simulate human vision.

[0075] For example, the resolution of the panoramic camera can be 512×1024, covering a 360° horizontal field of view (hereinafter referred to as horizontal FOV) and a 180° vertical field of view (hereinafter referred to as vertical FOV).

[0076] Alternatively, the lidar can be fixed to the upper middle part of the robot's torso, tilted slightly downwards (e.g., 10°-15°) to cover ground obstacles.

[0077] The control system includes a processor, a memory, and a communication module. The memory stores machine-readable instructions that the processor can execute. When the robot's control system is running, the processor executes the machine-readable instructions to process and store the sensor data obtained by the sensing system, and executes the steps of the semantic map construction method provided in this application embodiment to plan tasks for the robot and thus control the robot's operation.

[0078] Optionally, the robot may also include a human-computer interaction interface to enable human-computer interaction.

[0079] The semantic map construction method provided in this application will be described in detail below with reference to several embodiments.

[0080] Figure 1 A flowchart of a semantic map construction method provided in an embodiment of this application is shown below. Figure 1 As shown, the executing entity of this method can be any electronic device with processing capabilities, such as the control system of the robot described above. The method includes:

[0081] S101. Acquire panoramic images and point cloud data.

[0082] Optionally, panoramic images can be acquired in real time using a panoramic camera, and point cloud data can be acquired in real time using a LiDAR. The point cloud data includes multiple point clouds.

[0083] For example, a panoramic image of the current frame and point cloud data of the current frame can be obtained to construct a semantic map of the current frame.

[0084] S102. Input the panoramic image into the pre-trained panoramic image encoder for multi-scale encoding processing to generate panoramic feature maps at multiple scales.

[0085] Optionally, the panoramic image can be input into a pre-trained panoramic image encoder to perform multi-scale encoding processing on the panoramic image, generating panoramic feature maps at multiple scales.

[0086] Optionally, the panoramic image can be input into a pre-trained panoramic image encoder, and spherical projection distortion can be processed by spherical convolution and multi-scale encoding can be performed to generate panoramic feature maps at multiple scales.

[0087] For example, a panoramic image can be input into a pre-trained panoramic image encoder, which first preprocesses the panoramic image. The preprocessing includes: normalizing the pixel values ​​and then injecting spherical coordinates (θ, ...). As an additional channel, it enhances geometry awareness. Specifically, injecting spherical coordinates involves: calculating the spherical coordinates (θ, θ) of each pixel. (and splice them into a 5-channel input, thereby explicitly encoding the spherical geometric relationships.)

[0088] For example, a panoramic image encoder performs hierarchical feature extraction on the preprocessed image features to generate panoramic feature maps at multiple scales. Specifically, the preprocessed image features are first processed sequentially through convolution, batch normalization, 4x8 local window attention, and relative position encoding to generate high-resolution shallow features. Then, 3x3 convolution is used for downsampling, halving the size to 125x256, and 8x16 local window attention is applied to generate medium-resolution features. Convolution downsampling is repeated, successively reducing the size to 64x128 and 32x64, and global attention is applied to each to generate low-resolution deep features. Specifically, the global attention processing includes: employing spherical adaptive attention at higher levels, for example, weighting the spherical distance in Q, K, and V calculations to preserve spherical distortion.

[0089] For example, the obtained high-resolution shallow features, medium-resolution features, and low-resolution deep features can be horizontally connected to obtain an output feature pyramid, with each layer's resolution corresponding to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original input, respectively.

[0090] S103. Determine the set of reference points for the point cloud data in the camera coordinate system of the panoramic image.

[0091] Optionally, the point cloud data can be transformed into the camera coordinate system using a pre-calibrated extrinsic matrix, generating reference points for the point cloud data in the camera coordinate system of the panoramic image. Uniform sampling is then performed to obtain a set of reference points for the point cloud data in the camera coordinate system of the panoramic image. The pre-calibrated extrinsic matrix refers to the rigid transformation matrix that transforms the point cloud data from the point cloud coordinate system to the panoramic camera coordinate system, representing the geometric alignment between the point cloud coordinate system and the panoramic camera coordinate system.

[0092] For example, the point cloud data can be transformed into the camera coordinate system by using a pre-calibrated extrinsic parameter matrix, generating reference points of the point cloud data in the camera coordinate system of the panoramic image, and filtering out noise points, thereby obtaining a set of reference points of the point cloud data in the camera coordinate system of the panoramic image.

[0093] S104. Based on the pre-trained prior semantic network, reference point set, and panoramic feature maps at multiple scales, perform cross-modal feature fusion processing to obtain the target feature map.

[0094] It is understandable that the prior semantic network obtained through pre-training is optimized through the training process and can encode the distribution of typical objects in different locations. Furthermore, each grid in the prior semantic network corresponds to a sub-range in space, forming a spatially structured topologically ordered representation.

[0095] Optionally, after obtaining the set of reference points and panoramic feature maps at multiple scales, geometrically aware cross-modal feature alignment and fusion processing can be performed based on the pre-trained prior semantic network, the set of reference points, and the panoramic feature maps at multiple scales to obtain the target feature map.

[0096] This means that in the process of generating target feature maps, the initial semantic hypothesis can be provided by the prior semantic network obtained through pre-training, and geometric correction and feature fusion can be performed by combining the reference point set and panoramic feature maps at multiple scales. In this way, visual evidence can be extracted from the panoramic image and the reference point set to verify or correct the initial semantic hypothesis, thereby obtaining the target feature map. The target feature map can be understood as a weighted synthesis of prior knowledge and real-time sensor evidence.

[0097] For example, geometric constraints can be generated from panoramic feature maps at multiple scales, and cross-modal perception can be performed based on the geometric constraints, a pre-trained prior semantic network, and a set of reference points, thereby achieving geometric alignment between panoramic images and point cloud data, and aggregating hierarchical 3D features from panoramic images and point cloud data to obtain target feature maps.

[0098] The target feature map is a feature map in the BEV space, which includes geometric information and semantic information. The prior semantic network corresponds to the target spatial range, which is determined based on the field of view of the panoramic camera and the LiDAR. The prior semantic network includes multiple grids, each grid corresponding to a spatial sub-range, and each grid has semantic information. The semantic information is used to characterize the object of interest of the grid.

[0099] For example, the prior semantic network can be the BEV query space, which can be learned through a query matrix. The representation is performed, where the parameter matrix includes multiple query vectors. The prior semantic network is essentially a set of dynamic request vectors used to generate features for the BEV semantic map. Each query vector in this parameter matrix corresponds to a preset location in the semantic map, and each query vector is used to determine the features needed for that location from the panoramic image and point cloud data.

[0100] For example, the prior semantic network is learnable and can be trained. During training, the learnable parameter matrix is ​​defined as the model parameters, randomly initialized, and then automatically optimized through training. Simultaneously, the 2D coordinate information of the corresponding grid is injected into each query for positional encoding enhancement and grid mapping, thereby achieving the association between the prior semantic network, the grid, and the target spatial range.

[0101] For example, the target spatial range is determined based on the field of view of the panoramic camera and LiDAR, and can be understood as the map range of the final semantic map. The spatial sub-range is a sub-range in the final semantic map.

[0102] S105. Generate a semantic map based on the target feature map and the pre-trained decoder.

[0103] Optionally, the decoder can be a BEV decoder. After obtaining the target feature map, the target feature map can be decoded and upsampled according to the pre-trained decoder, thereby gradually converting the low-resolution BEV features into a high-resolution semantic segmentation map to obtain a semantic map.

[0104] The semantic map can be understood as a lightweight and semantically-oriented bird's-eye view, including both geometric and semantic information, serving as a semantic representation of the environment. Geometric information can be understood as the location of obstacles, while semantic information can be understood as object category information. This semantic map includes: the perception range (X, Y) of the BEV space and the predicted number of categories.

[0105] In this embodiment, panoramic images and point cloud data are acquired, and the panoramic images are input into a pre-trained panoramic image encoder for multi-scale encoding processing to generate panoramic feature maps at multiple scales. A set of reference points for the point cloud data in the camera coordinate system of the panoramic image is determined. This allows for cross-modal feature fusion processing based on a pre-trained prior semantic network, the reference point set, and the panoramic feature maps at multiple scales to obtain a target feature map. A semantic map is then generated based on the target feature map and a pre-trained decoder. This approach leverages the ray characteristics of the panoramic camera to guide feature sampling offset through geometric constraints, reducing spherical projection distortion. It also integrates the advantages of panoramic cameras and laser point clouds, achieving geometric compatibility alignment between panoramic vision and LiDAR, overcoming the limitations of a single sensor, efficiently aggregating hierarchical 3D features from panoramic images and point cloud data, capturing the complete representation of distorted objects in real 3D space, and achieving cross-modal alignment of geometric perception. This enables real-time, high-precision semantic mapping for robots in dynamic environments, solving the perception bottleneck of robots in complex environments, transforming physical limitations into perception advantages, and significantly improving the robot's perception capabilities in complex environments.

[0106] In one possible implementation, Figure 2 This is a flowchart illustrating the process of determining the set of reference points for point cloud data in the camera coordinate system of a panoramic image within the semantic map construction method provided in this application embodiment. (Refer to...) Figure 2 As shown, the set of reference points for determining the point cloud data in the camera coordinate system of the panoramic image in S103 above includes:

[0107] S201. Map the point cloud data to the camera coordinate system to obtain the target points corresponding to each point cloud in the camera coordinate system.

[0108] Optionally, the point cloud data is transformed into the camera coordinate system using a pre-calibrated extrinsic parameter matrix, generating target points in the panoramic image's camera coordinate system, thereby eliminating points outside the panoramic camera's field of view. The pre-calibrated extrinsic parameter matrix refers to the rigid body transformation matrix that transforms the point cloud from its coordinate system to the panoramic camera coordinate system, representing the geometric alignment between the point cloud coordinate system and the panoramic camera coordinate system.

[0109] S202. Project the target points corresponding to each point cloud in the camera coordinate system onto the depth image plane to generate a depth map.

[0110] The depth image plane is a two-dimensional grid aligned with the pixels of the panoramic image, and each grid point stores the depth value of the corresponding 3D spatial point.

[0111] Optionally, the pitch angle θ and azimuth angle of the target point corresponding to each point cloud in the camera coordinate system can be calculated. And the pitch angle θ and azimuth angle Convert to pixel coordinates (u, v), calculate the depth value of each point, and generate a depth map.

[0112] For example, taking the coordinates of the target point corresponding to the point cloud in the camera coordinate system as (x, y, z), the pitch angle θ and azimuth angle can be calculated by referring to the following formula. :

[0113]

[0114]

[0115] For example, and referring to the following formula, the pitch angle θ and azimuth angle ϕ are converted into pixel coordinates:

[0116]

[0117] Where W is the width of the panoramic image and H is the length of the panoramic image.

[0118] For example, the depth value of that point can be obtained. .

[0119] S203. Perform back projection on the depth map to obtain multiple reference points, and use the set of multiple reference points as the reference point set.

[0120] Optionally, each point in the depth map is back-projected into 3D space to obtain multiple reference points, and the set of multiple reference points is used as a reference point set, thereby associating the depth information with the 3D spatial structure.

[0121] By obtaining a set of reference points through back projection, the 2D perception results can be mapped back to 3D space through the inverse transformation of imaging geometry. This compensates for the sparsity of point cloud data, generates a more complete scene geometry, and bridges visual features with depth information to achieve modal alignment. Furthermore, it can be applied to any camera model.

[0122] In one possible implementation, Figure 3 This is a flowchart illustrating the process of obtaining a target feature map in the semantic map construction method provided in this application embodiment, with reference to... Figure 3 As shown, in S104 above, cross-modal feature fusion processing is performed based on the pre-trained prior semantic network, reference point set, and panoramic feature maps at multiple scales to obtain the target feature map, including:

[0123] S301. Perform semantic geometry dynamic binding based on the prior semantic network and the reference point set to generate a sampling index set.

[0124] Optionally, the reference point set can be precisely mapped to the panoramic image coordinate system through spherical projection and prior semantic network to establish geometric alignment and generate a sampling index set, thereby eliminating the projection distortion of traditional stitching, reducing category confusion caused by depth mismatch, and preserving the geometric consistency of the 360° field of view.

[0125] The sampling index set includes multiple sampling indices. Each sampling index corresponds to a reference point, and each sampling index is used to represent the position of the corresponding reference point in the panoramic feature map.

[0126] By providing semantic priors through a prior semantic network and combining them with the location of a set of reference points, it is possible to determine where to extract features from the panoramic image and how to populate them into the semantic map.

[0127] S302. Based on the sampling index set, perform adaptive feature sampling and aggregation processing on the panoramic feature maps at multiple scales to obtain the target feature map.

[0128] Optionally, adaptive feature sampling extraction and aggregation can be performed from panoramic feature maps at multiple scales according to the sampling index set, and then filled into the corresponding positions in the prior semantic network to obtain the target feature map. This allows for dynamic adjustment of feature sampling positions and adaptive compensation for occlusion and sparsity (such as bypassing arm occlusion to complete the features of the blocked object).

[0129] In one possible implementation, Figure 4 This is a flowchart illustrating the generation of a sampling index set in the semantic map construction method provided in this application embodiment, with reference to... Figure 4 As shown, S301 above performs semantic-geometric dynamic binding based on the prior semantic network and the reference point set, generating a sampling index set, including:

[0130] S401. Transform each reference point in the reference point set to obtain the spherical angle corresponding to each reference point.

[0131] Optionally, each reference point in the reference point set can be transformed using spherical projection to calculate the spherical angle corresponding to each reference point. This establishes the projection relationship between each reference point and the panoramic image. The spherical angle includes the azimuth angle. and pitch angle .

[0132] S402. The linear layer obtained through pre-training offsets the spherical angles corresponding to each reference point based on the prior semantic network to obtain the corrected spherical coordinates corresponding to each reference point.

[0133] Optionally, a linear layer obtained through pre-training f Δ s The offset is predicted by performing offset prediction on the spherical angle corresponding to each reference point. It generates corrected spherical coordinates S' for each reference point, thereby enabling geometrically perceptual feature fusion through the corrected spherical coordinates S' to correct the spherical distortion of the panoramic image.

[0134] For example, the offset prediction can be performed using the following formula:

[0135]

[0136] in, This is the query matrix corresponding to the prior semantic network. For a pre-trained linear layer, PosEnc(•) is the position encoding function, and S is the initial spherical angular coordinate.

[0137] For example, the corrected spherical coordinates S' corresponding to each reference point are generated by referring to the following formula:

[0138]

[0139] S403. Calculate the sampling index corresponding to each reference point based on the corrected spherical coordinates of each reference point.

[0140] Optionally, the corrected spherical coordinates S' corresponding to each reference point are mapped to the width range W and height range H of the panoramic image, and the sampling index corresponding to each reference point is calculated. The sampling index corresponding to each reference point points to the position in the panoramic feature map that is precisely aligned with each reference point.

[0141] For example, the sampling index corresponding to each reference point can be calculated by referring to the following formula. :

[0142]

[0143]

[0144] Where H represents the height range of the panoramic image, and W represents the width range of the panoramic image.

[0145] By calculating the sampling index corresponding to each reference point using the corrected spherical coordinates, we can ensure the geometric alignment of the point cloud data with the panoramic image, avoid errors introduced by distortion, and reduce modal fusion errors.

[0146] In one possible implementation, Figure 5 Another flowchart for obtaining the target feature map in the semantic map construction method provided in this application embodiment is shown below. Figure 5 As shown, in step S302 above, adaptive feature sampling and aggregation are performed on panoramic feature maps at multiple scales based on the sampling index set to obtain the target feature map, including:

[0147] S501. Generate sampling index sets for multiple scales based on the sampling index set.

[0148] It can be understood that the sampling index set is a sampling index set at a single scale. The sampling index set at a single scale can be used to generate sampling index sets at multiple scales according to the different scales of each panoramic feature map, thereby ensuring that the sampling index set can be mapped to the corresponding position on feature maps of different resolutions.

[0149] Optionally, the sampling index set can be normalized according to different scales to generate sampling index sets at multiple scales.

[0150] S502. Based on the prior semantic network, determine the offset information corresponding to the panoramic feature map at each scale.

[0151] Optionally, deformable offset prediction can be performed on the panoramic feature maps at various scales based on a prior semantic network to obtain the offset information corresponding to the panoramic feature maps at each scale. The offset information corresponding to the panoramic feature maps at each scale is used to adapt to geometric deformations at different resolutions.

[0152] S503. Based on the sampling index set at multiple scales and the offset information corresponding to the panoramic feature map at each scale, sample features at each scale are obtained.

[0153] Optionally, sampling features at the same scale are obtained by sampling the set of sampling indices at the same scale and the offset information corresponding to the panoramic feature map at the same scale.

[0154] Specifically, at the same scale, the offset information corresponding to each sampling index in the sampling index set and the panoramic feature map are superimposed, and the superimposed result is sampled by bilinear interpolation to obtain the sampling features at that scale.

[0155] S504. Based on the sampling features at each scale, aggregate to obtain the target feature map.

[0156] Optionally, the target feature map can be obtained by combining the sampling features at each scale with the prior semantic network to calculate attention weights and aggregate multi-scale features.

[0157] In one possible implementation, Figure 6 This is a flowchart illustrating the generation of a set of sampling indexes at multiple scales in the semantic map construction method provided in this application embodiment, with reference to... Figure 6 As shown, in S501 above, a sampling index set at multiple scales is generated based on the sampling index set, including:

[0158] S601, Obtain the height and width of the panoramic image.

[0159] Optionally, the height of the panoramic image can be obtained. and width .

[0160] S602. Obtain the height and width of panoramic feature maps at multiple scales.

[0161] Optionally, the height and width of panoramic feature maps at multiple scales can be obtained.

[0162] S603. Based on the height of the panoramic image, the height of the panoramic feature maps at multiple scales, and the sampling index set, determine the row components of the sampling index corresponding to each reference point at multiple scales.

[0163] Optionally, the height of the panoramic image can be used as a reference. The height of the panoramic feature map at multiple scales and the set of sampling indices are used to calculate the row components of the sampling index corresponding to each reference point at multiple scales.

[0164] For example, the height of the panoramic image can be used as a reference. , Height of panoramic feature map at scale and the sampling index corresponding to each reference point Calculations yielded The row components of the sampling index corresponding to each reference point at the scale.

[0165] Specifically, it can be calculated Height of panoramic feature map at scale Row components of the sampling index corresponding to each reference point The product of the product and the height of the panoramic image are calculated. The ratio, as The row components of the sampling index corresponding to each reference point at the scale.

[0166] S604. Based on the width of the panoramic image, the width of the panoramic feature maps at multiple scales, and the sampling index set, determine the column components of the sampling index corresponding to each reference point at multiple scales.

[0167] Optionally, based on the width of the panoramic image The width of the panoramic feature map at multiple scales and the set of sampling indices are used to calculate the column components of the sampling index corresponding to each reference point at multiple scales.

[0168] For example, it can be based on the width of the panoramic image , Width of panoramic feature map at scale and the sampling index corresponding to each reference point Calculations yielded The column components of the sampling index corresponding to each reference point at the scale.

[0169] Specifically, it can be calculated Width of panoramic feature map at scale Column components of the sampling index corresponding to each reference point The product of the two images is calculated, and the product is multiplied by the width of the panoramic image. The ratio, as The column components of the sampling index corresponding to each reference point at the scale.

[0170] S605. Generate a set of sampling indices for multiple scales based on the row components of the sampling indices corresponding to each reference point at multiple scales and the column components of the sampling indices corresponding to each reference point at multiple scales.

[0171] Optionally, the row and column components of the sampling index corresponding to each reference point at the same scale can be combined to generate a set of sampling indexes at multiple scales.

[0172] In one possible implementation, step S502 above determines the offset information corresponding to the panoramic feature maps at each scale based on a prior semantic network, including:

[0173] The prior semantic network is input into the pre-trained linear layer corresponding to the first scale to generate offset information corresponding to the panoramic feature map at the first scale.

[0174] The first scale can be any scale.

[0175] Optionally, the query matrix corresponding to the prior semantic network is predicted through a pre-trained linear layer at each scale to obtain 2D offset information at each scale, thereby achieving efficient prediction.

[0176] In one possible implementation, Figure 7 This is a flowchart illustrating the aggregation of target feature maps in the semantic map construction method provided in this application embodiment, with reference to... Figure 7 As shown, in S504 above, the target feature map is obtained by aggregating the sampling features at each scale, including:

[0177] S701. Based on the sampling features at each scale and the prior semantic network, feature projection is performed to generate projection matrices, first projected features, and second projected features at each scale.

[0178] Optionally, the pre-trained projection matrices Wq, Wk, and Wv can be obtained, and the sampled features at each scale and the prior semantic network can be projected onto the query, key, and value spaces respectively to generate the projection matrices at each scale, the first projected features at each scale, and the second projected features.

[0179] Specifically, each scale has the same projection matrix Wq, and different projection matrices Wk and Wv.

[0180] For example, with Taking scale as an example, Sampling features at scale and the query matrix corresponding to the prior semantic network Project them onto the query, key, and value spaces respectively to generate... Projection matrix at scale , First projected features at scale and the second projected features Specifically, it can be achieved by referring to the following formula:

[0181]

[0182]

[0183]

[0184] S702. Based on the projection matrix at each scale and the first projected features at each scale, calculate the attention weights at each scale.

[0185] Optionally, with Taking scale as an example, according to Projection matrix at scale as well as First projected features at scale Calculations yielded Attention weights at different scales For specific calculations, please refer to the following formula:

[0186]

[0187] in, This is a pre-defined scaling factor used to prevent the gradient from vanishing due to excessively large dot product values. The attention weights reflect the... l The importance of layer features to the current query vector.

[0188] S703. Based on the attention weights at each scale and the second projected features at each scale, the target feature map is aggregated.

[0189] Optionally, for the second projected features at each scale According to attention weights at each scale The results are weighted and summed at each scale to obtain the target feature map.

[0190] By projecting features from sampling features at various scales and prior semantic networks, projection matrices, first projected features, and second projected features at each scale are generated. Attention weights at each scale are calculated using the projection matrices and first projected features. Based on these attention weights and second projected features, the target feature map is aggregated. This approach allows attention weights to focus only on the semantic relevance of features without explicitly modeling geometry. It also enables features at different levels to compete for weights, avoiding redundancy and thus enabling adaptive selection of the optimal combination of features across multiple scales.

[0191] In one possible implementation, Figure 8 This is a schematic diagram of a decoder structure in the semantic map construction method provided in this application embodiment, with reference to... Figure 8 As shown, the decoder includes: convolutional layers, multiple cascaded upsampling modules, and a classification head. Figure 9 This is a flowchart illustrating the generation of a semantic map in the semantic map construction method provided in this application embodiment, with reference to... Figure 9 As shown, in step S105 above, a semantic map is generated based on the target feature map and the pre-trained decoder, including:

[0192] S901. Perform grid-based reconstruction of the target feature map to generate a spatial grid feature map.

[0193] Optionally, the target feature map can be reconstructed into a mesh, reshaping it into a 3D mesh to restore its spatial structure and generate a spatial mesh feature map. The spatial mesh feature map is a two-dimensional spatial mesh structure.

[0194] For example, a spatial grid feature map can be generated by reshaping the target feature map by mapping each BEV query to a 2D spatial location according to a preset grid order.

[0195] For example, after reshaping, 2D coordinate encoding of the BEV mesh can also be injected to explicitly preserve the spatial location information of the query.

[0196] S902. Input the spatial grid feature map into the convolutional layer for aggregation processing to generate aggregated features.

[0197] Optionally, the spatial grid feature map is input into the convolutional layer for global context aggregation processing to generate aggregated features.

[0198] S903. The aggregated features are input into multiple cascaded upsampling modules for progressive upsampling processing to generate target size features.

[0199] Optionally, the aggregated features are progressively upsampled through multiple cascaded upsampling modules to gradually improve the resolution.

[0200] For example, the aggregated features are first enlarged by transpose convolution, and then upsampled repeatedly. After each upsampling stage, convolution and non-linear activation are performed to enhance local consistency until the resolution reaches the target size, thus obtaining the target size features.

[0201] S904. Generate a semantic map based on the target size features and the classification head.

[0202] Optionally, the target size features are processed by a convolutional layer to predict the category, and the number of categories is activated by a classification head to generate a pixel-by-pixel category probability map. Then, through hard discrimination, the category with the highest probability is selected as the semantic label of the grid, thereby generating a semantic map.

[0203] Optionally, the semantic labels of each grid can be optimized using conditional random fields and morphological operations to convert them into a semantic map that the robot can understand.

[0204] By reconstructing the target feature map into a grid, a spatial grid feature map is generated. This spatial grid feature map is then input into a convolutional layer for aggregation, generating aggregated features. These aggregated features are then input into multiple cascaded upsampling modules for progressive upsampling, generating target size features. Based on these target size features and the classification head, a semantic map is generated. This approach enables accurate mapping from panoramic features to semantic maps. Furthermore, by incorporating geometric perception, redundant computation is reduced, computational efficiency is improved, and the real-time nature of the semantic map construction process is ensured.

[0205] It is understood that during the training phase, the query matrix corresponding to the prior semantic network in the semantic map construction method provided in this application embodiment can be trained using labeled data. Transformer parameters and linear layers in a panoramic image encoder f Δ s Optimize the parameters of learnable components.

[0206] Optionally, during the training phase, raw data such as panoramic images, point cloud data, and semantic labels of semantic networks can be acquired, and after synchronous data augmentation, the parameters of the learnable components can be trained through forward propagation, loss calculation, and backpropagation.

[0207] Specifically, the synchronization data enhancements include: horizontal flipping, random rotation, and MixUp blending.

[0208] For example, horizontal flipping includes: horizontally flipping the panoramic image, inverting the x-coordinate of the point cloud data, and synchronously flipping the semantic labels of the semantic network, thereby enhancing robustness in left-right symmetrical scenes.

[0209] For example, random rotation includes: rotating the panoramic image according to a preset spherical rotation angle, rotating the point cloud data around the Z-axis by a first preset angle, and synchronously rotating the semantic labels of the semantic network by a second preset angle.

[0210] For example, MixUp blending includes linearly blending any two panoramic images with semantic labels from a semantic network.

[0211] This application also provides a robot control system, such as... Figure 10 As shown, Figure 10 The schematic diagram of the robot control system provided in this application embodiment includes: a processor 1001 and a memory 1002, and optionally, a bus 1003. The memory 1002 stores machine-readable instructions executable by the processor 1001. When the robot control system is running, the processor 1001 and the memory 1002 communicate via the bus 1003. When the machine-readable instructions are executed by the processor 1001, the steps of the semantic map construction method described above are performed.

[0212] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0213] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0214] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A semantic map construction method characterized by, The method comprises the following steps: acquire a panoramic image and point cloud data, the point cloud data comprising a plurality of point clouds, and input the panoramic image into a pre-trained panoramic image encoder for multi-scale encoding processing to generate panoramic feature maps at multiple scales, and determine a reference point set of the point cloud data in a camera coordinate system of the panoramic image; convert each reference point in the reference point set to obtain a spherical angle corresponding to each reference point, and offset the spherical angle corresponding to each reference point based on a prior semantic network through a pre-trained linear layer to obtain a corrected spherical coordinate corresponding to each reference point, and calculate a sampling index corresponding to each reference point according to the corrected spherical coordinate corresponding to each reference point, wherein the spherical angle comprises an azimuth angle and a pitch angle; perform adaptive feature sampling and aggregation processing on the panoramic feature maps at multiple scales according to the sampling index set to obtain a target feature map, wherein the target feature map is a feature map in a BEV space, comprising geometric information and semantic information, the prior semantic network corresponds to a target space range, the target space range is determined according to the field of view range of the panoramic camera and the laser radar, each grid in the prior semantic network corresponds to a spatial sub-range, and each grid has semantic information, the semantic information is used to represent the object of interest of the grid, the sampling index set comprises a plurality of sampling indexes, each sampling index corresponds to a reference point, and each sampling index is used to represent the corresponding position of the corresponding reference point in the panoramic feature map; generate a semantic map according to the target feature map and a pre-trained decoder.

2. The semantic map construction method of claim 1, wherein, The determination of the reference point set of the point cloud data in the camera coordinate system of the panoramic image comprises: mapping the point cloud data to the camera coordinate system to obtain a target point corresponding to each point cloud in the camera coordinate system; projecting the target point corresponding to each point cloud in the camera coordinate system into a depth image plane to generate a depth map; back-projecting the depth map to obtain a plurality of reference points, and taking the set of the plurality of reference points as the reference point set.

3. The semantic map construction method of claim 1, wherein, The adaptive feature sampling and aggregation processing on the panoramic feature maps at multiple scales according to the sampling index set to obtain the target feature map comprises: generating a sampling index set at multiple scales according to the sampling index set; determining offset information corresponding to the panoramic feature maps at multiple scales according to the prior semantic network; sampling the sampling features at multiple scales according to the sampling index set at multiple scales and the offset information corresponding to the panoramic feature maps at multiple scales; aggregating the sampling features at multiple scales to obtain the target feature map.

4. The semantic map construction method according to claim 3, characterized in that, The generation of the sampling index set at multiple scales according to the sampling index set comprises: acquiring the height and width of the panoramic image; acquiring the height and width of the panoramic feature maps at multiple scales; determining the row component of the sampling index corresponding to each reference point at multiple scales according to the height of the panoramic image, the height of the panoramic feature maps at multiple scales, and the sampling index set. According to the width of the panoramic image, the width of the panoramic feature map at the plurality of scales, and the sampling index set, a column component of a sampling index corresponding to each reference point at the plurality of scales is determined; According to the row component of the sampling index corresponding to each reference point at the plurality of scales and the column component of the sampling index corresponding to each reference point at the plurality of scales, a sampling index set at the plurality of scales is generated.

5. The semantic map construction method of claim 3, wherein, The offset information corresponding to the panoramic feature map at each scale is determined according to the prior semantic network, including: The prior semantic network is input into a linear layer pre-trained at a first scale to generate offset information corresponding to the panoramic feature map at the first scale, and the first scale is any scale.

6. The semantic map construction method of claim 3, wherein, The target feature map is aggregated according to the sampling feature at each scale, including: According to the sampling feature at each scale and the feature projection of the prior semantic network, a projection matrix at each scale, a first projected feature at each scale, and a second projected feature at each scale are generated; According to the projection matrix at each scale and the first projected feature at each scale, an attention weight at each scale is calculated; According to the attention weight at each scale and the second projected feature at each scale, the target feature map is aggregated.

7. The semantic map construction method of claim 1, wherein, The decoder includes a convolutional layer, a plurality of cascaded upsampling modules, and a classification head; and the semantic map is generated according to the target feature map and the pre-trained decoder, including: The target feature map is grid-reduced to generate a spatial grid feature map; The spatial grid feature map is input into the convolutional layer for aggregation processing to generate an aggregated feature; The aggregated feature is input into the plurality of cascaded upsampling modules for step-by-step upsampling processing to generate a target size feature; According to the target size feature and the classification head, a semantic map is generated.

8. A control system of a robot characterized by, A processor and a memory are included, and the memory stores machine readable instructions executable by the processor; when the control system of the robot is running, the processor executes the machine readable instructions to perform the steps of the semantic map construction method according to any one of claims 1 to 7.

Images