Semantic map construction method and robot control system

Abstract

The invention provides a semantic map construction method and a robot control system, and the method comprises the steps: obtaining a panoramic image and point cloud data; inputting the panoramic image into a panoramic image encoder for multi-scale encoding processing, and generating panoramic feature maps under multiple scales; determining a reference point set of the point cloud data under a camera coordinate system of the panoramic image; performing cross-modal feature fusion processing according to the prior semantic network, the reference point set and the panoramic feature maps under the multiple scales to obtain a target feature map; and generating a semantic map according to the target feature map and a decoder. According to the method, feature sampling offset can be guided through geometric constraints, spherical projection distortion is reduced, the advantages of the panoramic camera and the laser point cloud are fused, geometric compatibility alignment of panoramic vision and laser radar is realized, the perception bottleneck of a robot in a complex environment is solved, physical constraints are converted into perception advantages, and the method is suitable for being popularized and applied. And the sensing capability of the robot in a complex environment is obviously improved.

Description

Technical Field

[0001] The present application relates to the field of visual perception technology, and in particular to a semantic map construction method and a robot control system. Background Art

[0002] The robot's environmental perception is the key to its interaction with the outside world. Through multi-sensor fusion, it can accurately identify the environment, objects and its own status, and realize intelligent interaction and action.

[0003] The existing robot environment perception process can achieve environmental perception by stitching together the image features and point cloud features of the environment captured by panoramic cameras and lidar. Specifically, the panoramic camera captures a panoramic image of the environment, and the lidar obtains point cloud data of the surrounding environment. After performing point cloud feature extraction, the extracted point cloud features are projected into the coordinate system of the panoramic image and then feature stitching is performed, thereby integrating the feature information of the two different modalities to achieve environmental perception.

[0004] However, this approach cannot handle dynamic occlusions caused by the robot's biomimetic structure (such as joint motion), resulting in poor performance in dynamic environments. Furthermore, while panoramic images provide a 360° field of view, they suffer from feature degradation issues under spherical projection (such as linear structure breakage, scale inconsistency, and depth perception distortion). Summary of the Invention

[0005] The purpose of this application is to address the deficiencies in the above-mentioned prior art and provide a semantic map construction method and a robot control system to solve the problems in the prior art such as poor performance in dynamic environments and feature degradation under spherical projection.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows: In a first aspect, an embodiment of the present application provides a method for constructing a semantic map, the method comprising: Acquire a panoramic image and point cloud data, wherein the point cloud data includes a plurality of point clouds; Inputting the panoramic image into a pre-trained panoramic image encoder for multi-scale encoding processing to generate panoramic feature maps at multiple scales; Determining a reference point set of the point cloud data in a camera coordinate system of the panoramic image; A cross-modal feature fusion process is performed based on the pre-trained prior semantic network, the reference point set, and the panoramic feature maps at multiple scales to obtain a target feature map, where the target feature map is a feature map in the BEV space, including geometric information and semantic information. The prior semantic network corresponds to the target space range, which is determined according to the field of view of the panoramic camera and the lidar. The prior semantic network includes multiple grids, each of which corresponds to a spatial sub-range, and each grid has semantic information, and the semantic information is used to characterize the object of interest of the grid; A semantic map is generated based on the target feature map and a pre-trained decoder.

[0007] Optionally, determining a reference point set of the point cloud data in a camera coordinate system of the panoramic image includes: Mapping the point cloud data to the camera coordinate system to obtain the target point corresponding to each point cloud in the camera coordinate system; Project the target points corresponding to each point cloud in the camera coordinate system onto the depth image plane to generate a depth map; Back-projecting is performed on the depth map to obtain a plurality of reference points, and a set consisting of the plurality of reference points is used as a reference point set.

[0008] Optionally, performing cross-modal feature fusion processing on the prior semantic network obtained by pre-training, the reference point set, and the panoramic feature maps at multiple scales to obtain a target feature map includes: Performing dynamic semantic geometry binding based on the prior semantic network and the reference point set to generate a sampling index set, wherein the sampling index set includes multiple sampling indices, each sampling index corresponds to a reference point, and each sampling index is used to represent the corresponding position of the corresponding reference point in the panoramic feature map; According to the sampling index set, adaptive feature sampling and aggregation processing are performed on the panoramic feature maps at multiple scales to obtain a target feature map.

[0009] Optionally, performing semantic geometry dynamic binding according to the prior semantic network and the reference point set to generate a sampling index set includes: Convert each reference point in the reference point set to obtain a spherical angle corresponding to each reference point, where the spherical angle includes an azimuth angle and a pitch angle; The linear layer obtained through pre-training offsets the spherical angle corresponding to each reference point based on the prior semantic network to obtain the corrected spherical coordinates corresponding to each reference point; According to the corrected spherical coordinates corresponding to each reference point, the sampling index corresponding to each reference point is calculated.

[0010] Optionally, performing adaptive feature sampling and aggregation processing on panoramic feature maps at multiple scales according to the sampling index set to obtain a target feature map includes: Generating sampling index sets at multiple scales according to the sampling index set; Determining offset information corresponding to the panoramic feature map at each scale based on the prior semantic network; According to the sampling index sets at multiple scales and the offset information corresponding to the panoramic feature maps at each scale, sampling is performed to obtain the sampling features at each scale; According to the sampling features at each scale, the target feature map is aggregated.

[0011] Optionally, generating sampling index sets at multiple scales according to the sampling index set includes: Obtaining the height and width of the panoramic image; Get the height and width of the panoramic feature map at multiple scales; Determining, according to the height of the panoramic image, the heights of the panoramic feature maps at the multiple scales, and the sampling index set, a row component of the sampling index corresponding to each reference point at the multiple scales; Determining, according to the width of the panoramic image, the widths of the panoramic feature maps at the multiple scales, and the sampling index set, a column component of the sampling index corresponding to each reference point at the multiple scales; A sampling index set at multiple scales is generated according to the row components of the sampling index corresponding to each reference point at multiple scales and the column components of the sampling index corresponding to each reference point at multiple scales.

[0012] Optionally, determining offset information corresponding to the panoramic feature map at each scale according to the prior semantic network includes: The prior semantic network is input into a pre-trained linear layer corresponding to a first scale to generate offset information corresponding to a panoramic feature map at the first scale, where the first scale is any scale.

[0013] Optionally, aggregating the target feature map based on the sampling features at each scale includes: Perform feature projection according to the sampling features at each scale and the prior semantic network to generate a projection matrix at each scale, a first projected feature at each scale, and a second projected feature at each scale; Calculate the attention weight at each scale according to the projection matrix at each scale and the first projection feature at each scale; According to the attention weights at each scale and the second projection features at each scale, the target feature map is aggregated.

[0014] Optionally, the decoder includes: a convolutional layer, a plurality of cascaded upsampling modules, and a classification head; generating a semantic map based on the target feature map and a pre-trained decoder includes: Performing grid restoration on the target feature map to generate a spatial grid feature map; Inputting the spatial grid feature map into the convolutional layer for aggregation processing to generate aggregated features; Inputting the aggregated features into the cascaded multiple upsampling modules for step-by-step upsampling processing to generate target size features; A semantic map is generated according to the object size feature and the classification head.

[0015] In the second aspect, another embodiment of the present application provides a control system for a robot, wherein the control system for the robot comprises: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the control system of the robot is running, the processor executes the machine-readable instructions to perform the steps of the method described in the first aspect.

[0016] The beneficial effects of the present application are: by acquiring panoramic images and point cloud data, and inputting the panoramic images into a pre-trained panoramic image encoder for multi-scale encoding processing, panoramic feature maps at multiple scales are generated, and a reference point set of the point cloud data in the camera coordinate system of the panoramic image is determined, so that cross-modal feature fusion processing can be performed based on the pre-trained prior semantic network, the reference point set, and the panoramic feature maps at multiple scales to obtain a target feature map, and a semantic map is generated based on the target feature map and the pre-trained decoder. The ray characteristics of the panoramic camera can be utilized to guide feature sampling offset through geometric constraints to reduce spherical projection distortion, and the advantages of the panoramic camera and laser point cloud can be integrated to achieve geometric compatibility alignment of panoramic vision and lidar, making up for the limitations of a single sensor, efficiently aggregating hierarchical 3D features of panoramic images and point cloud data, capturing a complete representation of distorted objects in real 3D space, and achieving cross-modal alignment of geometric perception, thereby realizing real-time, high-precision semantic mapping of robots in dynamic environments, solving the perception bottleneck of robots in complex environments, converting physical limitations into perception advantages, and significantly improving the perception ability of robots in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A flowchart of a semantic map construction method provided in an embodiment of the present application; Figure 2 A flowchart for determining a reference point set of point cloud data in a camera coordinate system of a panoramic image in the semantic map construction method provided in an embodiment of the present application; Figure 3 A flowchart for obtaining a target feature map in the semantic map construction method provided in an embodiment of the present application; Figure 4 A flowchart for generating a sampling index set in the semantic map construction method provided in an embodiment of the present application; Figure 5 Another flowchart for obtaining a target feature map in the semantic map construction method provided in an embodiment of the present application; Figure 6 A flowchart for generating sampling index sets at multiple scales in the semantic map construction method provided in an embodiment of the present application; Figure 7 A flowchart for obtaining a target feature map by aggregating in the semantic map construction method provided in an embodiment of the present application; Figure 8 A schematic diagram of the structure of a decoder in the semantic map construction method provided in an embodiment of the present application; Figure 9 A flowchart of generating a semantic map in the semantic map construction method provided in an embodiment of the present application; Figure 10 A schematic diagram of the structure of the control system of the robot provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of illustration and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can be implemented out of sequence, and steps without logical context can be reversed or implemented simultaneously. In addition, those skilled in the art, under the guidance of the contents of this application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0020] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0021] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0022] The existing robot environment perception process can achieve environmental perception by stitching together the image features and point cloud features of the environment captured by panoramic cameras and lidar. Specifically, the panoramic camera captures a panoramic image of the environment, and the lidar obtains point cloud data of the surrounding environment. After feature extraction, the extracted point cloud features are projected into the coordinate system of the panoramic image. The point cloud features are then aligned with the image features, and the image and point cloud features are stitched together along the channel dimension. This integrates the feature information of the two different modalities to achieve environmental perception.

[0023] However, this approach cannot handle dynamic occlusions caused by the robot's biomimetic structure (such as joint motion), resulting in poor performance in dynamic environments. Furthermore, while panoramic images provide a 360° field of view, they suffer from feature degradation under spherical projection (such as linear structure breakage, scale inconsistency, and depth perception distortion), ignoring the geometric mismatch between modalities.

[0024] In summary, the existing robot environment perception has serious self-occlusion and limited field of view (FOV) problems.

[0025] To address the above-mentioned issues, embodiments of the present application propose a semantic map construction method. This method obtains panoramic images and point cloud data, inputs the panoramic images into a pre-trained panoramic image encoder for multi-scale encoding, generates panoramic feature maps at multiple scales, and determines a reference point set for the point cloud data in the camera coordinate system of the panoramic image. Cross-modal feature fusion processing is then performed based on the pre-trained prior semantic network, the reference point set, and the panoramic feature maps at multiple scales to obtain a target feature map. A semantic map is then generated based on the target feature map and the pre-trained decoder. This method utilizes the ray characteristics of the panoramic camera and guides feature sampling offsets through geometric constraints, reducing spherical projection distortion. Furthermore, the method integrates the advantages of the panoramic camera and laser point cloud to achieve geometric compatibility alignment between panoramic vision and lidar, overcoming the limitations of a single sensor. The method efficiently aggregates hierarchical 3D features of panoramic images and point cloud data, captures a complete representation of distorted objects in real 3D space, and achieves cross-modal alignment of geometric perception, thereby enabling real-time, high-precision semantic mapping for robots in dynamic environments. This addresses the perception bottleneck of robots in complex environments, transforms physical limitations into perception advantages, and significantly improves the perception capabilities of robots in complex environments.

[0026] First, the relevant background involved in the semantic map construction method provided in the embodiment of the present application is explained.

[0027] It should be understood that the relevant background involved in the semantic map construction method provided in the embodiment of the present application can be deployed in any device that needs to perform dynamic environment perception, such as: a robot, specifically, a humanoid robot or a humanoid-like robot.

[0028] Illustratively, by executing the semantic map construction method provided in the embodiments of the present application, the robot can obtain the semantic map in real time, so that it can more intelligently understand the environment, interact with objects, and perform complex tasks based on the semantic map.

[0029] For example, the robot may be a humanoid robot. Specifically, a humanoid robot is a robot that mimics a human's appearance, movement, or behavior. It typically possesses a head, torso, arms, legs, and other human-like structures, and can perform human-like functions such as walking, grasping, and expressing emotions. Furthermore, each arm is composed of multiple joints, enabling it to perform different actions depending on the scenario.

[0030] Specifically, the robot may include a mechanical structure, a drive system, a sensing system, a control system, and a power supply system.

[0031] Exemplarily, the mechanical structure includes at least one robotic arm, an end effector corresponding to each robotic arm, two legs, and two feet, and the mechanical structure is used to perform specific actions.

[0032] Among them, the drive system includes: servo motors, etc., which are used to control the rotation or linear motion of each joint in the robotic arm and provide precise power output.

[0033] The sensing system includes a panoramic camera and a laser radar. Optionally, the sensing system may also include a torque sensor, a visual camera, an inertial measurement unit, and a tactile sensor. The sensing system is used to detect and obtain real-time environmental information to control the robot.

[0034] Optionally, a panoramic camera can be set on the head or top of the torso of the robot and ensure horizontal 360° and vertical wide range coverage (e.g., ±90°) to simulate human vision.

[0035] For example, the panoramic camera may have a resolution of 512×1024, covering a 360° horizontal field of view (horizontal FOV) and a 180° vertical field of view (vertical FOV).

[0036] Alternatively, the lidar can be fixed to the upper middle part of the robot's torso, tilted slightly downward (e.g., 10°-15°) to cover ground obstacles.

[0037] Among them, the control system includes: a processor, a memory, and a communication module, etc. The memory stores machine-readable instructions executable by the processor. When the robot's control system is running, the processor executes the machine-readable instructions to process and store the sensor data obtained by the sensing system, and executes the steps of the semantic map construction method provided in the embodiment of the present application to plan tasks for the robot, thereby controlling the operation of the robot.

[0038] Optionally, the robot may also include a human-computer interaction interface to achieve human-computer interaction.

[0039] The following describes in detail the semantic map construction method provided by the embodiments of the present application in combination with multiple embodiments.

[0040] Figure 1 A flowchart of the semantic map construction method provided in the embodiment of the present application, referring to Figure 1 As shown, the execution subject of the method can be any electronic device with processing capabilities, such as the control system of the above-mentioned robot, and the method includes: S101: Obtain panoramic images and point cloud data.

[0041] Alternatively, a panoramic image can be acquired in real time by a panoramic camera, and point cloud data can be acquired in real time by a laser radar, wherein the point cloud data includes multiple point clouds.

[0042] Exemplarily, a panoramic image of the current frame and point cloud data of the current frame may be acquired to construct a semantic map of the current frame.

[0043] S102: Input the panoramic image into a pre-trained panoramic image encoder for multi-scale encoding processing to generate panoramic feature maps at multiple scales.

[0044] Optionally, the panoramic image may be input into a pre-trained panoramic image encoder, and encoding processing at multiple scales may be performed on the panoramic image to generate panoramic feature maps at multiple scales.

[0045] Optionally, the panoramic image can be input into a pre-trained panoramic image encoder, spherical projection distortion can be processed by spherical convolution, and multi-scale encoding processing can be performed to generate panoramic feature maps at multiple scales.

[0046] For example, a panoramic image can be fed into a pre-trained panoramic image encoder, which preprocesses the image. This includes normalizing pixel values ​​and injecting spherical coordinates (θ, ϕ) as an additional channel to enhance geometric perception. Specifically, injecting spherical coordinates involves calculating the spherical coordinates (θ, ϕ) for each pixel and concatenating them into a five-channel input, thereby explicitly encoding spherical geometry.

[0047] Exemplarily, the panoramic image encoder performs hierarchical feature extraction on the preprocessed image features to generate panoramic feature maps at multiple scales. Specifically, the preprocessed image features are first subjected to convolution, batch normalization, 4x8 local window attention, and relative position encoding in sequence to generate high-resolution shallow features, which are then downsampled by 3x3 convolution to halve the size to 125x256, and processed by 8x16 local window attention to generate medium-resolution features. The convolution downsampling is repeated to reduce the size to 64x128 and 32x64 respectively, and global attention processing is performed to generate low-resolution deep features. Specifically, the global attention processing includes: using spherical adaptive attention at a high level, for example: weighting the spherical distance in the Q, K, and V calculations to preserve spherical distortion.

[0048] For example, the obtained high-resolution shallow features, medium-resolution features and low-resolution deep features can also be horizontally connected to obtain an output feature pyramid, where the resolution of each layer corresponds to 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original input respectively.

[0049] S103: Determine a reference point set of the point cloud data in the camera coordinate system of the panoramic image.

[0050] Optionally, the point cloud data can be converted to a camera coordinate system using a pre-calibrated extrinsic parameter matrix. Reference points for the point cloud data in the camera coordinate system of the panoramic image are generated, and uniform sampling is performed to obtain a set of reference points for the point cloud data in the camera coordinate system of the panoramic image. The pre-calibrated extrinsic parameter matrix is ​​a rigid body transformation matrix that transforms the point cloud in the point cloud coordinate system to the panoramic camera coordinate system, and is used to represent the geometric alignment relationship between the point cloud coordinate system and the panoramic camera coordinate system.

[0051] For example, the point cloud data can be converted into a camera coordinate system through a pre-calibrated extrinsic parameter matrix to generate reference points of the point cloud data in the camera coordinate system of the panoramic image, and the noise points can be filtered to obtain a set of reference points of the point cloud data in the camera coordinate system of the panoramic image.

[0052] S104: Perform cross-modal feature fusion processing based on the pre-trained prior semantic network, the reference point set, and the panoramic feature maps at multiple scales to obtain a target feature map.

[0053] It can be understood that the pre-trained prior semantic network is optimized through the training process and can encode the typical distribution of objects in different locations. Moreover, each grid in the prior semantic network corresponds to a sub-range in each space, forming a spatially structured topologically ordered representation.

[0054] Optionally, after obtaining a reference point set and panoramic feature maps at multiple scales, geometrically aware cross-modal feature alignment and fusion processing can be performed based on the pre-trained prior semantic network, the reference point set and panoramic feature maps at multiple scales to obtain a target feature map.

[0055] This means that in the process of generating the target feature map, the initial semantic hypothesis can be provided by the pre-trained prior semantic network, and geometric correction and feature fusion can be performed in combination with the reference point set and the panoramic feature maps at multiple scales, so as to extract visual evidence from the panoramic image and the reference point set, verify or correct the initial semantic hypothesis, and obtain the target feature map. The target feature map can be understood as a weighted combination of prior knowledge and real-time sensor evidence.

[0056] For example, geometric constraints can be generated through panoramic feature maps at multiple scales, and cross-modal perception can be performed based on the geometric constraints, the pre-trained prior semantic network, and the reference point set, so as to achieve geometric alignment of the panoramic image and the point cloud data, and aggregate the hierarchical 3D features in the panoramic image and the point cloud data to obtain the target feature map.

[0057] Among them, the target feature map is a feature map under the BEV space, including geometric information and semantic information. The prior semantic network corresponds to the target space range, which is determined according to the field of view of the panoramic camera and lidar. The prior semantic network includes multiple grids, each grid corresponds to a spatial sub-range, and each grid has semantic information. The semantic information is used to represent the focus object of the grid.

[0058] For example, the prior semantic network can be a BEV query space, which can be obtained by a learnable query matrix Characterize, where the parameter matrix includes multiple query vectors The prior semantic network is essentially a set of dynamic query vectors used to generate BEV semantic map features. Each query vector in the parameter matrix corresponds to a preset location in the semantic map, and each query vector is used to determine the features required for that location from the panoramic image and point cloud data.

[0059] For example, the prior semantic network is learnable and can be obtained through training. During training, the learnable parameter matrix is ​​defined as the model parameters, randomly initialized, and then automatically optimized through training. Simultaneously, the 2D coordinate information of the corresponding grid is injected into each query for position encoding enhancement and grid mapping, thereby achieving an association between the prior semantic network, the grid, and the target spatial range.

[0060] For example, the target spatial range is determined based on the field of view of the panoramic camera and the lidar, and can be understood as the map range of the final semantic map. The spatial sub-range is a sub-range in the final semantic map.

[0061] S105: Generate a semantic map based on the target feature map and the pre-trained decoder.

[0062] Optionally, the decoder can be a BEV decoder. After obtaining the target feature map, the target feature map can be decoded and upsampled according to the pre-trained decoder, thereby gradually converting the low-resolution BEV features into a high-resolution semantic segmentation map to obtain a semantic map.

[0063] The semantic map can be thought of as a lightweight, semantically-focused bird's-eye view. It includes both geometric and semantic information, providing a semantic representation of the environment. Geometric information includes the location of obstacles, while semantic information includes object categories. The semantic map includes the BEV's spatial perception range (X, Y) and the predicted number of categories.

[0064] In this embodiment, panoramic images and point cloud data are acquired and input into a pre-trained panoramic image encoder for multi-scale encoding processing to generate panoramic feature maps at multiple scales. A reference point set of the point cloud data in the camera coordinate system of the panoramic image is determined. Thus, cross-modal feature fusion processing can be performed based on the pre-trained prior semantic network, the reference point set, and the panoramic feature maps at multiple scales to obtain a target feature map. A semantic map is then generated based on the target feature map and the pre-trained decoder. The ray characteristics of the panoramic camera are utilized to guide feature sampling offset through geometric constraints, reduce spherical projection distortion, and integrate the advantages of the panoramic camera and laser point cloud to achieve geometric compatibility alignment between panoramic vision and lidar, thereby overcoming the limitations of a single sensor. The hierarchical 3D features of the panoramic image and point cloud data are efficiently aggregated to capture a complete representation of distorted objects in real 3D space. Cross-modal alignment of geometric perception is achieved, thereby enabling real-time, high-precision semantic mapping for robots in dynamic environments. This solves the perception bottleneck of robots in complex environments, transforms physical limitations into perception advantages, and significantly improves the perception capabilities of robots in complex environments.

[0065] In one possible implementation, Figure 2 A flowchart for determining a reference point set of point cloud data in a camera coordinate system of a panoramic image in a semantic map construction method provided in an embodiment of the present application, referring to Figure 2 As shown, the determination of the reference point set of the point cloud data in the camera coordinate system of the panoramic image in the above S103 includes: S201 , mapping the point cloud data to the camera coordinate system to obtain the target point corresponding to each point cloud in the camera coordinate system.

[0066] Optionally, the point cloud data is converted to the camera coordinate system using a pre-calibrated extrinsic matrix to generate target points in the point cloud data in the camera coordinate system of the panoramic image, thereby removing points that are outside the panoramic camera's field of view. The pre-calibrated extrinsic matrix is ​​a rigid body transformation matrix that transforms the point cloud in the point cloud coordinate system to the panoramic camera coordinate system, and is used to represent the geometric alignment relationship between the point cloud coordinate system and the panoramic camera coordinate system.

[0067] S202: Project the target points corresponding to each point cloud in the camera coordinate system onto the depth image plane to generate a depth map.

[0068] The depth image plane is a two-dimensional grid aligned with the pixels of the panoramic image, and each grid point stores the depth value of the corresponding 3D space point.

[0069] Optionally, the pitch angle θ and azimuth angle ϕ of the target point corresponding to each point cloud in the camera coordinate system can be calculated, and the pitch angle θ and azimuth angle ϕ can be converted into pixel coordinates (u, v), and the depth value of each point can be calculated to generate a depth map.

[0070] For example, taking the coordinates of the target point corresponding to the point cloud in the camera coordinate system as (x, y, z) as an example, the pitch angle θ and the azimuth angle ϕ can be calculated according to the following formula:

[0071]

[0072] For example, and with reference to the following formula, the pitch angle θ and the azimuth angle ϕ are converted into pixel coordinates:

[0073] Wherein, W is the width of the panoramic image, and H is the length of the panoramic image.

[0074] For example, the depth value of the point can be obtained .

[0075] S203 : Back-project the depth map to obtain a plurality of reference points, and use the set of the plurality of reference points as a reference point set.

[0076] Optionally, each point in the depth map is back-projected (referred to as Back-Projection) into 3D space to obtain multiple reference points, and a set of the multiple reference points is used as a reference point set, thereby associating the depth information with the 3D space structure.

[0077] By back-projecting the reference point set, the 2D perception results can be mapped back to 3D space through the inverse transformation of the imaging geometry, thereby compensating for the sparsity of the point cloud data, generating a more complete scene geometry, and bridging visual features with depth information to achieve modal alignment. In addition, it can also be applied to any camera model.

[0078] In one possible implementation, Figure 3 A flow chart of obtaining a target feature map in the semantic map construction method provided in the embodiment of the present application, referring to Figure 3 As shown, in the above S104, cross-modal feature fusion processing is performed based on the pre-trained prior semantic network, the reference point set, and the panoramic feature maps at multiple scales to obtain a target feature map, including: S301 , performing semantic geometry dynamic binding according to a priori semantic network and a reference point set to generate a sampling index set.

[0079] Optionally, the reference point set can be accurately mapped to the panoramic image coordinate system through spherical projection and a priori semantic network to establish geometric alignment and generate a sampling index set, thereby eliminating the projection distortion of traditional stitching, reducing category confusion caused by depth mismatching, and preserving the geometric consistency of the 360° field of view.

[0080] Among them, the sampling index set includes multiple sampling indexes , each sampling index corresponds to a reference point, and each sampling index is used to represent the corresponding position of the corresponding reference point in the panoramic feature map.

[0081] The semantic prior is provided by the prior semantic network, and combined with the collective position of the reference points, it is possible to determine where to extract features from the panoramic image and how to fill them into the semantic map.

[0082] S302 : According to the sampling index set, adaptive feature sampling and aggregation processing are performed on the panoramic feature maps at multiple scales to obtain a target feature map.

[0083] Optionally, adaptive feature sampling extraction and aggregation processing can be performed from the panoramic feature maps at multiple scales according to the sampling index set, and the corresponding positions of the prior semantic network can be filled in to obtain the target feature map, so that the feature sampling position can be dynamically adjusted and occlusion and sparsity can be adaptively compensated (such as bypassing arm occlusion and completing the features of the blocked object).

[0084] In one possible implementation, Figure 4 A flowchart of generating a sampling index set in the semantic map construction method provided in the embodiment of the present application, referring to Figure 4 As shown, the above S301 performs semantic geometry dynamic binding according to the prior semantic network and the reference point set to generate a sampling index set, including: S401: Convert each reference point in the reference point set to obtain a spherical angle corresponding to each reference point.

[0085] Optionally, each reference point in the reference point set can be transformed by spherical projection to calculate the spherical angle corresponding to each reference point ( , thereby establishing the projection relationship between each reference point and the panoramic image. Among them, the spherical angle includes the azimuth and pitch angle .

[0086] S402 : The linear layer obtained through pre-training offsets the spherical angle corresponding to each reference point based on the prior semantic network to obtain the corrected spherical coordinates corresponding to each reference point.

[0087] Optionally, a pre-trained linear layer f Δ sThe spherical angle corresponding to each reference point is predicted to get the offset , and generate the corrected spherical coordinates S' corresponding to each reference point, so that the feature fusion of geometric perception can be achieved through the corrected spherical coordinates S' to correct the spherical distortion of the panoramic image.

[0088] For example, the following formula may be used for offset prediction:

[0089] in, is the query matrix corresponding to the prior semantic network, is the pre-trained linear layer, PosEnc(•) is the position encoding function, and S is the initial spherical angle coordinate.

[0090] Exemplarily, the corrected spherical coordinates S' corresponding to each reference point are generated with reference to the following formula:

[0091] S403 : Calculate the sampling index corresponding to each reference point according to the corrected spherical coordinates corresponding to each reference point.

[0092] Optionally, the corrected spherical coordinates S' corresponding to each reference point are mapped to the width range W and height range H of the panoramic image, and the sampling index corresponding to each reference point is calculated. Among them, the sampling index corresponding to each reference point points to the position in the panoramic feature map that is precisely aligned with each reference point.

[0093] For example, the sampling index corresponding to each reference point can be calculated by referring to the following formula: :

[0094]

[0095] Wherein, H is the height range of the panoramic image, and W is the width range of the panoramic image.

[0096] The sampling index corresponding to each reference point is calculated through the corrected spherical coordinates corresponding to each reference point, which can ensure the geometric alignment of the point cloud data and the panoramic image, avoid errors introduced by distortion, and reduce modal fusion errors.

[0097] In one possible implementation, Figure 5 Another flow chart for obtaining a target feature map in the semantic map construction method provided in the embodiment of the present application, referring to Figure 5 As shown, in the above S302, adaptive feature sampling and aggregation processing are performed on the panoramic feature maps at multiple scales according to the sampling index set to obtain the target feature map, including: S501: Generate sampling index sets at multiple scales according to the sampling index set.

[0098] It can be understood that the sampling index set is a sampling index set at a single scale. The sampling index set at a single scale can be used to generate sampling index sets at multiple scales according to the different scales of each panoramic feature map, thereby ensuring that the sampling index set can be mapped to the corresponding position on the feature maps of different resolutions.

[0099] Optionally, the sampling index set may be normalized according to the difference in scales to generate sampling index sets at multiple scales.

[0100] S502: Determine offset information corresponding to the panoramic feature map at each scale based on the prior semantic network.

[0101] Optionally, a deformable offset prediction can be performed on the panoramic feature maps at each scale based on the prior semantic network to obtain the offset information corresponding to the panoramic feature maps at each scale. The offset information corresponding to the panoramic feature maps at each scale is used to adapt to geometric deformations at different resolutions.

[0102] S503 : Sampling to obtain sampling features at each scale according to the sampling index sets at multiple scales and the offset information corresponding to the panoramic feature maps at each scale.

[0103] Optionally, sampling features at the same scale are obtained by sampling according to a sampling index set at the same scale and offset information corresponding to a panoramic feature map at the same scale.

[0104] Specifically, at the same scale, the offset information corresponding to each sampling index in the sampling index set and the panoramic feature map is superimposed, and the superimposed result is sampled by bilinear interpolation to obtain the sampling features at this scale.

[0105] S504: Aggregate the sampled features at each scale to obtain a target feature map.

[0106] Optionally, attention weight calculation and multi-scale feature aggregation can be performed based on the sampling features at each scale in combination with the prior semantic network to obtain the target feature map.

[0107] In one possible implementation, Figure 6 A flowchart of generating a sampling index set at multiple scales in the semantic map construction method provided in the embodiment of the present application, referring to Figure 6 As shown, in the above S501, sampling index sets at multiple scales are generated according to the sampling index set, including: S601: Acquire the height and width of the panoramic image.

[0108] Optionally, the height of the panoramic image can be obtained and width .

[0109] S602: Obtain the height and width of the panoramic feature map at multiple scales.

[0110] Optionally, the height and width of the panoramic feature map at multiple scales may be obtained.

[0111] S603: Determine row components of sampling indices corresponding to reference points at multiple scales according to the height of the panoramic image, the heights of the panoramic feature maps at multiple scales, and a sampling index set.

[0112] Optionally, the height of the panoramic image can be , the height of the panoramic feature map at multiple scales and the sampling index set, and calculate the row component of the sampling index corresponding to each reference point at multiple scales.

[0113] For example, the height of the panoramic image can be 、 Height of the panoramic feature map at this scale And the sampling index corresponding to each reference point , calculated The row component of the sampling index corresponding to each reference point under the scale.

[0114] Specifically, we can calculate Height of the panoramic feature map at this scale The row component of the sample index corresponding to each reference point and calculate the product of the product and the height of the panoramic image The ratio of The row component of the sampling index corresponding to each reference point under the scale.

[0115] S604: Determine the column components of the sampling index corresponding to each reference point at the multiple scales according to the width of the panoramic image, the width of the panoramic feature map at the multiple scales, and the sampling index set.

[0116] Optionally, depending on the width of the panoramic image , the width of the panoramic feature map at multiple scales and the sampling index set, and calculate the column components of the sampling index corresponding to each reference point at multiple scales.

[0117] For example, the width of the panoramic image can be 、 The width of the panoramic feature map at this scale And the sampling index corresponding to each reference point , calculated The column component of the sampling index corresponding to each reference point under the scale.

[0118] Specifically, we can calculate The width of the panoramic feature map at this scale Column components of the sample index corresponding to each reference point and calculate the product of the product and the width of the panoramic image The ratio of The column component of the sampling index corresponding to each reference point under the scale.

[0119] S605 : Generate a sampling index set at multiple scales according to the row components of the sampling index corresponding to each reference point at multiple scales and the column components of the sampling index corresponding to each reference point at multiple scales.

[0120] Optionally, the row components and column components of the sampling indexes corresponding to the reference points at the same scale are combined to generate sampling index sets at multiple scales.

[0121] In a possible implementation, determining the offset information corresponding to the panoramic feature map at each scale based on the prior semantic network in S502 includes: The prior semantic network is input into the pre-trained linear layer corresponding to the first scale to generate the offset information corresponding to the panoramic feature map at the first scale.

[0122] The first scale is any scale.

[0123] Optionally, the query matrix corresponding to the prior semantic network is predicted through a pre-trained linear layer corresponding to each scale to obtain 2D offset information at each scale, thereby achieving efficient prediction.

[0124] In one possible implementation, Figure 7 A flow chart of the method for constructing a semantic map provided in the embodiment of the present application when aggregating and obtaining a target feature map, referring to Figure 7 As shown, in the above S504, the target feature map is obtained by aggregating the sampling features at each scale, including: S701 , performing feature projection according to the sampling features at each scale and the prior semantic network to generate a projection matrix at each scale, a first projected feature at each scale, and a second projected feature at each scale.

[0125] Optionally, the pre-trained projection matrices Wq, Wk, and Wv can be obtained, and the sampling features at each scale and the prior semantic network can be projected into the query, key, and value spaces, respectively, to generate the projection matrices at each scale, the first projected features at each scale, and the second projected features at each scale.

[0126] Specifically, each scale has the same projection matrix Wq, and different projection matrices Wk and Wv.

[0127] For example, For example, Sampling characteristics at scale And the query matrix corresponding to the prior semantic network Projected into query, key, and value spaces respectively, generating Projection matrix at scale 、 First projection features at scale And the second projection feature Specifically, it can be implemented by referring to the following formula:

[0128]

[0129]

[0130] S702: Calculate the attention weight at each scale based on the projection matrix at each scale and the first projection features at each scale.

[0131] Optionally, Scale as an example, according to Projection matrix at scale as well as First projection features at scale , calculated Attention weights at scale , which can be calculated by referring to the following formula:

[0132] in, is a pre-obtained scaling factor used to prevent the gradient from disappearing due to excessive dot product values. l The importance of layer features to the current query vector.

[0133] S703: Aggregate the attention weights at each scale and the second projection features at each scale to obtain a target feature map.

[0134] Optionally, the second projection feature at each scale , according to the attention weights at each scale The weighted results are summed up at each scale to obtain the target feature map.

[0135] Feature projection is performed through the sampling features at each scale and the prior semantic network to generate the projection matrix at each scale, the first projection feature at each scale and the second projection feature at each scale. The attention weight at each scale is calculated through the projection matrix at each scale and the first projection feature at each scale. Then, according to the attention weight at each scale and the second projection feature at each scale, the target feature map is aggregated, so that the attention weight only needs to focus on the semantic relevance of the features without the need for explicit modeling geometry. At the same time, it can also enable features at different levels to compete for weights to avoid redundancy, thereby being able to adaptively select the optimal combination of multi-scale features.

[0136] In one possible implementation, Figure 8 A schematic diagram of the structure of the decoder in the semantic map construction method provided in the embodiment of the present application, referring to Figure 8 As shown in Figure 5, the decoder includes: a convolutional layer, multiple cascaded upsampling modules, and a classification head. Figure 9 A flowchart of generating a semantic map in the semantic map construction method provided in the embodiment of the present application, referring to Figure 9 As shown, in the above S105, a semantic map is generated based on the target feature map and the pre-trained decoder, including: S901. Perform grid restoration on the target feature map to generate a spatial grid feature map.

[0137] Optionally, the target feature map can be grid-restored to reshape the target feature map into a 3D grid form, restore the spatial structure, and generate a spatial grid feature map, wherein the spatial grid feature map is a two-dimensional spatial grid structure.

[0138] For example, tensor reshaping can be used to map each BEV query in the target feature map to a 2D spatial position according to a preset grid order to generate a spatial grid feature map.

[0139] For example, the 2D coordinate encoding of the BEV mesh can also be injected after reshaping to explicitly preserve the spatial location information of the query.

[0140] S902: Input the spatial grid feature map into the convolution layer for aggregation processing to generate aggregated features.

[0141] Optionally, the spatial grid feature map is input into the convolutional layer for global context aggregation processing to generate aggregated features.

[0142] S903: Input the aggregated features into multiple cascaded upsampling modules for step-by-step upsampling processing to generate target size features.

[0143] Optionally, the aggregated features are gradually upsampled through multiple cascaded upsampling modules to gradually improve the resolution.

[0144] For example, the aggregated features are first enlarged in size by transposed convolution and repeatedly upsampled. Convolution and nonlinear activation are performed after each upsampling stage to enhance local consistency until the resolution reaches the target size, thereby obtaining the target size features.

[0145] S904: Generate a semantic map based on the target size features and the classification head.

[0146] Optionally, the target size features are processed for category prediction through a convolutional layer, and the number of categories is activated through a classification head to generate a pixel-by-pixel category probability map. The category with the highest probability is selected as the semantic label of the grid through hard discrimination processing, thereby generating a semantic map.

[0147] Optionally, the semantic labels of each grid can be optimized through conditional random fields and morphological operations to convert them into a semantic map that can be understood by the robot.

[0148] By gridding and restoring the target feature map, a spatial grid feature map is generated, and the spatial grid feature map is input into the convolution layer for aggregation processing to generate aggregated features. The aggregated features are then input into multiple cascaded upsampling modules for step-by-step upsampling processing to generate target size features. Based on the target size features and the classification head, a semantic map is generated, which can achieve accurate mapping of panoramic features to semantic maps, and combines geometric perception to reduce redundant calculations. In addition, it also improves computational efficiency and ensures the real-time performance of the semantic map construction process.

[0149] It is understood that the query matrix corresponding to the prior semantic network in the semantic map construction method provided in the embodiment of the present application can be obtained by annotating data during the training phase. , Transformer parameters and linear layers in the panoramic image encoder f Δ s Optimize the parameters of the learnable components.

[0150] Optionally, during the training phase, raw data such as panoramic images, point cloud data, and semantic labels of semantic networks can be obtained, and after synchronous data enhancement, the parameters of the learnable components can be trained through forward propagation, loss calculation, and back propagation.

[0151] Specifically, synchronous data enhancement includes: horizontal flipping, random rotation, and MixUp mixing.

[0152] Exemplarily, the horizontal flipping includes: horizontally flipping the panoramic image, inverting the x-coordinate of the point cloud data, and synchronously flipping the semantic labels of the semantic network, thereby enhancing the robustness in bilaterally symmetrical scenes.

[0153] Exemplarily, the random rotation includes: rotating the panoramic image according to a preset spherical rotation angle, rotating the point cloud data around the Z axis by a first preset angle, and synchronously rotating the semantic labels of the semantic network by a second preset angle.

[0154] Exemplarily, MixUp includes linearly mixing any two panoramic images with semantic labels of a semantic network.

[0155] The present application also provides a robot control system. Figure 10 As shown, Figure 10 This is a schematic diagram of the structure of a robot control system provided in an embodiment of the present application, comprising: a processor 1001, a memory 1002, and optionally, a bus 1003. Memory 1002 stores machine-readable instructions executable by processor 1001. When the robot control system is running, processor 1001 communicates with memory 1002 via bus 1003. When the machine-readable instructions are executed by processor 1001, the steps of the semantic map construction method described above are performed.

[0156] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0157] In addition, the functional units in the various embodiments of the present application can be integrated into a single processing unit, each unit can exist physically separately, or two or more units can be integrated into a single unit. If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0158] The above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the protection scope of the present application.

Claims

1. A semantic map construction method, characterized in that: include: Acquire a panoramic image and point cloud data, wherein the point cloud data includes a plurality of point clouds; Inputting the panoramic image into a pre-trained panoramic image encoder for multi-scale encoding processing to generate panoramic feature maps at multiple scales; Determining a reference point set of the point cloud data in a camera coordinate system of the panoramic image; A cross-modal feature fusion process is performed based on the pre-trained prior semantic network, the reference point set, and the panoramic feature maps at multiple scales to obtain a target feature map, where the target feature map is a feature map in the BEV space, including geometric information and semantic information. The prior semantic network corresponds to the target space range, which is determined according to the field of view of the panoramic camera and the lidar. The prior semantic network includes multiple grids, each of which corresponds to a spatial sub-range, and each grid has semantic information, and the semantic information is used to characterize the object of interest of the grid; A semantic map is generated based on the target feature map and a pre-trained decoder.

2. The semantic map construction method according to claim 1, characterized in that: Determining a reference point set of the point cloud data in a camera coordinate system of the panoramic image includes: Mapping the point cloud data to the camera coordinate system to obtain the target point corresponding to each point cloud in the camera coordinate system; Project the target points corresponding to each point cloud in the camera coordinate system onto the depth image plane to generate a depth map; Back-projecting is performed on the depth map to obtain a plurality of reference points, and a set consisting of the plurality of reference points is used as a reference point set.

3. The semantic map construction method according to claim 1, characterized in that: The cross-modal feature fusion processing is performed based on the pre-trained prior semantic network, the reference point set, and the panoramic feature maps at multiple scales to obtain a target feature map, including: Performing dynamic semantic geometry binding based on the prior semantic network and the reference point set to generate a sampling index set, wherein the sampling index set includes multiple sampling indices, each sampling index corresponds to a reference point, and each sampling index is used to represent the corresponding position of the corresponding reference point in the panoramic feature map; According to the sampling index set, adaptive feature sampling and aggregation processing are performed on the panoramic feature maps at multiple scales to obtain a target feature map.

4. The semantic map construction method according to claim 3, characterized in that: The performing semantic geometry dynamic binding according to the prior semantic network and the reference point set to generate a sampling index set includes: Convert each reference point in the reference point set to obtain a spherical angle corresponding to each reference point, where the spherical angle includes an azimuth angle and a pitch angle; The linear layer obtained through pre-training offsets the spherical angle corresponding to each reference point based on the prior semantic network to obtain the corrected spherical coordinates corresponding to each reference point; According to the corrected spherical coordinates corresponding to each reference point, the sampling index corresponding to each reference point is calculated.

5. The semantic map construction method according to claim 3, characterized in that: The method of performing adaptive feature sampling and aggregation processing on the panoramic feature maps at multiple scales according to the sampling index set to obtain a target feature map includes: Generating sampling index sets at multiple scales according to the sampling index set; Determining offset information corresponding to the panoramic feature map at each scale based on the prior semantic network; According to the sampling index sets at multiple scales and the offset information corresponding to the panoramic feature maps at each scale, sampling is performed to obtain the sampling features at each scale; According to the sampling features at each scale, the target feature map is aggregated.

6. The method for constructing a semantic map according to claim 5, wherein: Generating sampling index sets at multiple scales according to the sampling index set includes: Obtaining the height and width of the panoramic image; Get the height and width of the panoramic feature map at multiple scales; Determining, according to the height of the panoramic image, the heights of the panoramic feature maps at the multiple scales, and the sampling index set, a row component of the sampling index corresponding to each reference point at the multiple scales; Determining, according to the width of the panoramic image, the widths of the panoramic feature maps at the multiple scales, and the sampling index set, a column component of the sampling index corresponding to each reference point at the multiple scales; A sampling index set at multiple scales is generated according to the row components of the sampling index corresponding to each reference point at multiple scales and the column components of the sampling index corresponding to each reference point at multiple scales.

7. The semantic map construction method according to claim 5, characterized in that: Determining offset information corresponding to the panoramic feature map at each scale according to the prior semantic network includes: The prior semantic network is input into a pre-trained linear layer corresponding to a first scale to generate offset information corresponding to a panoramic feature map at the first scale, where the first scale is any scale.

8. The method for constructing a semantic map according to claim 5, wherein: The target feature map is obtained by aggregating the sampling features at each scale, including: Perform feature projection according to the sampling features at each scale and the prior semantic network to generate a projection matrix at each scale, a first projected feature at each scale, and a second projected feature at each scale; Calculate the attention weight at each scale according to the projection matrix at each scale and the first projection feature at each scale; According to the attention weights at each scale and the second projection features at each scale, the target feature map is aggregated.

9. The semantic map construction method according to claim 1, characterized in that: The decoder includes: a convolutional layer, a plurality of cascaded upsampling modules and a classification head; the semantic map is generated according to the target feature map and the pre-trained decoder, including: Performing grid restoration on the target feature map to generate a spatial grid feature map; Inputting the spatial grid feature map into the convolutional layer for aggregation processing to generate aggregated features; Inputting the aggregated features into the cascaded multiple upsampling modules for step-by-step upsampling processing to generate target size features; A semantic map is generated according to the object size feature and the classification head.

10. A robot control system, characterized in that: It includes a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the control system of the robot is running, the processor executes the machine-readable instructions to perform the steps of the semantic map construction method as described in any one of claims 1 to 9.

Images