Map construction method, robot scheduling method, device, and storage medium

Abstract

The application provides a map construction method, a robot scheduling method and device, and a storage medium, relates to the fields of three-dimensional mapping technology and intelligent warehousing technology, and comprises the following steps: determining a global pose sequence of a robot in a mapping data collection process according to mapping data of a warehouse space collected by the robot; determining a three-dimensional model of the warehouse space based on the global pose sequence and the mapping data; determining semantic information of each pixel in image data; and mapping the semantic information of each pixel to the three-dimensional model of the warehouse space to obtain a three-dimensional semantic map of the warehouse space. The three-dimensional model of the warehouse space can be constructed in a mapping stage, the semantic information of the pixels in the image data is mapped to the three-dimensional model, and the three-dimensional semantic map of the warehouse space is constructed, which can not only provide accurate three-dimensional spatial scales, support obstacle perception and path planning of the robot in a height direction, but also enhance the distinguishing ability of the robot for a high-repetition scene by means of the semantic information, and improve positioning accuracy.

Description

Technical Field

[0001] This application relates to the fields of 3D mapping technology and intelligent warehousing technology, and in particular to a map building method, robot scheduling method, device and storage medium. Background Technology

[0002] In smart warehousing scenarios, robots are often used to achieve unmanned operation and efficient scheduling of the entire process of goods storage, handling, and sorting. The robot's positioning and navigation accuracy, environmental perception adaptability, and global collaborative efficiency are key factors in warehousing efficiency and intelligence, and these key factors are all closely related to map accuracy. Therefore, how to build high-precision maps to meet the growing positioning and navigation needs of smart warehousing scenarios and improve the operational efficiency and accuracy of robots in smart warehousing has become a direction that needs to be studied. Summary of the Invention

[0003] This application provides a map building method, a robot scheduling method, an apparatus, and a storage medium to improve the accuracy and precision of map building, thereby enhancing the positioning and navigation accuracy of the robot.

[0004] In a first aspect, embodiments of this application provide a map construction method, including:

[0005] Based on the mapping data of the warehouse space collected by the robot, the global pose sequence of the robot during the mapping data acquisition process is determined; wherein, the mapping data includes at least the image data collected by the robot through the detection module;

[0006] A 3D model of the warehouse space is determined based on the global pose sequence and mapping data;

[0007] Determine the semantic information of each pixel in the image data;

[0008] The semantic information of each pixel is mapped to the three-dimensional model of the warehouse space to obtain a three-dimensional semantic map of the warehouse space.

[0009] In some embodiments, determining the global pose sequence of the robot during the mapping data acquisition process based on the mapping data of the warehouse space collected by the robot includes: determining the initial pose sequence of the robot during the mapping data acquisition process based on image data; extracting feature information of at least some image frames in the image data; performing loop closure detection in the image data based on the feature information to obtain a loop closure frame; and optimizing the initial pose sequence based on the loop closure frame to obtain the global pose sequence.

[0010] In some embodiments, the mapping data also includes other modal data collected by the robot through the detection module; based on the mapping data of the warehouse space collected by the robot, the global pose sequence of the robot during the mapping data acquisition process is determined, including: establishing the correlation between image data and other modal data to obtain multi-frame fusion data; and determining the global pose sequence based on the specified frame fusion data in the multi-frame fusion data.

[0011] In some embodiments, determining a global pose sequence based on specified frame fusion data in multi-frame fusion data includes: determining an initial pose sequence of specified frame fusion data according to first modal data and second modal data in specified frame fusion data; determining a closed-loop frame in specified frame fusion data using image data and third modal data in specified frame fusion data; constructing a pose factor map according to specified frame fusion data, the initial pose sequence of specified frame fusion data and the closed-loop frame; and optimizing the initial pose sequence of specified frame fusion data based on the pose factor map to obtain a global pose sequence.

[0012] In some embodiments, the first modal data is inertial measurement data; and / or, the second modal data is wheel speed data; and / or, the third modal data is point cloud data.

[0013] In some embodiments, determining a 3D model of a storage space based on a global pose sequence and mapping data includes: constructing a 3D model boundary of the storage space based on the global pose sequence; initializing a Gaussian distribution in the 3D model boundary to obtain an initial 3D model; for any Gaussian distribution in the initial 3D model, projecting the Gaussian distribution onto the pixel surface of the corresponding image frame to obtain the corresponding projection region, determining the predicted color of the projection region and the pixel loss of the predicted color compared to the actual pixel color in the corresponding image frame; determining the total loss function based on the mean pixel loss of the image frame, iteratively updating the position parameters, shape parameters, and color parameters of the Gaussian distribution in the initial 3D model through a backpropagation algorithm, and using the obtained optimized Gaussian distribution set as the 3D model of the storage space.

[0014] In some embodiments, mapping the semantic information of each pixel to a three-dimensional model of the warehouse space to obtain a three-dimensional semantic map of the warehouse space includes: back-projecting the Gaussian distribution in the three-dimensional model to the corresponding image frame in the image data to determine the pixel region covered by each Gaussian distribution; determining the semantic attributes of each Gaussian distribution based on the semantic information of each pixel in the pixel region covered by each Gaussian distribution to obtain a set of Gaussian distributions with semantic attributes; and generating a three-dimensional semantic map of the warehouse space based on the set of Gaussian distributions with semantic attributes.

[0015] Secondly, embodiments of this application provide a robot scheduling method, including:

[0016] Based on the robot's task to be performed and the 3D semantic map of the storage space, the robot's task topology path is determined;

[0017] Extract node matching data corresponding to at least one key node in the task topology path from the 3D semantic map;

[0018] Acquire environmental images at key nodes as the robot travels along the task topology path;

[0019] The robot is scheduled based on the environmental image and the node matching data corresponding to the key nodes.

[0020] In some embodiments, the three-dimensional semantic map is obtained by mapping the semantic information of each pixel in the image data collected by the robot during its movement in the warehouse space to a three-dimensional model of the warehouse space; the three-dimensional model of the warehouse space is calculated based on the global pose sequence and mapping data of the robot during its movement in the warehouse space, and the mapping data includes at least the image data collected by the detection module when the robot is moving in the warehouse space.

[0021] In some embodiments, determining the robot's task topology path based on the robot's task to be performed and a three-dimensional semantic map of the storage space includes: determining the robot's traversable area in the three-dimensional semantic map; and determining the task topology path in the traversable area based on a path planning algorithm, starting from the robot's current position and ending at the task target area of ​​the task to be performed.

[0022] In some embodiments, the node matching data includes the semantic features of at least one semantic marker of the location of a key node. The robot is scheduled based on the environmental image and the node matching data corresponding to the key node, including: extracting a target pixel region from the environmental image that is consistent with the semantic category of the semantic marker in the node matching data; determining the robot's deviation from the task topology path based on the target pixel region and the node matching data; and scheduling the robot's movement based on the deviation.

[0023] In some embodiments, the robot's deviation from the task topology path is determined based on the target pixel region and node matching data, and the robot's movement is scheduled according to the deviation. This includes: determining the matching degree between the target pixel region and semantic markers based on the node matching data; if the matching degree is greater than or equal to a preset threshold, determining that the robot has not deviated from the task topology path and controlling the robot to continue moving along the task topology path; if the matching degree is less than the preset threshold, determining that the robot has deviated from the task topology path, determining the robot's actual pose in the warehouse space based on the environmental image and the three-dimensional semantic map of the warehouse space, and replanning the task topology path for the robot based on the actual pose.

[0024] In some embodiments, the node matching data further includes 3DGS information of at least one semantic marker at the location of the key node. Controlling the robot to continue traveling along the task topology path further includes: based on the 3DGS information of the semantic marker and the robot's current pose at the key node, projecting the semantic marker onto the plane where the target pixel region is located to obtain the ideal projection feature corresponding to the semantic marker; calculating the projection error between the ideal projection feature and the actual image feature of the target pixel region; iteratively optimizing the pose parameters using the projection error as a loss function to determine the corrected pose of the robot at the key node; sending the corrected pose to the robot, controlling the robot to update its current pose using the corrected pose, and continuing to travel based on the updated pose and the task topology path.

[0025] In some embodiments, the method further includes: in response to receiving a relocalization request from the robot, acquiring a current environment image collected by the robot; determining at least one candidate region matching the current environment image based on a 3D semantic map and the current environment image; determining at least one candidate pose of the robot based on the 3D semantic map and the at least one candidate region; and determining the robot's relocalization pose among the at least one candidate pose.

[0026] In some embodiments, determining at least one candidate region matching the current environment image based on a three-dimensional semantic map and a current environment image includes: extracting features from the current environment image to obtain a local feature set of the current environment image; determining at least one mapping image frame matching the local feature set based on the local feature set and the mapping image frame feature set of the three-dimensional semantic map; and determining at least one candidate region matching the current environment image based on the at least one mapping image frame.

[0027] In some embodiments, determining at least one candidate pose of the robot based on a 3D semantic map and at least one candidate region includes: extracting the 3DGS model and mapping pose corresponding to each candidate region based on the 3D semantic map; generating a virtual rendering image corresponding to each candidate region based on the 3DGS model and mapping pose corresponding to each candidate region; and optimizing the mapping pose of each candidate region based on the difference between the virtual rendering image corresponding to each candidate region and the current environment image to obtain at least one candidate pose.

[0028] In some embodiments, based on the difference between the virtual rendered image corresponding to each candidate region and the current environment image, the mapping pose of each candidate region is optimized to obtain at least one candidate pose, including: for any candidate region, using the photometric loss and structural similarity loss between the virtual rendered image and the current environment image as image difference loss to construct a loss function, and iteratively optimizing the pose parameters of the mapping pose through backpropagation algorithm to obtain a candidate pose; determining the robot's relocalization pose from at least one candidate pose, including: determining the image difference loss corresponding to each candidate pose; and determining the candidate pose with the smallest corresponding image difference loss as the relocalization pose.

[0029] Thirdly, embodiments of this application provide a map building apparatus, including:

[0030] The first determining module is used to determine the global pose sequence of the robot during the mapping data acquisition process based on the mapping data of the warehouse space collected by the robot; wherein, the mapping data includes at least the image data collected by the robot through the detection module;

[0031] The model building module is used to determine the 3D model of the warehouse space based on the global pose sequence and mapping data;

[0032] The second determining module is used to determine the semantic information of each pixel in the image data;

[0033] The map building module is used to map the semantic information of each pixel to the 3D model of the warehouse space, thereby obtaining a 3D semantic map of the warehouse space.

[0034] Fourthly, embodiments of this application provide a robot scheduling device, including:

[0035] The path generation module is used to determine the robot's task topology path based on the robot's task to be performed and the 3D semantic map of the storage space.

[0036] The data extraction module is used to extract node matching data corresponding to at least one key node in the task topology path from the 3D semantic map;

[0037] The image acquisition module is used to acquire environmental images collected at key nodes as the robot travels along the task topology path;

[0038] The driving scheduling module is used to schedule the robot based on environmental images and node matching data corresponding to key nodes.

[0039] Fifthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0040] Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0041] In a seventh aspect, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements any of the implementation methods in the first aspect described above.

[0042] Based on the method of this application embodiment, a three-dimensional model of the warehouse space can be constructed in the mapping stage. By mapping the pixel semantic information in the image data to the three-dimensional model of the warehouse space, a three-dimensional semantic map of the warehouse space can be constructed. This not only provides an accurate three-dimensional spatial scale to support the robot's obstacle perception and path planning in the height direction, but also enhances the robot's ability to distinguish highly repetitive scenes and improves the positioning accuracy by using semantic information.

[0043] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0044] The accompanying drawings are provided for a better understanding of this solution and do not constitute a limitation of this application. Wherein:

[0045] Figure 1 A schematic diagram of a warehousing system provided for an exemplary embodiment of this application;

[0046] Figure 2 This is a flowchart illustrating an exemplary embodiment of the map construction method provided in this application. Figure 1 ;

[0047] Figure 3 This is a flowchart illustrating an exemplary embodiment of the map construction method provided in this application. Figure 2 ;

[0048] Figure 4 This is a flowchart illustrating an exemplary embodiment of the map construction method provided in this application. Figure 3 ;

[0049] Figure 5 This is a flowchart illustrating an exemplary embodiment of the robot scheduling method provided in this application. Figure 1 ;

[0050] Figure 6 This is a flowchart illustrating an exemplary embodiment of the robot scheduling method provided in this application. Figure 2 ;

[0051] Figure 7 This is a flowchart illustrating an exemplary embodiment of the robot scheduling method provided in this application. Figure 3 ;

[0052] Figure 8 This is a schematic diagram of a map building apparatus provided in an exemplary embodiment of this application;

[0053] Figure 9 This is a schematic diagram of a robot scheduling device provided in an exemplary embodiment of this application;

[0054] Figure 10 This is a schematic diagram of the internal structure of an electronic device provided in an exemplary embodiment of this application. Detailed Implementation

[0055] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0056] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0057] Figure 1 A schematic diagram of a warehousing system provided for an exemplary embodiment of this application, as shown below. Figure 1 As shown, the warehousing system 100 includes a storage area 10, multiple handling devices 20, at least one workstation 30, and a control device 40.

[0058] In some embodiments, such as Figure 1 As shown, the storage area 10 may include multiple aisles 101, each aisle 101 has multiple storage locations on both sides, and each storage location can accommodate a vehicle 102. In other words, the storage area has multiple rows or columns of vehicles, and the area between two adjacent rows or columns of vehicles can be called an aisle 101.

[0059] In some embodiments, each carrier 102 may include multiple cargo positions. These cargo positions can be used to place containers, bins, goods, or original packaging. This application does not limit this; the following embodiments use the placement of containers as an example for illustrative purposes. For example, a cargo position can be a cuboid-shaped storage space, and multiple cargo positions on the carrier 102 can be neatly arranged along the length, width, and height directions of the carrier. The container can be a product specifically designed for the carrier, a regular cargo box (also called a bin), or the packaging of goods (also called the original packaging). This application does not limit this.

[0060] In some embodiments, the carrier 102 can be a shelf, a turnover cart, a cage cart, etc. For example, the carrier 102 can be a mobile carrier (including but not limited to partition shelves, container shelves, picking shelves, mobile shelves, etc.) or a fixed carrier. The carriers provided in the embodiments of this application can refer to any carrier used to place containers.

[0061] The shelf may include at least one partition, which divides the carrier into at least two layers. The partition of the carrier is provided with at least one storage location. Each storage location can accommodate at least one container. The container placed in each storage location may be a box or a pallet. This application embodiment does not limit this.

[0062] For example, the handling equipment 20, also known as an automated handling equipment or a carrier handling equipment, is used to perform handling tasks. For instance, when the carrier 102 is a mobile carrier, the handling equipment 20 can move the mobile carrier within the storage area 10 of the warehousing system, or move the mobile carrier between the storage area 10 and the workstation 30.

[0063] In some embodiments, the handling device 20 may be a handling robot, such as an Autonomous Mobile Robot (AMR). For example, the handling robot may move under the mobile vehicle and then lift the mobile vehicle off the ground, thereby carrying the mobile vehicle on its journey.

[0064] In some embodiments, the number of workstations 30 in the warehousing system 100 may be one or more, and this application embodiment does not limit this. The following embodiments are exemplified by the warehousing system 100 including multiple workstations 30.

[0065] For example, the control device 40 can communicate with the workstation 30 and the handling equipment 20 via a local area network (LAN), wireless local area network (WLAN), and other networks to control the operation of the workstation 30 and the handling equipment 20. For instance, the control device 40 can assign handling tasks to the corresponding handling equipment 20 so that each handling equipment 20 performs handling operations based on the assigned handling tasks.

[0066] In some embodiments, the control device 40 may be a server or a terminal device, or a device deployed with a warehouse management system (WMS) and a robot management system (RMS). The terminal device may include at least one of a personal computer, laptop computer, smartphone, tablet computer, and portable wearable device; the server may include a standalone server or a server cluster consisting of multiple servers, and this embodiment of the application does not limit this.

[0067] Understandably, when the handling equipment 20 performs handling operations within the storage area 10, or between the workstation 30 and the storage area 10, the WMS or RMS system needs to plan the travel path of the handling equipment 20 based on the warehouse map. However, there are significant drawbacks to using traditional 2D grid maps for path planning and navigation of handling equipment, for example:

[0068] 1) Using 2D grid maps, the passable area of ​​the transport equipment can be determined in the horizontal direction. In actual operation, the transport equipment needs to use LiDAR to detect obstacles in the vertical direction. Some obstacles that LiDAR cannot detect can only be detected when a collision occurs.

[0069] 2) There are many highly repetitive scenarios in the warehousing system, especially in the storage area where there are highly repetitive 2D features between different aisles and different shelves. When the handling equipment deviates from the positioning based on the 2D features, it is easy to match the wrong work area.

[0070] 3) After the robot restarts, it needs to be repositioned at the designated restart point using fixed features (such as specific markers or fixed obstacles). If the robot shuts down abnormally during operation, it cannot be repositioned in place after the abnormality is repaired.

[0071] Based on this, this application proposes a map construction method that can construct a three-dimensional model of the warehouse space during the mapping stage. By mapping the pixel semantic information in the image data to the three-dimensional model of the warehouse space, a three-dimensional semantic map of the warehouse space is constructed. The constructed three-dimensional semantic map can not only provide an accurate three-dimensional spatial scale to support the robot's obstacle perception and path planning in the height direction, but also enhance the ability to distinguish highly repetitive scenes with the help of semantic information, thereby improving the accuracy of positioning.

[0072] Figure 2 This is a schematic flowchart of a map construction method provided in an exemplary embodiment of this application. This embodiment can be applied to electronic devices and... Figure 1 The system in, such as Figure 2 As shown, in some embodiments, the map construction method provided in this application includes steps S210-S240:

[0073] Step S210: Determine the global pose sequence of the robot during the mapping data acquisition process based on the mapping data of the warehouse space collected by the robot.

[0074] The mapping data collected by the detection module includes at least the image data collected by the robot through the detection module. The data collection path of the robot in the warehouse space can be planned in advance so that the data collection path covers the entire area of ​​the warehouse space, including but not limited to all aisles in the inventory area, the area around the workstation, equipment passages and potential obstacle areas, so as to ensure that the map built based on the collected mapping data can fully reflect the environmental characteristics of the warehouse system.

[0075] In some embodiments, step S210, which determines the global pose sequence of the robot during the mapping data acquisition process based on the mapping data of the warehouse space collected by the robot, includes: determining the initial pose sequence of the robot during the mapping data acquisition process based on the image data; extracting feature information of at least some image frames in the image data; performing loop closure detection in the image data based on the feature information to obtain a loop closure frame; and optimizing the initial pose sequence based on the loop closure frame to obtain the global pose sequence.

[0076] The image data includes a continuous sequence of image frames. The feature points between adjacent image frames can be matched by Simultaneous Localization and Mapping (SLAM) technology to estimate the camera motion between adjacent image frames. This gives each image frame in the image data an initial pose estimate, and the initial pose estimate of each image frame is used to obtain the initial pose sequence.

[0077] For example, local features can also be extracted from image frames of image data using the Structure from Motion (SFM) algorithm, and feature matching can be performed between image frames. The matched feature point pairs can be used to estimate the relative pose between cameras using the epipolar geometry algorithm or the PnP (Perspective-n-Point) algorithm, thereby obtaining the camera extrinsic parameters corresponding to each image frame and determining the initial pose sequence.

[0078] After obtaining the initial pose sequence of each image frame in the image data, feature information of image frames with discriminative and stable characteristics can be extracted from the image data to perform loop closure detection, identify different image frames (i.e., loop closure frames) collected when the robot repeatedly visits the same location, and use the detected loop closure frames to perform global optimization of the initial pose sequence to eliminate drift and obtain the global pose sequence.

[0079] For example, when performing loop closure detection, the extracted current frame can be compared with the historical keyframe database. If the similarity exceeds the threshold, the current frame can be regarded as a potential loop closure frame. Furthermore, for potential loop closure frames, the Random Sample Consensus (RANSAC) algorithm, epipolar geometry, or PnP algorithm can be used to verify whether they are real loop closure frames, thus avoiding false matching.

[0080] When using closed-loop frames to perform global optimization of the initial pose sequence, graph optimization can also be used. The pose of each image frame is used as a variable node. The constraint relationship between nodes is determined based on the relative pose between image frames and the closed-loop frame. A pose graph is constructed, and graph optimization is performed with the goal of minimizing the loss value of all constraint relationships. The optimized pose of each image frame is output based on the graph optimization, and then the global pose sequence is determined.

[0081] The method in this embodiment can effectively correct the accumulated errors in the initial pose sequence through closed-loop detection and global optimization, ensuring the accuracy and consistency of pose estimation during long-term, large-scale mapping data collection. This allows the global pose sequence to more realistically reflect the robot's motion trajectory and posture changes in the warehouse space, laying a solid pose foundation for the accurate construction of subsequent 3D maps.

[0082] In some embodiments, the robot's detection module may also include LiDAR, inertial measurement unit (IMU), and wheel speed odometer, etc. In addition to image data, based on the types of sensors in the detection module, the mapping data may also include other modal data such as point cloud data, IMU data, and wheel speed data collected by the detection module.

[0083] During data acquisition, the robot can collect data at a preset driving speed and sampling frequency to ensure the time synchronization of the multimodal data collected by the detection module and avoid pose estimation errors caused by data asynchrony. After acquiring the multimodal data collected by the detection module, the data can also be preprocessed, such as performing distortion correction and brightness equalization on image data, and filtering and ground segmentation on point cloud data.

[0084] In some embodiments, where the mapping data includes modal data other than image data, such as Figure 3 As shown, in step S210, the global pose sequence of the robot during the mapping data acquisition process is determined based on the mapping data of the warehouse space collected by the robot, including steps S301-S302:

[0085] Step S301: Establish the correlation between image data and other modal data to obtain multi-frame fused data.

[0086] Establishing the correlation between image data and other modal data can be understood as achieving temporal and spatial alignment between multimodal data, unifying different modal data into the same reference coordinate system, and fusing data, which is the multimodal mapping data after temporal and spatial alignment.

[0087] Step S302: Determine the global pose sequence based on the specified frame fusion data in the multi-frame fusion data.

[0088] The specified frame fusion data includes at least one frame from the multi-frame fusion data. For example, it can be multiple keyframes selected from the multi-frame fusion data that have a good spatial distribution and sufficient overlap between adjacent frames. The pose of each frame in the specified frame fusion data can be estimated by combining different modal data in the fusion data, thereby obtaining the global pose sequence of the specified frame fusion data. For the global pose sequence of the specified frame fusion data, different estimation methods can be used depending on the specific circumstances of the different modal data contained in the fusion data.

[0089] For example, when the fused data includes image data and point cloud data, the global pose sequence can be calculated using only image frames as described above. The relative pose between each frame of fused data can be initially estimated by using image frames through image feature matching. Then, iterative closest point (ICP) matching can be performed using point cloud frames in the point cloud data to provide absolute scale. The relative pose between frames is obtained by fusing the image feature matching results and the point cloud frame matching results. Based on this, the global pose sequence of the fused data of the specified frame can be obtained by arranging the fused data of the other frames in the order of timestamps, with the first frame of fused data as the origin of the coordinate system.

[0090] In some embodiments, when the fused data includes image data, IMU data, and wheel speed data, the initial global pose of the fused data of the specified frame can be determined first based on kinematic estimation using IMU data and wheel speed data. Then, the initial global pose can be corrected by constructing a factor map in combination with visual constraints in the image data, and finally, the global pose sequence of the fused data of the specified frame can be obtained.

[0091] For example, extended Kalman filtering can be used to fuse IMU data and wheel speed data to construct state equations and observation equations. The state variables can include robot position, attitude quaternions, linear velocity, IMU angular velocity bias, and acceleration bias. In the prediction phase, the attitude quaternions and linear velocity can be updated based on the IMU data to predict the robot position and update the state covariance matrix simultaneously. In the update phase, the linear velocity and angular velocity converted from the wheel speed data can be used as observation values ​​to calculate the Kalman gain and correct the state variables and covariance matrix. Based on each iteration of extended Kalman filtering, the optimal output position is extracted as the robot pose at the corresponding timestamp, which is the initial global pose of the corresponding frame in the fused data of the specified frame.

[0092] In some embodiments, when the fused data includes image data, point cloud data, IMU data, and wheel speed data, the initial global pose of the fused data of a specified frame can be determined first based on kinematic estimation using IMU data and wheel speed data. Then, the initial global pose can be corrected by combining the pose constraint information between the fused data of each frame determined based on image data and point cloud data, to obtain a global pose sequence.

[0093] For example, loop closure detection can be performed on fused data of specified frames using image data and point cloud data. For instance, ORB (Oriented Fast and Rotated BRIEF) features can be extracted from image frames, and candidate loop closure frames in the image dimension can be determined by calculating the feature similarity between the current frame and historical frames during the loop closure detection process. For point cloud data, key geometric features can be extracted from point cloud frames, and the point cloud feature similarity between the current frame and historical frames can be calculated to determine candidate loop closure frames in the point cloud dimension. For the fused data, only specific frames where both the image and point cloud dimensions are determined to be candidate loop closure frames are identified as loop closure frames to avoid mismatches due to a single modality.

[0094] After obtaining the global pose sequence and the closed-loop frame, a pose factor map can be constructed based on the specified frame fusion data, the initial pose sequence of the specified frame fusion data, and the closed-loop frame. The pose factor map is used to optimize each initial pose in the initial pose sequence, and the global pose sequence is determined based on the optimized pose.

[0095] For example, when constructing the pose factor graph, the initial pose corresponding to each frame in the initial pose sequence of the specified frame fusion data can be used as a variable node. Motion constraint information between adjacent frames in the specified frame fusion data is determined based on IMU data and wheel velocity data. Point cloud matching between adjacent frames is performed based on point cloud data to determine geometric constraint information between adjacent frames. Closed-loop constraint information is determined based on closed-loop frames. The motion constraint information, geometric constraint information, and closed-loop constraint information are used as edges connecting each node in the pose factor graph. Each constraint edge in the pose factor graph corresponds to an error function. The global error function is the weighted sum of all constraint errors. When performing graph optimization using the pose factor graph, the parameters of each variable node are iteratively adjusted with the goal of minimizing the global error function. In each iteration, the gradient of the error with respect to each pose parameter is calculated, and the parameters are adjusted along the gradient descent direction until the error converges and the pose factor graph optimization is complete. The optimized poses corresponding to each frame of fusion data are arranged in timestamp order to obtain the global pose sequence.

[0096] By using the method in this embodiment, the robustness and accuracy of robot global pose sequence estimation can be effectively improved through the fusion of multimodal data. Even under complex conditions such as changes in lighting, interference from dynamic obstacles, or failure of some sensor data in the warehouse space, it can still maintain high pose estimation accuracy, providing a more reliable pose reference for the subsequent construction of high-precision 3D semantic maps.

[0097] Step S220: Determine the three-dimensional model of the storage space based on the global pose sequence and mapping data.

[0098] In some embodiments, such as Figure 4 As shown, step S220, which determines the 3D model of the storage space based on the global pose sequence and mapping data, includes steps S401-S404:

[0099] Step S401: Construct the three-dimensional model boundary of the storage space based on the global pose sequence.

[0100] Specifically, the three-dimensional boundary of the storage space can be determined based on the location range involved in the global pose sequence, and a map can be constructed based on the determined three-dimensional boundary of the storage space. The 3D Gaussian Splatting (3DGS) technology uses the Gaussian distribution function as the core building element. Each Gaussian distribution represents a local feature in the scene. By combining and adjusting multiple Gaussian distributions, the shape, color, and lighting information in the three-dimensional scene can be accurately described. This expression based on the Gaussian distribution function enables the 3DGS technology to capture the surface texture and light and shadow changes of objects in the scene in a delicate way. Here, the 3DGS technology can be used to initialize the Gaussian distribution in the three-dimensional model boundary of the storage space, and the initialized Gaussian distribution can be iteratively optimized based on the feature information of each image frame in the image data and the global pose corresponding to each image frame in the global pose sequence, so as to represent the three-dimensional scene details in the image frame in the form of Gaussian distribution.

[0101] Step S402: Initialize a Gaussian distribution in the boundary of the three-dimensional model to obtain the initial three-dimensional model.

[0102] In constructing the three-dimensional boundary, a three-dimensional mesh can be uniformly divided within the boundary of the three-dimensional model (for example, the three-dimensional size of each mesh can be set to 0.1m×0.1m×0.1m). A preset number of Gaussian distributions are randomly initialized within each mesh to obtain the initial three-dimensional model.

[0103] For example, different numbers of Gaussian distributions can be initialized based on the regional attributes corresponding to each 3D grid in the warehouse space. For instance, three Gaussian distributions can be initialized in densely stocked shelving areas, and one Gaussian distribution can be initialized in open aisle areas. This allows for targeted optimization of the feature representation of different areas in the warehouse space, avoiding redundant initialization of Gaussian distributions in areas with simple features, which would otherwise waste computational resources. At the same time, it ensures that there are enough Gaussian distributions in areas with complex features to capture detailed information. When initializing the Gaussian distribution, its position parameters can be randomly offset based on the center coordinates of the 3D grid, the scaling parameters can be set to an initial range according to the grid size, the rotation parameters can be randomly generated, and the color parameters can be temporarily assigned a default value or initially estimated based on the average color of the corresponding region's image frame.

[0104] Step S403: For any Gaussian distribution in the initial 3D model, project the Gaussian distribution onto the pixel surface of the corresponding image frame to obtain the corresponding projection area, and determine the predicted color of the projection area and the pixel loss of the predicted color compared with the actual pixel color in the corresponding image frame.

[0105] For example, for each image frame in the mapping data, its camera intrinsic parameters are known information, defining the camera imaging rules (including focal length, principal point, and distortion coefficients). Its camera extrinsic parameters (including rotation matrix and translation vector) define the camera's position in the storage space during image acquisition, which can be obtained based on the previously determined global pose sequence. Thus, for any Gaussian distribution in the initial 3D model, the Gaussian distribution in 3D space can be projected onto the corresponding associated image pixel plane using a perspective projection algorithm, obtaining the coverage area of ​​the Gaussian distribution on the image frame. Based on the initial color parameters of the Gaussian distribution (such as RGB values), the "predicted color" of its projection area is calculated. If the Gaussian color is uniform, the predicted color of all pixels in the projection area is the Gaussian color. If the Gaussian color has a gradient, the predicted color of different pixels is assigned according to the Gaussian shape parameters (such as variance), and the pixel loss of each pixel is obtained by comparing the predicted color with the actual pixel color of the image.

[0106] Step S404: Determine the total loss function based on the mean pixel loss of the image frame, and iteratively update the position parameters, shape parameters, and color parameters of the Gaussian distribution in the initial 3D model through the backpropagation algorithm. Use the resulting optimized Gaussian distribution set as the 3D model of the storage space.

[0107] For example, in step S403, the pixel loss in each image frame has been determined. The average of the pixel loss in each image frame is used to obtain the total loss function of the model. The total loss is then propagated back to the parameters of each Gaussian distribution through the backpropagation algorithm. The position, shape, and color parameters of each Gaussian distribution are iteratively adjusted. The loss is recalculated after each iteration until the total loss converges, thereby obtaining an optimized set of Gaussian distributions. The Gaussian parameters are continuously corrected through error feedback, making the overall projection effect of each Gaussian distribution infinitely close to the real image. This allows for accurate fitting of the geometric structure and color texture of the warehousing system. For example, in the shelving area, the optimized Gaussian distribution can accurately represent the three-dimensional structure of the shelving, including the thickness of the shelves, the position of the uprights, and the relative distance between the shelves. At the same time, by adjusting the color parameters, the surface texture of the shelving and the color differences in different areas are realistically reproduced. For the goods stacking area, the Gaussian distribution can finely depict the shape outline and stacking layers of the goods, avoiding the problem of losing details.

[0108] In some embodiments, in step S402, if the mapping data includes point cloud data, the Gaussian distribution can be initialized and generated directly with the sampling points in the point cloud frame corresponding to the grid as the center of the Gaussian sphere based on the downsampling results of the point cloud data, so that the initialized Gaussian distribution fits the geometric features of the corresponding area in the storage space, saving the iteration optimization time in step S404.

[0109] The optimized Gaussian distribution set retains the precise geometric structure of the warehouse scene (shelves, aisles, columns, etc.) and contains realistic color and texture information. Based on the optimized Gaussian distribution set, the 3DGS volume rendering algorithm renders the Gaussian distribution in different areas by layering and overlaying according to the transparency parameter and depth information of the Gaussian distribution during the rendering process. This enables high-quality visualization of the warehouse space and forms a three-dimensional model of the warehouse space.

[0110] The method described in this embodiment, which constructs a 3D model using 3DGS technology, not only possesses the geometric accuracy of traditional point cloud models but also has near-image-level texture details. Especially when modeling objects with complex surface features, such as shelves and goods commonly found in warehouse environments, 3DGS technology, through the combination and optimization of a large number of Gaussian distributions, can delicately reproduce the surface texture, color gradients, and lighting effects of objects. This provides a more intuitive and accurate environmental perception foundation for robots to perform fine operations in warehouse scenarios (such as goods picking and shelf inventory).

[0111] Furthermore, due to the parameterization characteristics of the Gaussian distribution, this 3D model also has certain advantages in data storage and transmission. By adjusting the number and precision of the Gaussian distribution, the model can be expressed at multiple scales, meeting the needs of different application scenarios for model detail and lightweighting.

[0112] Step S230: Determine the semantic information of each pixel in the image data.

[0113] For each image frame used for mapping in the image data, semantic information of pixels in the image frame can be extracted. For example, pixel-level semantic annotation can be performed by a pre-trained semantic segmentation model to obtain the semantic category to which each pixel belongs. A list of semantic categories can be predefined based on the core elements of the warehousing scenario. Semantic categories include, but are not limited to, static facilities (shelves, walls, columns, floors), goods-related (goods, pallets, storage location labels), and functional areas (entrances and exits, charging areas), etc. Semantic IDs are determined for different semantic categories. For example, the semantic ID of partition shelves is 1, and the semantic ID of container shelves is 2. The annotation rules for different semantic categories are also clarified. For example, storage location labels need to accurately annotate the character area, and shelves need to distinguish between the main body, shelves, and columns to avoid category confusion.

[0114] For example, the semantic segmentation model can adopt a dual-branch network structure. For instance, ResNet can be used as the backbone network to extract multi-scale features from the image. Low-level features retain rich details such as edges and textures for accurate object boundary segmentation; high-level features have stronger semantic expressive power for accurate semantic category identification. During the feature fusion stage, an attention mechanism is used to assign different weights to feature maps at different levels, focusing on feature regions that contribute significantly to semantic segmentation, such as features of key semantic objects like shelf pillars and product labels. For common semantic categories in warehousing scenarios (such as shelves, goods, floors, aisles, walls, and pillars), the model has already learned the corresponding feature patterns during training. During inference, image frames are input into the model, and after feature extraction, feature fusion, and upsampling, a semantic segmentation map of the same size as the image frame is output. Each pixel in the semantic segmentation map corresponds to a semantic category probability distribution. The category with the highest probability is selected as the semantic category of that pixel, thus determining the semantic information of each pixel in the image data.

[0115] When semantic segmentation models are used for semantic annotation, for areas with annotation errors (such as category confusion caused by overlapping goods, boundary offset caused by blurred shelf edges, and broken characters on storage location labels), the semantic boundaries can be manually adjusted using annotation tools to ensure the accuracy of annotation in key areas.

[0116] In some embodiments, to improve the accuracy of semantic annotation, if the mapping data also includes point cloud data, multimodal semantic fusion can be performed by combining the point cloud data. For example, for a pixel in an image frame, its corresponding three-dimensional spatial position can be obtained through depth estimation or point cloud registration, thereby obtaining the geometric features of the point cloud data at that position (such as normal vector, curvature, neighborhood point density, etc.). The geometric features and image features (such as color, texture, contextual information) are input into the semantic classifier, and the semantic category of the pixel is comprehensively judged through decision fusion, effectively making up for the semantic recognition defects of a single image modality under conditions such as illumination changes and occlusion.

[0117] Step S240: Map the semantic information of each pixel to the three-dimensional model of the storage space to obtain a three-dimensional semantic map of the storage space.

[0118] For example, based on the determined camera intrinsic and extrinsic parameters, each Gaussian distribution in the 3D model can be back-projected onto the corresponding image frame to determine the pixel region it covers and establish the association between pixel semantic information and Gaussian distribution. Then, the semantic information of these covered pixels is statistically analyzed. For example, a voting mechanism can be used to take the semantic category with the highest proportion in the projected region as the semantic attribute of the Gaussian distribution; or, considering that the semantic confidence of different pixels may differ, a weighted average method can be used to weight the semantic category according to the semantic probability of the pixel (such as the category probability output by the semantic segmentation model), thereby more accurately determining the semantic affiliation of the Gaussian distribution to obtain a set of Gaussian distributions with semantic attributes.

[0119] In some embodiments, for Gaussian distributions with relatively dispersed semantic categories or low confidence levels within the projection area, context inference can be performed by combining their spatial location information in the 3D model with the semantic categories of adjacent Gaussian distributions. For example, Gaussian distributions located near the main body of the shelf are more likely to belong to related semantic categories such as shelf panels or uprights, thereby improving the accuracy and consistency of semantic mapping.

[0120] By integrating a set of Gaussian distributions with semantic attributes, the original 3D model containing only geometric and color information is endowed with rich semantic attributes, resulting in a 3D model of the warehouse space. Each Gaussian distribution in the 3D model not only represents a geometric and appearance feature of the warehouse space, but also has its own semantic category.

[0121] Based on the 3D model of the warehouse space, it can be further output as a 3D semantic point cloud map or a semantic mesh map. The 3D semantic point cloud map can be generated by sampling the Gaussian distribution in the 3D semantic model, with each sampling point carrying corresponding semantic category information; the semantic mesh map can be constructed based on the 3D semantic model to build a 3D mesh and assign a semantic label to each mesh unit to meet the map data format requirements of different robot navigation and task planning systems.

[0122] Through steps S210 to S240, this embodiment realizes the process of constructing a high-precision, semantically rich three-dimensional semantic map of the warehouse space from mapping data. This map can not only accurately reflect the geometric structure and appearance features of the warehouse environment, but also clearly express the semantic attributes of each area and object, providing a comprehensive and accurate environmental cognition foundation for robots to perform tasks such as autonomous localization, path planning, target recognition and grasping in the warehouse scenario.

[0123] The specific settings and implementation methods of the embodiments of this application have been described above from different perspectives. Using the methods provided in the above embodiments, a warehouse 3D semantic map with both high-precision geometric details and rich semantic information can be constructed through 3DGS technology. While ensuring map accuracy, it also enhances adaptability and robustness to complex warehouse environments, providing key support for robots to achieve efficient autonomous operation in warehouse environments.

[0124] Figure 5 This is a flowchart illustrating an exemplary embodiment of a robot scheduling method provided in this application. This embodiment can be applied to electronic devices and RMS systems, such as... Figure 5 As shown, in some embodiments, the robot scheduling method provided in this application includes steps S510-S540:

[0125] Step S510: Based on the robot's task to be performed and the three-dimensional semantic map of the storage space, determine the robot's task topology path.

[0126] In step S510, the robot can be a robot that collects mapping data during the map building phase and operates in the warehouse space, or it can be another robot performing tasks in the warehouse space. The 3D semantic map can be a pre-constructed 3D map of the warehouse space containing semantic information. It can be based on the map building method described in the above embodiments, calculating the 3D model of the warehouse space based on the global pose sequence and mapping data of the robot during its movement in the warehouse space, and mapping the semantic information of each pixel in the image data collected by the robot during its movement in the warehouse space to the 3D model of the warehouse space. Alternatively, the 3D semantic map can be obtained based on other existing 3D semantic map construction methods, and this application embodiment does not limit this.

[0127] In some embodiments, the task to be performed may include a task objective (such as moving goods from storage location A to storage location B) and a task type (such as receiving, shipping, inventory counting, replenishment, etc.). The task objective area can be determined based on the robot's task to be performed, and the robot's traversable area can be determined in a 3D semantic map. Thus, starting from the robot's current position and ending at the task objective area, a task topology path can be determined within the traversable area based on a path planning algorithm.

[0128] The determination of passable areas requires combining semantic categories and geometric structures. For example, spatial areas with the semantic ID "passage" and a height range of 0.1m-1.8m are selected, while 3D spaces corresponding to static obstacles such as "shelves" and "pillars" are excluded. An improved A / B algorithm can be used for path planning. The algorithm uses semantic information as a weighting factor in its heuristic function. For example, it prioritizes path segments with the semantic label "main passage" (width ≥ 2m) to improve traffic efficiency, while avoiding semantically sensitive areas with high dynamic obstacle incidence, such as "temporary goods storage areas." After generating the initial topology path, path smoothing is performed based on the precise geometric parameters of the warehouse 3DGS geometric model. For instance, Bézier curves are used to fit inflection points in the path to ensure that the path curvature meets the robot's kinematic constraints, thus avoiding the risk of local collisions caused by minor protrusions in structures such as shelf uprights.

[0129] A task topology path refers to the sequence of path nodes traversed from the robot's current position (or task starting point) to the task target area under the semantic topology relationship of the warehouse space. For example, "charging area → main aisle → aisle of shelf A area → target storage location A", where "charging area", "main aisle", "aisle of shelf A area", and "target storage location A" are all path nodes in the task topology path.

[0130] Using the method of this embodiment, when determining the robot's task topology path based on a 3D semantic map, the robot's passable area can be determined based on the 3D geometric features of the warehouse space. This solves the problem that traditional 2D grid maps cannot accurately determine obstacles in the height direction, avoids the risk of local collisions, and ensures the safe passage of the robot.

[0131] Step S520: Extract node matching data corresponding to at least one key node in the task topology path from the 3D semantic map.

[0132] Key nodes can include critical decision points in the process of a robot navigating to the target area. For example, key decision points can include area entrances and exits, aisle intersections, lane entrances and exits, and corners. At the intersection of the main aisle and branch lanes of the warehousing system, since there are multiple path choices, it can be set as a key node to assist the robot in making directional decisions. At the entrances and exits of the shelving aisles, since it involves the robot's posture adjustment when entering and exiting the shelving area, it can also be set as a key node.

[0133] For example, at least one semantic marker in the environment surrounding a key node can be identified in a 3D semantic map, and the semantic information of the semantic marker can be extracted as node matching information. The semantic marker can be selected based on the location of the key node in the warehouse space, including but not limited to entrance / exit signs, aisle numbers, shelf numbers, and storage location numbers. For example, at an aisle entrance / exit, a sign marked "Aisle A01" can be used as a semantic marker. In addition to the semantic features of the semantic marker, the node matching data can also include the semantic marker's precise 3D coordinates, size parameters (such as the length, width, and height of the sign, and the font size and spacing of the characters), color features (such as red for the number characters and blue for the background), and relative positional relationship with the surrounding environment (such as the sign being 1.5m above the ground and 0.3m to the right of the shelf upright). For semantic markers containing character information, such as storage location numbers, the text content and layout format of the characters can be further extracted as key components of the node matching data.

[0134] Step S530: Obtain environmental images collected at key nodes as the robot travels along the task topology path.

[0135] In some embodiments, when the robot travels along the task topology path, it can perform regular navigation based on a 2D grid map pre-extracted from a 3D semantic map combined with its own odometer or positioning tags affixed to the warehouse floor. When the robot approaches a key node, it can adjust its driving state (e.g., decelerate) so that when it reaches the key node, it can take pictures of the area where the semantic markers at the key node are located based on a preset pose and shooting angle, and upload the captured environmental images to the RMS system.

[0136] Step S540: Schedule the robot based on the environmental image and the node matching data corresponding to the key nodes.

[0137] After receiving environmental images collected by the robot at key nodes, the RMS system can determine whether the robot has deviated from its course based on the node matching data corresponding to the key nodes.

[0138] In some embodiments, such as Figure 6 As shown, in step S540, the robot is scheduled based on the environmental image and the node matching data corresponding to the key nodes, including steps S601-S602:

[0139] Step S601: Extract target pixel regions from the environmental image that match the semantic category of semantic markers in the node matching data.

[0140] For example, the semantic category of each pixel can be obtained by performing pixel-level semantic segmentation on the environmental image. Then, pixel regions that match the semantic category of semantic markers (such as the semantic category of "cargo location label" corresponding to the "lane A01" sign) in the node matching data can be selected as target pixel regions. For instance, if the semantic category of the semantic marker in the node matching data is "cargo location label", then all pixels with the semantic category of "cargo location label" can be extracted from the semantic segmentation results of the environmental image as target pixel regions.

[0141] Step S602: Based on the target pixel region and node matching data, determine the robot's deviation from the task topology path, and schedule the robot's movement according to the deviation.

[0142] The node matching data contains semantic features of semantic markers. The matching degree between the target pixel region and the semantic markers can be determined based on the node matching data. For example, by calculating the cosine similarity or Euclidean distance between the feature vector of the target pixel region (such as character sequence, color distribution, geometric contour) and the standard feature vector of the semantic marker in the node matching data, if the matching degree is greater than a preset threshold, it is determined that the robot's current position matches the key node, confirming that the robot has not deviated from the task topology path, and controlling the robot to continue traveling along the task topology path.

[0143] If the matching degree is less than the preset threshold, it is determined that the robot has deviated from the task topology path. It is necessary to determine the robot's actual pose in the warehouse space based on the environmental image and the 3D semantic map of the warehouse space. For example, deviations may include positional deviations (such as the deviation of the robot's actual position from the 3D coordinates of the key node exceeding 0.5m) and directional deviations (such as the deviation of the robot's heading angle from the preset direction exceeding 15°). For different types of deviations, the RMS system can generate corresponding scheduling instructions: for slight positional deviations, real-time path correction can be performed by adjusting the robot's speed increment (such as increasing the speed of the left wheel and decreasing the speed of the right wheel); for directional deviations, a stationary rotation instruction (such as "rotate 10° clockwise") can be sent to calibrate the heading.

[0144] In some embodiments, if the deviation from the position of the key node exceeds a preset distance, the determined actual pose of the robot can be used as the starting point and the target area of ​​the task as the ending point to re-plan the task topology path for the robot in the three-dimensional semantic map.

[0145] The method in this embodiment, by deeply integrating the three-dimensional semantic map of the warehouse space during robot scheduling, not only achieves accurate planning of the task topology path, but also enables the robot to make intelligent decisions based on rich semantic information in complex and ever-changing warehouse environments through semantic marker matching of key nodes and real-time analysis of environmental images. This effectively improves navigation accuracy and provides a strong guarantee for the efficient and safe operation of warehouse robots.

[0146] In some embodiments, if the three-dimensional semantic map constructed in Embodiment 1 is used as the basis, the node matching data may also include 3DGS information of at least one semantic marker of the location of the key node. The 3DGS information can be extracted from the three-dimensional semantic map constructed based on 3DGS. Based on the semantic features of the semantic marker, the geometric morphology data of the semantic marker based on Gaussian distribution set representation is further included. In step S602, the robot's deviation from the task topology path can be determined more accurately based on the semantic features of the semantic marker and the 3DGS information.

[0147] In some embodiments, when the node matching data contains 3DGS information of semantic markers, real-time pose correction of the robot can also be performed based on the 3DGS information of the semantic markers to control the robot to continue traveling along the task topology path. The steps of real-time pose correction may include:

[0148] Based on the 3DGS information of semantic markers and the robot's current pose at key nodes, the semantic markers are projected onto the plane containing the target pixel region to obtain the ideal projection features corresponding to the semantic markers. The projection error between the ideal projection features and the actual image features of the target pixel region is calculated. The projection error is used as a loss function to iteratively optimize the pose parameters, determine the corrected pose of the robot at the key nodes, and send the corrected pose to the robot. The robot is then controlled to update its current pose using the corrected pose and continue driving based on the updated pose and task topology path.

[0149] First, based on the robot's current pose at key nodes (provided by the robot's odometry) and the camera intrinsics of the robot's detection module, a projection relationship from the storage space to the image plane can be established. Then, a set of Gaussian distributions of semantic markers is extracted from the 3DGS information, including the 3D coordinates, shape parameters (covariance matrix), and color parameters of each Gaussian. Each Gaussian distribution is projected onto the image plane where the target pixel region is located using a perspective projection algorithm to obtain the 2D projection elliptical region of each Gaussian. Finally, the ideal color distribution of the projection region is calculated based on the color parameters, and integrated to form an ideal projection feature that includes contour shape, color texture, and spatial topological relationship.

[0150] Secondly, key parameters of the ideal projection features and the actual image features of the target pixel region are extracted. For example, for contour features, the contour point sets of both can be extracted using edge detection algorithms, and the Euclidean distance between corresponding point pairs can be calculated as the contour error. For color and texture features, the difference in color distribution is calculated using the structural similarity index or mean square error as the texture error. For spatial topological relationships, the topological error is calculated by judging the relative positional deviation between the character region and the background region (such as whether the character is centered or whether the character spacing matches). The contour error, texture error, and topological error are then weighted and summed according to preset weights to obtain the total projection error.

[0151] Subsequently, an optimization equation for the robot's pose parameters is constructed using the projection error as the loss function, and the optimal solution for the pose parameters is iteratively solved. During the iteration process, the ideal projection features are recalculated and the projection error is updated after each adjustment of the pose parameters, until the error value is less than a set threshold or the maximum number of iterations is reached. Finally, the converged pose estimate is output, and the pose estimate is used as the robot's corrected pose at key nodes. The RMS system can send the corrected pose to the robot, which can update the current odometry pose based on the corrected pose and continue driving based on the updated current pose.

[0152] The method employed in this embodiment introduces 3DGS information to model the precise geometric shape of semantic markers. This allows robot pose correction to no longer rely on traditional sparse feature point matching, but instead utilizes the dense geometric information and texture details represented by a continuous Gaussian distribution set, thereby reducing the interference of image noise on the matching results. Simultaneously, the iterative optimization process based on projection error can quickly converge to the optimal pose parameters, ensuring the robot completes pose correction within milliseconds, meeting the real-time requirements of warehousing scenarios. The scheduling strategy, which deeply integrates 3D semantic information with 3DGS geometric representation, further enhances the robot's autonomous navigation capabilities and environmental adaptability in dynamic warehousing environments, providing more reliable technical support for the intelligent and unmanned upgrade of warehousing logistics.

[0153] In some embodiments, the robot scheduling method of this application further includes a relocation scheme for robots in the warehouse space, such as... Figure 7 As shown, the robot scheduling method also includes steps S710-S740:

[0154] Step S710: In response to receiving the robot's relocation request, acquire the current environment image collected by the robot.

[0155] For example, the robot in step S710 can be a robot that needs repositioning in the storage space after an abnormal restart, or it can be a robot that needs to be repositioned in the storage space and replanned based on the repositioning result after being detected to have deviated from the task topology path in step S602. When the robot generates a repositioning requirement, it can use the detection module to capture multiple frames of images of the surrounding environment as the robot's current environment image, and send the current environment image along with the repositioning request to the RMS system; the robot can also first send a repositioning request to the RMS system, and capture and feed back the current environment image after receiving the response from the RMS system.

[0156] Step S720: Based on the 3D semantic map and the current environment image, determine at least one candidate region that matches the current environment image.

[0157] For example, feature extraction can be performed on the current environment image, such as texture feature extraction and semantic feature extraction, to obtain a local feature set of the current environment image. The local feature set contains the pixel coordinates, semantic information, and texture descriptors of each pixel. During the mapping stage, key image frames used to construct the 3D semantic map can be retained, and the features of the key image frames can be pre-extracted and constructed into a mapping image frame feature set. Based on the local feature set of the current environment image and the mapping image frame feature set of the 3D semantic map, at least one mapping image frame matching the local feature set can be determined from the mapping image frame feature set, and the region corresponding to the matching mapping image frame in the 3D semantic map can be determined as at least one candidate region matching the current environment image.

[0158] For example, if the semantic information of the "Lane A01" sign and the color features of red characters and blue background are extracted from the current environmental image, mapping image frames containing the same semantic category and color features can be retrieved from the feature set of the mapping image frames. The "Lane A01" entrance / exit area corresponding to that mapping image frame can then be used as a candidate region. The number of candidate regions can be one or more, depending on the degree of matching between the current environmental image features and the mapping image frame features, as well as the distribution of semantic markers in the warehouse space.

[0159] Step S730: Based on the 3D semantic map and at least one candidate region, determine at least one candidate pose of the robot.

[0160] For each candidate region, the 3DGS model and mapping pose corresponding to the candidate region are extracted from the 3D semantic map. The 3DGS model includes the 3D coordinates, covariance matrix, color information and semantic labels of all Gaussian distributed 3D regions in the region, ensuring that the model covers the geometric and semantic details of key semantic elements such as shelves, aisles and signs. The mapping pose is the global pose of the mapping image frame corresponding to the candidate region in the 3D semantic map.

[0161] For each candidate region, according to the 3DGS model and mapping pose corresponding to each candidate region, project the Gaussian distribution in the 3DGS model onto the two-dimensional image plane to generate a virtual rendering image corresponding to each candidate region, and optimize the mapping pose of each candidate region based on the difference between the virtual rendering image corresponding to each candidate region and the current environment image, so as to obtain at least one candidate pose.

[0162] When calculating the difference between the virtual rendering image corresponding to the candidate region and the current environment image, the photometric loss and structural similarity loss between the virtual rendering image and the current environment image can be calculated respectively, and an image difference loss is constructed as a loss function by combining the weights between the preset photometric loss and structural similarity loss, and the translation amount and rotation angle of the mapping pose are iteratively optimized through the gradient descent algorithm until the image difference loss is less than the preset threshold, and the optimized pose at this time is the candidate pose corresponding to the candidate region. For example, if the candidate region is the entrance and exit of "Tunnel A01" and its mapping pose is (x0, y0, z0, θ0, φ0, ψ0), then taking this mapping pose as the initial value, by adjusting the translation parameters (Δx, Δy, Δz) and rotation parameters (Δθ, Δφ, Δψ), the weighted sum of the photometric loss (such as the mean square error of pixel gray values) and structural similarity loss (such as the SSIM index) between the virtual image rendered based on the adjusted pose and the current environment image is minimized, and the finally obtained optimized pose (x0 + Δx , y0 + Δy , z0 + Δz , θ0 + Δθ , φ0 + Δφ , ψ0 + Δψ ) is the candidate pose corresponding to the candidate region. If there are multiple candidate regions, each candidate region can generate a candidate pose through the above method, so as to obtain at least one candidate pose of the robot.

[0163] Step S740, determine the relocalization pose of the robot among at least one candidate pose.

[0164] Exemplarily, based on the number of candidate regions, the corresponding number of candidate poses can be obtained, so there may be multiple candidate poses, and the most accurate relocalization pose needs to be selected from them. Specifically, the candidate pose with the smallest loss value can be determined as the relocalization pose of the robot through the image difference loss values corresponding to each candidate pose after iterative optimization. For example, for the two candidate regions of the entrance and exit of "Tunnel A01" and the entrance and exit of "Tunnel B02", the candidate poses P1 and P2 are respectively optimized, and the corresponding image difference loss values are L1 and L2. If L1 < L2, then P1 is determined as the relocalization pose of the robot.

[0165] The method in this embodiment combines the rich semantic information in the 3D semantic map with the precise geometric representation of the 3DGS model, which can quickly and accurately restore the robot's global position in the warehouse space when the robot loses its pose or deviates significantly from the path, thus significantly improving the robustness and fault tolerance of the warehouse robot system.

[0166] The specific settings and implementation methods of the embodiments of this application have been described above from different perspectives. Using the methods provided in the above embodiments, the robot's pose is accurately estimated and dynamically corrected at key nodes by using 3DGS information of semantic markers. Combined with the global planning capabilities of a 3D semantic map, the robot's scheduling in the warehouse environment possesses a closed-loop capability of "semantic understanding - geometric matching - real-time optimization." This scheduling method, which deeply integrates semantic and geometric information, not only solves the problems of traditional sparse feature-based navigation being susceptible to occlusion and lighting changes, but also provides the robot with millimeter-level pose control accuracy through precise modeling of environmental details using 3DGS technology, meeting the stringent requirements for robot travel paths in high-density warehouse scenarios. Simultaneously, the semantic-first matching strategy in the relocation scheme significantly shortens the pose recovery time, ensuring that even if the robot briefly loses its position in a complex environment, it can quickly reintegrate into the work process, effectively avoiding a decline in overall logistics efficiency due to single-machine failure.

[0167] As an implementation of the method in the above embodiments, such as Figure 8 As shown in the figure, this application embodiment also provides a map building apparatus, which may include:

[0168] The first determining module 801 is used to determine the global pose sequence of the robot during the mapping data acquisition process based on the mapping data of the warehouse space collected by the robot; wherein, the mapping data includes at least the image data collected by the robot through the detection module;

[0169] Model building module 802 is used to determine the three-dimensional model of the warehouse space based on the global pose sequence and mapping data;

[0170] The second determining module 803 is used to determine the semantic information of each pixel in the image data;

[0171] The map building module 804 is used to map the semantic information of each pixel to the three-dimensional model of the warehouse space to obtain a three-dimensional semantic map of the warehouse space.

[0172] In some embodiments, the first determining module 801 is configured to: determine the initial pose sequence of the robot during the mapping data acquisition process based on the image data; extract feature information of at least some image frames in the image data; perform loop closure detection in the image data based on the feature information to obtain a loop closure frame; and optimize the initial pose sequence based on the loop closure frame to obtain a global pose sequence.

[0173] In some embodiments, the mapping data also includes other modal data collected by the robot through the detection module, and the first determining module 801 is further configured to: establish the correlation between image data and other modal data to obtain multi-frame fusion data; and determine the global pose sequence based on the specified frame fusion data in the multi-frame fusion data.

[0174] In some embodiments, the first determining module 801 is configured to: determine the initial pose sequence of the specified frame fusion data based on the first modal data and the second modal data in the specified frame fusion data; determine the closed-loop frame in the specified frame fusion data using image data and the third modal data in the specified frame fusion data; construct a pose factor map based on the specified frame fusion data, the initial pose sequence of the specified frame fusion data, and the closed-loop frame; and optimize the initial pose sequence of the specified frame fusion data based on the pose factor map to obtain a global pose sequence.

[0175] In some embodiments, the first modal data is inertial measurement data; and / or, the second modal data is wheel speed data; and / or, the third modal data is point cloud data.

[0176] In some embodiments, the map construction module 804 is used to: construct a three-dimensional model boundary of the storage space based on a global pose sequence; initialize a Gaussian distribution in the three-dimensional model boundary to obtain an initial three-dimensional model; for any Gaussian distribution in the initial three-dimensional model, project the Gaussian distribution onto the pixel surface of the corresponding image frame to obtain the corresponding projection area, determine the predicted color of the projection area and the pixel loss of the predicted color compared to the actual pixel color in the corresponding image frame; determine the total loss function based on the mean pixel loss of the image frame, iteratively update the position parameters, shape parameters and color parameters of the Gaussian distribution in the initial three-dimensional model through a backpropagation algorithm, and use the obtained optimized Gaussian distribution set as the three-dimensional model of the storage space.

[0177] In some embodiments, the map building module 804 is used to: back-project the Gaussian distribution in the 3D model to the corresponding image frame in the image data, and determine the pixel area covered by each Gaussian distribution; determine the semantic attributes of each Gaussian distribution based on the semantic information of each pixel in the pixel area covered by each Gaussian distribution, so as to obtain a set of Gaussian distributions with semantic attributes; and generate a 3D semantic map of the warehouse space based on the set of Gaussian distributions with semantic attributes.

[0178] As an implementation of the method in the above embodiments, such as Figure 9 As shown in the figure, this application embodiment also provides a robot scheduling device, which may include:

[0179] The path generation module 901 is used to determine the robot's task topology path based on the robot's task to be performed and the three-dimensional semantic map of the storage space.

[0180] Data extraction module 902 is used to extract node matching data corresponding to at least one key node in the task topology path from the 3D semantic map;

[0181] The image acquisition module 903 is used to acquire environmental images collected at key nodes when the robot travels along the task topology path;

[0182] The driving scheduling module 904 is used to schedule the robot based on the environmental image and the node matching data corresponding to the key nodes.

[0183] In some embodiments, the three-dimensional semantic map is obtained by mapping the semantic information of each pixel in the image data collected by the robot during its movement in the warehouse space to a three-dimensional model of the warehouse space; the three-dimensional model of the warehouse space is calculated based on the global pose sequence and mapping data of the robot during its movement in the warehouse space, and the mapping data includes at least the image data collected by the detection module when the robot is moving in the warehouse space.

[0184] In some embodiments, the path generation module 901 is used to: determine the traversable area of ​​the robot in a three-dimensional semantic map; and determine the task topology path in the traversable area based on a path planning algorithm, starting from the robot's current position and ending at the task target area of ​​the task to be performed.

[0185] In some embodiments, the node matching data includes the semantic features of at least one semantic marker of the location of a key node, and the driving scheduling module 904 is used to: extract target pixel regions from the environmental image that are consistent with the semantic category of the semantic markers in the node matching data; determine the robot's driving deviation from the task topology path based on the target pixel regions and the node matching data, and schedule the robot's driving based on the driving deviation.

[0186] In some embodiments, the driving scheduling module 904 is used to: determine the matching degree between the target pixel region and the semantic marker based on node matching data; if the matching degree is greater than or equal to a preset threshold, determine that the robot has not deviated from the task topology path, and control the robot to continue driving along the task topology path; if the matching degree is less than the preset threshold, determine that the robot has deviated from the task topology path, determine the robot's actual pose in the warehouse space based on the environmental image and the three-dimensional semantic map of the warehouse space, and replan the task topology path for the robot according to the actual pose.

[0187] In some embodiments, the node matching data further includes 3DGS information of at least one semantic marker at the location of the key node. The driving scheduling module 904 is further configured to: project the semantic marker onto the plane of the target pixel region based on the 3DGS information of the semantic marker and the robot's current pose at the key node to obtain the ideal projection feature corresponding to the semantic marker; calculate the projection error between the ideal projection feature and the actual image feature of the target pixel region; iteratively optimize the pose parameters using the projection error as a loss function to determine the corrected pose of the robot at the key node; send the corrected pose to the robot, control the robot to update the current pose using the corrected pose, and continue driving based on the updated pose and task topology path.

[0188] In some embodiments, the driving scheduling module 904 is further configured to: in response to receiving a relocation request from the robot, acquire a current environment image collected by the robot; determine at least one candidate region matching the current environment image based on the three-dimensional semantic map and the current environment image; determine at least one candidate pose of the robot based on the three-dimensional semantic map and the at least one candidate region; and determine the relocation pose of the robot in the at least one candidate pose.

[0189] In some embodiments, the driving scheduling module 904 is further configured to: extract features from the current environment image to obtain a local feature set of the current environment image; determine at least one mapping image frame that matches the local feature set based on the local feature set and the feature set of the mapping image frames of the three-dimensional semantic map; and determine at least one candidate region that matches the current environment image based on the at least one mapping image frame.

[0190] In some embodiments, the driving scheduling module 904 is further configured to: extract the 3DGS model and mapping pose corresponding to each candidate region based on the 3D semantic map; generate a virtual rendering image corresponding to each candidate region based on the 3DGS model and mapping pose corresponding to each candidate region; and optimize the mapping pose of each candidate region based on the difference between the virtual rendering image corresponding to each candidate region and the current environment image to obtain at least one candidate pose.

[0191] In some embodiments, the driving scheduling module 904 is further configured to: for any candidate region, construct a loss function by using the photometric loss and structural similarity loss between the virtual rendered image and the current environment image as image difference loss, iteratively optimize the pose parameters of the mapping pose through backpropagation algorithm to obtain a candidate pose; determine the robot's relocalization pose in at least one candidate pose, including: determining the image difference loss corresponding to each candidate pose; and determining the candidate pose with the smallest corresponding image difference loss as the relocalization pose.

[0192] The functions of each unit, module, or sub-module in the various devices of this application embodiment can be found in the corresponding descriptions in the above method embodiments, and they have corresponding beneficial effects, which will not be repeated here.

[0193] like Figure 10 The diagram shown is a schematic representation of the internal structure of an electronic device provided in this embodiment. This electronic device can be a server. It includes a processor, a memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores height parameters and 3D map data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements the method of this embodiment.

[0194] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the robots and electronic devices to which the present application is applied. Specific robots and electronic devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0195] In a specific implementation, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method in any of the above embodiments.

[0196] In a specific implementation, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method in any of the above embodiments.

[0197] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0198] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in this application.

[0199] It should be understood that the aforementioned processor can be a CPU, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), FPGAs, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0200] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0201] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0202] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0203] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0204] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0205] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A map construction method characterized by comprising: include: Based on the mapping data of the warehouse space collected by the robot, the global pose sequence of the robot during the mapping data acquisition process is determined; wherein, the mapping data includes at least the image data collected by the robot through the detection module; The three-dimensional model of the storage space is determined based on the global pose sequence and the mapping data; wherein the three-dimensional model of the storage space includes a set of Gaussian distributions constructed by three-dimensional Gaussian sputtering; Determine the semantic information of each pixel in the image data; The semantic information of each pixel is mapped to the three-dimensional model of the warehouse space to obtain a Gaussian distribution set with semantic attributes, and a three-dimensional semantic map of the warehouse space is generated based on the Gaussian distribution set with semantic attributes. The step of determining the 3D model of the warehouse space based on the global pose sequence and the mapping data includes: The three-dimensional model boundary of the warehouse space is constructed based on the global pose sequence; A Gaussian distribution is initialized within the boundary of the three-dimensional model to obtain the initial three-dimensional model; For any Gaussian distribution in the initial 3D model, the Gaussian distribution is projected onto the pixel surface of the corresponding image frame to obtain the corresponding projection region. The predicted color of the projection region and the pixel loss of the predicted color compared to the actual pixel color in the corresponding image frame are determined. The total loss function is determined based on the mean pixel loss of the image frame. The position, shape, and color parameters of the Gaussian distribution in the initial 3D model are iteratively updated, and the resulting optimized Gaussian distribution set is used as the 3D model of the storage space.

2. The method of claim 1, wherein, The step of determining the global pose sequence of the robot during the mapping data acquisition process, based on the mapping data of the warehouse space collected by the robot, includes: The initial pose sequence of the robot during the mapping data acquisition process is determined based on the image data; Extract feature information from at least a portion of the image frames in the image data; Based on the feature information, loop closure detection is performed in the image data to obtain a closed-loop frame; The initial pose sequence is optimized based on the closed-loop frame to obtain the global pose sequence.

3. The method of claim 1, wherein, The mapping data also includes other modal data collected by the robot through the detection module; The step of determining the global pose sequence of the robot during the mapping data acquisition process, based on the mapping data of the warehouse space collected by the robot, includes: Establish the correlation between the image data and the other modal data to obtain multi-frame fused data; wherein, the other modal data includes at least one of the following: inertial measurement data, wheel speed data, and point cloud data; The global pose sequence is determined based on the specified frame fusion data in the multi-frame fusion data.

4. The method of claim 3, wherein, Determining the global pose sequence based on specified frame fusion data from the multi-frame fusion data includes: Based on the first modal data and the second modal data in the specified frame fusion data, determine the initial pose sequence of the specified frame fusion data; Using the image data and the third modal data in the specified frame fusion data, a closed-loop frame is determined in the specified frame fusion data; Construct a pose factor map based on the specified frame fusion data, the initial pose sequence of the specified frame fusion data, and the closed-loop frame; The initial pose sequence of the specified frame fusion data is optimized based on the pose factor map to obtain the global pose sequence. Wherein, the first modal data is inertial measurement data; and / or, the second modal data is wheel speed data; and / or, the third modal data is point cloud data.

5. The method according to any one of claims 1-4, characterized in that, The step of mapping the semantic information of each pixel to a 3D model of the storage space to obtain a Gaussian distribution set with semantic attributes, and generating a 3D semantic map of the storage space based on the Gaussian distribution set with semantic attributes, includes: The Gaussian distribution in the 3D model is back-projected onto the corresponding image frame in the image data to determine the pixel region covered by each Gaussian distribution; Based on the semantic information of each pixel in the pixel region covered by each Gaussian distribution, the semantic attributes of each Gaussian distribution are determined to obtain a set of Gaussian distributions with semantic attributes. A three-dimensional semantic map of the warehouse space is generated based on the set of Gaussian distributions with semantic attributes.

6. A robot scheduling method, characterized in that, include: Based on the robot's task to be performed and the three-dimensional semantic map of the storage space, the task topology path of the robot is determined; wherein, the three-dimensional semantic map of the storage space is a three-dimensional semantic map generated by the map construction method according to any one of claims 1-5; Extract node matching data corresponding to at least one key node in the task topology path from the three-dimensional semantic map; wherein, the key node includes key decision points in the process of the robot moving to the task target area of ​​the task to be performed, and the node matching data includes the semantic features of at least one semantic marker of the location of the key node; Acquire environmental images at key nodes as the robot travels along the task topology path; The robot is scheduled based on the environmental image and the node matching data corresponding to the key nodes.

7. The method according to claim 6, characterized in that, The three-dimensional semantic map based on the robot's task to be performed and the storage space determines the robot's task topology path, including: The traversable area of ​​the robot is determined in the three-dimensional semantic map; Starting from the robot's current position and ending at the target area of ​​the task to be performed, the topological path of the task is determined based on a path planning algorithm within the passable area.

8. The method according to claim 6, characterized in that, The step of scheduling the robot based on the environmental image and the node matching data corresponding to the key nodes includes: Extract target pixel regions from the environmental image that match the semantic category of semantic markers in the node matching data; Based on the target pixel region and the node matching data, the robot's deviation from the task topology path is determined, and the robot's movement is scheduled according to the deviation.

9. The method according to claim 8, characterized in that, The step of determining the robot's deviation from the task topology path based on the target pixel region and the node matching data, and scheduling the robot's movement based on the deviation, includes: The matching degree between the target pixel region and the semantic marker is determined based on the node matching data; If the matching degree is greater than or equal to a preset threshold, it is determined that the robot has not deviated from the task topology path, and the robot is controlled to continue traveling along the task topology path; If the matching degree is less than a preset threshold, it is determined that the robot has deviated from the task topology path. Based on the environmental image and the three-dimensional semantic map of the warehouse space, the actual pose of the robot in the warehouse space is determined, and the task topology path is replanned for the robot according to the actual pose.

10. The method according to claim 9, characterized in that, The node matching data also includes 3DGS information of at least one semantic marker of the location of the key node, and controlling the robot to continue traveling along the task topology path further includes: Based on the 3DGS information of the semantic marker and the current pose of the robot at the key node, the semantic marker is projected onto the plane where the target pixel region is located to obtain the ideal projection feature corresponding to the semantic marker; Calculate the projection error between the ideal projection feature and the actual image feature of the target pixel region; The projection error is used as a loss function to iteratively optimize the pose parameters and determine the corrected pose of the robot at the key node. The corrected pose is sent to the robot, which is then controlled to update its current pose using the corrected pose and continue driving based on the updated pose and the task topology path.

11. The method according to claim 6, characterized in that, Also includes: In response to receiving a relocation request from the robot, the current environmental image collected by the robot is acquired; Based on the three-dimensional semantic map and the current environment image, determine at least one candidate region that matches the current environment image; Based on the three-dimensional semantic map and the at least one candidate region, at least one candidate pose of the robot is determined; The robot's repositioning pose is determined from the at least one candidate pose.

12. The method according to claim 11, characterized in that, The step of determining at least one candidate region matching the current environment image based on the three-dimensional semantic map and the current environment image includes: Feature extraction is performed on the current environment image to obtain a local feature set of the current environment image; Based on the local feature set and the feature set of the mapping image frames of the three-dimensional semantic map, at least one mapping image frame that matches the local feature set is determined. At least one candidate region matching the current environment image is determined based on at least one mapped image frame.

13. The method according to claim 11, characterized in that, Determining at least one candidate pose of the robot based on the three-dimensional semantic map and the at least one candidate region includes: Based on the three-dimensional semantic map, extract the 3DGS model and mapping pose corresponding to each candidate region; Based on the 3DGS model and mapping pose corresponding to each candidate region, a virtual rendering image corresponding to each candidate region is generated. Based on the difference between the virtual rendering image corresponding to each candidate region and the current environment image, the mapping pose of each candidate region is optimized to obtain at least one candidate pose.

14. The method according to claim 13, characterized in that, The step of optimizing the mapping pose of each candidate region based on the difference between the virtual rendered image corresponding to each candidate region and the current environment image to obtain at least one candidate pose includes: For any candidate region, the photometric loss and structural similarity loss between the virtual rendered image and the current environment image are used as image difference loss to construct a loss function. The pose parameters of the mapping pose are iteratively optimized through the backpropagation algorithm to obtain the candidate pose. Determining the robot's repositioning pose from the at least one candidate pose includes: Determine the image difference loss corresponding to each of the candidate poses; The candidate pose with the smallest corresponding image difference loss is determined as the relocalization pose.

15. A map building apparatus, characterized in that, The device includes: The first determining module is used to determine the global pose sequence of the robot during the mapping data acquisition process based on the mapping data of the warehouse space collected by the robot; wherein, the mapping data includes at least the image data collected by the robot through the detection module; A model building module is used to determine a three-dimensional model of the storage space based on the global pose sequence and the mapping data; wherein, the three-dimensional model of the storage space includes a set of Gaussian distributions constructed by three-dimensional Gaussian sputtering; The second determining module is used to determine the semantic information of each pixel in the image data; The map building module is used to map the semantic information of each pixel to the three-dimensional model of the warehouse space to obtain a Gaussian distribution set with semantic attributes, and to generate a three-dimensional semantic map of the warehouse space based on the Gaussian distribution set with semantic attributes. The model building module is specifically used to construct the 3D model boundary of the storage space based on the global pose sequence; initialize a Gaussian distribution in the 3D model boundary to obtain an initial 3D model; for any Gaussian distribution in the initial 3D model, project the Gaussian distribution onto the pixel surface of the corresponding image frame to obtain the corresponding projection region, determine the predicted color of the projection region and the pixel loss of the predicted color compared to the actual pixel color in the corresponding image frame; determine the total loss function based on the mean pixel loss of the image frame, iteratively update the position parameters, shape parameters and color parameters of the Gaussian distribution in the initial 3D model, and use the obtained optimized Gaussian distribution set as the 3D model of the storage space.

16. A robot scheduling device, characterized in that, The device includes: A path generation module is used to determine the task topology path of the robot based on the robot's task to be performed and a three-dimensional semantic map of the storage space; wherein the three-dimensional semantic map of the storage space is a three-dimensional semantic map generated using the map building device of claim 15. The data extraction module is used to extract node matching data corresponding to at least one key node in the task topology path from the three-dimensional semantic map; wherein, the key node includes key decision points in the process of the robot moving to the task target area of ​​the task to be performed, and the node matching data includes the semantic features of at least one semantic marker of the location of the key node; The image acquisition module is used to acquire environmental images collected at key nodes when the robot travels along the task topology path; The driving scheduling module is used to schedule the robot based on the environmental image and the node matching data corresponding to the key nodes.

17. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-14.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-14.

19. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-14.

Images