Robot automatic navigation method and system based on depth vision fusion

Abstract

The invention discloses a robot automatic navigation method and system based on depth vision fusion, and relates to the technical field of intelligent navigation, and the method comprises the steps: carrying out the registration of a point cloud subset of continuous frames through employing an ICP algorithm, distributing weights for a static obstacle and a dynamic obstacle, and carrying out the fusion, and obtaining a fused environment perception data set. The method comprises the following steps: updating a dynamic environment in real time, generating a dynamic environment map, carrying out trajectory tracking on moving targets by using binocular vision equipment, calculating a real-time motion trajectory of each moving target, obtaining an environment hotspot map, generating a global navigation path for a robot by using an A * algorithm, planning a detouring path at the periphery of a hotspot area, and outputting a navigation path sequence. According to the method, intelligent optimization of the motion path of the robot is realized through path planning and an obstacle avoidance strategy, the safety is ensured, the path efficiency is maximized, and unnecessary detour and pause time is reduced.

Description

Technical Field

[0001] This invention relates to the field of intelligent navigation technology, and in particular to a robot automatic navigation method and system based on deep vision fusion. Background Technology

[0002] In recent years, robot autonomous navigation technology has been widely applied and developed in various scenarios such as industry and logistics. Previous navigation methods mainly relied on a single sensor to achieve path planning by constructing a static environment map. However, with the advancement of deep learning and computer vision technologies, machine vision-based navigation methods have gradually emerged, especially the application of RGB-D cameras combined with depth information, which enables robots to perceive the three-dimensional structure of their environment.

[0003] Existing robot autonomous navigation methods still have unresolved issues. For example, the real-time performance of semantic segmentation and point cloud matching is insufficient. Ordinary navigation methods often rely on computationally complex registration algorithms, making it difficult to achieve low-latency processing on embedded platforms and limiting the robot's responsiveness in variable environments. Furthermore, regarding the prediction of group behavior and obstacle avoidance, existing navigation methods only target single obstacle detection and lack systematic modeling and planning of crowd flow trends, easily leading to path congestion or obstacle avoidance failures. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a robot automatic navigation method based on deep vision fusion, which solves the problems of complex semantic segmentation calculation and limited group obstacle avoidance ability in existing robot automatic navigation methods.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a robot automatic navigation method based on deep vision fusion, which includes collecting environmental data and preprocessing it to obtain RGB-D image frames;

[0008] Semantic segmentation is performed on RGB-D image frames, dividing them into static backgrounds and dynamic objects. The dynamic object regions are then matched with the RGB-D image frames, and the dynamic object mask and point cloud subset are output.

[0009] The ICP algorithm is used to register a subset of point clouds in consecutive frames. At the same time, weights are assigned to static and dynamic obstacles and then fused to obtain a fused environmental perception dataset.

[0010] The system updates the dynamic environment in real time, generates a dynamic environment map, and uses a binocular vision device to track the trajectory of moving targets, calculates the real-time motion trajectory of each moving target, and obtains an environmental heat map.

[0011] The A* algorithm is used to generate a global navigation path for the robot, and detour paths are planned around hotspot areas, outputting a sequence of navigation paths.

[0012] As a preferred embodiment of the robot automatic navigation method based on deep vision fusion described in this invention, the environmental data includes RGB images and LiDAR point cloud data;

[0013] The preprocessing includes noise reduction, normalization, and cubic mesh generation.

[0014] As a preferred embodiment of the robot automatic navigation method based on deep vision fusion described in this invention, the output dynamic object mask includes selecting MobileNetV2 as the core architecture, constructing a DeepLabv3 network, inputting RGB-D image frames, obtaining a semantic segmentation mask, and converting it into a three-dimensional spatial representation to form a dynamic object mask.

[0015] The dynamic object mask and the lidar point cloud data are matched to obtain a subset of the point cloud.

[0016] As a preferred embodiment of the robot automatic navigation method based on deep vision fusion described in this invention, the registration of point cloud subsets of consecutive frames includes capturing changes in point cloud subsets and organizing them into two-frame point cloud data pairs.

[0017] The ICP algorithm is used to register each pair of two-frame point cloud data to form matching point pairs and determine the movement mode of the matching point pairs, thus obtaining a subset of continuous frame point clouds after registration.

[0018] As a preferred embodiment of the robot automatic navigation method based on deep vision fusion described in this invention, the fused environmental perception dataset refers to a dynamic obstacle model constructed using a subset of registered continuous frame point clouds.

[0019] By using a dynamic obstacle model, the registered point cloud subset and visual information are fused to generate a fused environmental perception dataset.

[0020] The visual information refers to the depth and image information provided by the RGB-D camera.

[0021] As a preferred embodiment of the robot automatic navigation method based on deep vision fusion described in this invention, the step of obtaining the environmental heat map includes creating a two-dimensional grid structure to improve the dynamic environment and generating an initial dynamic environment map, wherein the two-dimensional grid structure includes a static layer and a dynamic layer;

[0022] By processing static and dynamic layers, the initialized dynamic environment map is updated to obtain the updated dynamic environment map;

[0023] Using a binocular camera, the trajectory of moving targets in the updated dynamic environment map is tracked, matching points are found in the left and right images, and the depth of each matching point is calculated to generate a depth distribution map.

[0024] Locate moving targets based on the depth distribution map, assign an ID to each moving target, find the region corresponding to the ID of the moving target from the depth distribution map, obtain the preliminary motion trajectory of each moving target, and classify the moving targets into a set of motion parameters according to their IDs;

[0025] Based on the set of motion parameters of the moving target, the motion rules of the moving target are defined and the driving force of the target is calculated to obtain the real-time motion trajectory of the moving target;

[0026] The two-dimensional grid structure is divided into multiple smaller grids. The time points of the real-time motion trajectory of the moving target are checked and recorded in the corresponding smaller grids to form a distribution table.

[0027] Check the number of moving targets in the distribution table and mark them to form a marking result. Add the marking result to the updated dynamic environment map to form an environment hotspot map.

[0028] As a preferred embodiment of the robot automatic navigation method based on deep vision fusion described in this invention, the output navigation path sequence refers to the command robot to generate a global navigation path for the robot using the A* algorithm, with the current position as the starting point and the set target point as the ending point.

[0029] Check the coordinates on the global navigation path to determine the robot's detour direction, output the global navigation path coordinate sequence, predict and constrain the dynamic threats existing in the global navigation path coordinate sequence, find the conflict point and adjust the detour path to obtain the path sequence with dynamic obstacle avoidance constraints.

[0030] The path sequence under dynamic obstacle avoidance constraints is smoothed and optimized to generate a navigation path sequence.

[0031] Secondly, the present invention provides a robot automatic navigation system based on deep vision fusion, including a data acquisition module for acquiring environmental data and preprocessing it to obtain RGB-D image frames;

[0032] The segmentation and matching module performs semantic segmentation on RGB-D image frames, dividing them into static backgrounds and dynamic objects, and matching the dynamic object regions with the RGB-D image frames, outputting dynamic object masks and point cloud subsets.

[0033] The fusion module uses the ICP algorithm to register a subset of point clouds from consecutive frames, and assigns weights to static and dynamic obstacles and fuses them to obtain a fused environmental perception dataset.

[0034] The heat map generation module updates the dynamic environment in real time, generates a dynamic environment map, and uses a binocular vision device to track the trajectory of moving targets, calculates the real-time motion trajectory of each moving target, and obtains an environmental heat map.

[0035] The path output module uses the A* algorithm to generate a global navigation path for the robot and plans detour paths around hotspot areas, outputting a navigation path sequence.

[0036] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein the computer program, when executed by the processor, implements any step of the robot automatic navigation method based on depth vision fusion as described in the first aspect of the present invention.

[0037] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the robot automatic navigation method based on depth vision fusion as described in the first aspect of the present invention.

[0038] The beneficial effects of this invention are as follows: By combining multi-scale feature extraction and dilated convolution techniques, this invention optimizes the semantic segmentation process of RGB-D image frames, improving the real-time performance of semantic segmentation and overcoming the computational bottleneck of traditional high-complexity segmentation algorithms. Furthermore, the path planning and obstacle avoidance strategies enable intelligent optimization of the robot's motion path. This process not only considers static obstacles but also dynamically adjusts to avoid moving obstacles. While ensuring safety, path efficiency is maximized, reducing unnecessary detours and pauses, and improving the efficiency of robot task execution, significantly reducing energy consumption and time costs. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart of a robot automatic navigation method based on deep vision fusion.

[0041] Figure 2This is a system architecture diagram for robot autonomous navigation based on deep vision fusion.

[0042] Figure 3 This is a schematic diagram of dynamic obstacle detection and handling.

[0043] Figure 4 This is a schematic diagram of global path planning and hotspot avoidance. Detailed Implementation

[0044] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0045] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0046] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0047] Reference Figures 1-4 This embodiment provides a robot automatic navigation method based on deep vision fusion, including the following steps:

[0048] S1. Collect environmental data and preprocess it to obtain RGB-D image frames.

[0049] Includes the following steps,

[0050] S1.1 The robot's surrounding environment data is collected in real time by equipping it with an RGB-D camera and a LiDAR sensor. Specifically, the RGB-D camera is set to a fixed resolution of 1280×720 for both color and depth images, with a depth detection range limited to 0.5 meters to 10 meters to cover the robot's near and medium-range perception areas. The LiDAR sensor is a Velodyne VLP-16 model, generating 360-degree panoramic LiDAR point cloud data at a scanning frequency of approximately 10 times per second, with each degree of LiDAR point cloud data containing 200 points.

[0051] S1.2 After environmental data acquisition, denoising processing of the RGB images is required. A Gaussian filtering method (with a filter window size of 5×5 pixels in the example) is used to perform a weighted average of the neighborhood surrounding each pixel in the RGB image. This smooths out noise while preserving edge details in the RGB image as much as possible. For the depth image, preprocessing is performed through normalization, adjusting the original depth values ​​to the range of 0 to 1. Specifically, each original depth value is linearly scaled according to the depth range of the depth image (i.e., minimum 0.5 meters, maximum 10 meters), and the adjusted depth image is stored as a floating-point number. For the LiDAR point cloud data, a voxel filtering method is used to divide the LiDAR point cloud data into small cubic grids with fixed side lengths, such as 0.05 meters. For points within each small cubic grid, the average position of all points is calculated, and only the average position point is retained to represent the original point set. This significantly reduces the amount of point cloud data while maintaining the spatial structure of the LiDAR point cloud without distortion.

[0052] S2. Perform semantic segmentation on the RGB-D image frame, dividing the RGB-D image frame into static background and dynamic object, and matching the dynamic object region with the RGB-D image frame to output the dynamic object mask and point cloud subset.

[0053] Includes the following steps,

[0054] S2.1 The COCO dataset was selected. The COCO dataset is a publicly available dataset released by Microsoft, specifically sourced from a large number of everyday images obtained from the Flickr image sharing platform. It contains a standard dataset of 80 common object categories (such as people, vehicles, animals, etc.). Additionally, actual robot navigation environment data (such as warehouses, factories, etc., including RGB images and labeled masks) was collected, totaling 5000 images, which were divided into a training set (80%), a validation set (10%), and a test set (10%).

[0055] MobileNetV2 was chosen as the backbone architecture to build the DeepLabv3 network. Specifically, the decoder part of the DeepLabv3 network is first initialized with random weights, the loss function is cross-entropy loss, double weights are assigned to address class imbalance issues (i.e., dynamic object classes, such as pedestrians, forklifts, etc.), the optimizer is Adam, the initial learning rate is set to 0.001, and the batch size is set to 8.

[0056] The configured DeepLabv3 network was trained on an NVIDIA RTX 3090 GPU for a total of 50 epochs. In each epoch, the training set was iterated, and the network weights were updated by calculating the loss between the predicted and ground truth labeled masks (the learning rate decayed to one-tenth of its original value every 20 epochs). After the update, the pixel accuracy and mean intersection-over-union (mIoU) were evaluated every 5 epochs on the validation set. The network with the highest mIoU was selected as the best DeepLabv3 network (tested on the test set; the testing criteria were that the mIoU for dynamic objects should be greater than 0.8, and the mIoU for static backgrounds should be greater than 0.9).

[0057] S2.2. Input the RGB-D image frame into the DeepLabv3 network to perform semantic segmentation. The goal of segmentation is to classify the pixels of the RGB-D image frame one by one into two categories: static background and dynamic objects (static background includes the ground, walls, and fixed equipment, labeled as category 0; dynamic objects include pedestrians, forklifts, and other moving targets, labeled as category 1). Specifically, DeepLabv3 starts from the input RGB-D image frame and first extracts features layer by layer through the MobileNetV2 backbone network. Feature extraction is divided into multiple stages: the initial convolutional layer performs shallow processing on the RGB-D image frame to extract low-level features (such as edges and colors), and then depthwise separable convolutions are used to gradually deepen the network layers to capture more complex texture and shape information. The resolution of the output shallow feature maps (i.e. basic visual information, such as one shallow feature map highlighting the outline of a pedestrian, and another emphasizing the color distribution of a wall), medium feature maps (reflecting the local structure of the object, such as the shape of a pedestrian's arm), and deep feature maps (such as distinguishing the overall category between a pedestrian and a wall) will gradually decrease, for example from 512×512 to 256×256, then to 128×128, until the lowest resolution.

[0058] To generate pixel-level classification probability maps, multi-scale feature extraction and dilated convolution techniques are required. Specifically, multi-scale feature extraction is achieved by processing feature maps at different levels in parallel: feature maps from the intermediate layer (256×256) and deep layer (32×32) of the MobileNetV2 backbone network are extracted, representing medium-scale and large-scale information, respectively. Medium-scale features preserve local details (such as a pedestrian's arm or the outline of a forklift), while large-scale features capture global context (such as the overall layout of objects and background). Dilated convolution, on the other hand, expands the receptive field without reducing resolution by inserting holes (with a spacing of 2 or 4) into the deep feature maps' kernels. For example, a standard 3×3 convolution kernel has a receptive field of 3×3 pixels; after inserting holes, it can be expanded to 7×7 or 15×15 pixels, thus perceiving a wider range of contextual information (such as determining whether a pedestrian is in a crowd). The extracted multi-scale feature maps are uniformly adjusted back to the input resolution of 512×512 through an upsampling operation (i.e., bilinear interpolation), and then stitched together to form a comprehensive feature map.

[0059] The comprehensive feature map is output as a classification probability map. Specifically, the DeepLabv3 network's decoder uses 1×1 convolutions and performs feature dimensionality reduction on each pixel, reducing the number of output channels to 2 (corresponding to category 0 and category 1). For each pixel location, the probability value of each pixel location belonging to static background or dynamic object is calculated, with the probability ranging from 0 to 1. The calculation process involves using a 1×1 convolution kernel to scan the comprehensive feature map pixel by pixel, linearly combining the pixel features at each pixel location to generate a 2D output vector. This 2D output vector represents the preliminary score (also called logits) of the pixel belonging to category 0 and category 1. For example, for a pixel location (x, y), the pixel features are transformed into a 2D output vector [s0, s1] through a 1×1 convolution, where s0 is the score for category 0 and s1 is the score for category 1. For each 2D output vector, the exponent value of each score, i.e., exp(s0) and exp(s1), is calculated, and the sum of the exponent values ​​is calculated, i.e., sum = exp(s0) + exp(s1). Divide each index value by the sum to obtain the normalized probability value, i.e., p0 = exp(s0) ÷ sum, p1 = exp(s1) ÷ sum. For example, if the 2D output vector of a pixel is [2.197, 0.693], then exp(2.197) ≈ 9, exp(0.693) ≈ 2, the sum is 11, and the probabilities are p0 ≈ 9 ÷ 11 ≈ 0.818, p1 ≈ 2 ÷ 11 ≈ 0.182. Therefore, the final probability value for each pixel position belonging to the static background and the dynamic object is [0.818, 0.182]. For example, a pixel might have a probability of [0.9, 0.1], indicating a 90% probability of being a static background. Finally, the category with the highest probability is selected as the label for each pixel; for example, probability [0.9, 0.1] is labeled as category 0, generating a semantic segmentation mask consistent with the input resolution.

[0060] Context enhancement strategies are used to improve the recognition accuracy of semantic segmentation masks. Specifically, for regions in the semantic segmentation mask marked as dynamic objects (i.e., category 1), the category distribution within a 5×5 pixel area surrounding the dynamic object region is examined. If the boundary of a dynamic object is found to be adjacent to a static background (e.g., a pedestrian's feet near the ground), a morphological dilation operation is performed. Dilation uses 3×3 rectangular structuring elements to expand the dynamic object region outward by 1–2 pixels (the specific expansion range is determined by the probability value at the dynamic object boundary: if the probability value of the dynamic object at the boundary pixel is below 0.7, it expands by 2 pixels; if it is between 0.7 and 0.9, it expands by 1 pixel. This operation ensures that small or partially occluded dynamic objects (such as the edge of a forklift) are not misclassified as background, avoiding missed detections).

[0061] Finally, a unique identifier (ID) is assigned to each independent dynamic object. Connected component analysis identifies regions belonging to dynamic objects within the semantic segmentation mask. Specifically, this involves scanning from the top left to the bottom right corner of the semantic segmentation mask, checking if adjacent pixels of each pixel belong to the same category (Class 1). If adjacent, they are grouped into the same connected component, and each connected component is assigned an ID, incrementing from Class 1. For example, if two separate dynamic object regions are detected, they are labeled ID=1 and ID=2 respectively. The boundaries of connected components are determined by checking eight neighboring regions, ensuring that small-scale noise does not affect object segmentation. Finally, a semantic segmentation mask with a category label (0 or 1) and an ID is generated. For example, a region in the semantic segmentation mask might be labeled as Class 1 with ID=1, representing an independent pedestrian.

[0062] S2.3. Using the depth image of the RGB image in the RGB-D image frame, the semantic segmentation mask with category labels and IDs is extended into a three-dimensional spatial representation. Specifically, the normalized value of each pixel in the depth image is restored to the actual depth value. Based on the depth range of the depth image (i.e., 0.5 meters to 10 meters), the normalized value is multiplied by 9.5 and then added by 0.5 to obtain the true depth value. For pixels marked as dynamic objects in the semantic segmentation mask, the corresponding depth value of the dynamic object pixel is extracted (i.e., the actual depth value corresponding to the pixel position marked as a dynamic object in the semantic segmentation mask). Based on the pixel coordinates of the RGB image, such as (u, v), and using the parameters of the depth camera (i.e., focal length F), the depth value is calculated. X F Y and photocentric C X C Y 3D reconstruction is performed to generate the spatial coordinates (x, y, z) of the dynamic object. During reconstruction, in order to reduce the influence of depth noise, the median of the connected component of the depth value of each dynamic object (i.e., the corresponding depth value of the dynamic object pixel in the same ID identification pixel region) is taken as the representative value to generate smooth 3D position information and form a 3D mask of the dynamic object. Each 3D mask of the dynamic object contains the object category, ID and spatial coordinate set.

[0063] S2.4 Match the 3D mask of the dynamic object with the LiDAR point cloud data. The specific matching process is to project the LiDAR point cloud data onto the RGB image plane (for each point cloud point (X... l Y l Z l The image coordinates are transformed to the camera coordinate system using the camera's extrinsic parameters, and then projected to the image coordinate system using the camera's intrinsic parameters. P V PThe projected point cloud coordinates are compared with the dynamic object region in the 3D mask of the dynamic object. If the projected point of the point cloud coordinates falls within the mask of a certain dynamic object, it is marked as a LiDAR point cloud point with the corresponding ID. For each dynamic object, all matching LiDAR point cloud points with the corresponding ID are extracted to form a point cloud subset.

[0064] S3. Based on the dynamic object mask and point cloud subset, the ICP algorithm is used to register the point cloud subset of consecutive frames. At the same time, weights are assigned to static obstacles and dynamic obstacles and fused to obtain the fused environment perception dataset.

[0065] Includes the following steps,

[0066] S3.1 First, two consecutive frames of point cloud data are needed to capture the dynamic changes of the point cloud subset. Specifically, the current frame is the most recently acquired point cloud subset, and the previous frame is the point cloud subset immediately preceding the current frame. To track dynamic objects, the two consecutive frames of data are grouped by ID. For example, an object with ID 1 has a corresponding point cloud subset in both the current and previous frames. If an object does not have point cloud data in the previous frame, such as a pedestrian who has just entered the field of view, it is marked as a new object.

[0067] S3.2. Register each pair of point cloud data frames to find the distance and direction of movement of the dynamic object from the previous frame to the current frame. Specifically, the ICP algorithm (Iterative Closest Point) is used. Registration begins with initialization, assuming the point cloud position in the previous frame is the reference object and the point cloud position in the current frame is the object to be adjusted. Initialization requires that the positions of the two frames completely overlap. After initialization, starting from each point in the current frame, find the point in the previous frame that is closest to the current frame, forming a matching point pair (if a point in the current frame cannot find a sufficiently close matching point, it is ignored).

[0068] After matching, the movement (including rotation and displacement) of the matched point pairs is calculated to make the point cloud of the current frame as close as possible to the position of the previous frame. Specifically, first, the center positions of the point clouds in the two frames of the matched point pair are found (that is, the average of the coordinates of all points in each frame's point cloud is taken to obtain the center of the previous frame and the center of the current frame). The rotation angle of the matched point pairs is adjusted based on the center positions to make the shapes of the two frame point clouds more consistent. Specifically, the difference in the direction of the line connecting each matched point pair between the previous and current frames is calculated, and the directional differences of all matched point pairs are summarized to find a unified rotation method. Then, this rotation method is applied to the point cloud of the current frame, the coordinates of each point are adjusted, and the difference in the center positions of the two frame point clouds is recalculated to obtain a more accurate displacement distance.

[0069] S3.3. Using the distance and direction of movement of dynamic objects from the previous frame to the current frame, estimate the velocity information of each dynamic object. Specifically, the direction of velocity is directly determined by the direction of movement. If a dynamic object moves to the right and forward from the previous frame to the current frame, then the direction of velocity is also to the right and forward, maintaining consistency with the movement. The magnitude of the velocity is calculated by dividing the distance moved by the time interval between the two frames.

[0070] To ensure more stable velocity and avoid sudden changes in speed due to occasional jitter in point cloud data or inaccurate registration, Kalman filtering is used for smoothing. The first step of the filtering is to assume that the dynamic object is initially stationary, i.e., with zero velocity. Whenever a new direction and distance of movement are received, the filter first predicts the current velocity value of the dynamic object based on previous velocity trends (e.g., predicting that the dynamic object will continue to maintain its previous velocity direction and magnitude). Then, the predicted velocity value is compared with the actual measured velocity, and adjustments are made accordingly (if the actual measured velocity fluctuates greatly, the filter tends to predict a higher velocity value to ensure that the velocity does not jump too drastically; if the measured value is stable for more than ten consecutive frames, the filter gradually accepts the measured value to make the velocity closer to reality). For some objects, if registration fails for two consecutive frames, such as when they are obscured by other objects, the filter directly uses the predicted velocity value instead of the actual measured value to ensure that the velocity information is continuous. After smoothing, the velocity of the dynamic object will appear to be more consistent with the object's actual motion (e.g., a pedestrian will show a steady walking speed, rather than suddenly accelerating or stopping).

[0071] S3.4 Next, we need to construct a dynamic obstacle model. The first step is to determine the specific location of the dynamic objects (for each dynamic object's point cloud subset, sum the coordinates of all points and take the average value; the average value represents the core location of the object, such as the center point of a pedestrian). Based on the information from the dynamic object mask (the mask already labels the object's category, such as whether it's a pedestrian or a forklift, and also includes the object's 3D boundary shape), align the registered point cloud subset of the current frame with the dynamic object mask. During alignment, first check whether the area covered by the registered point cloud subset completely matches the shape of the semantic segmentation mask (if the points of the point cloud subset fall exactly within the boundary of the dynamic object mask, use this range directly; if the area covered by the point cloud subset is smaller than the semantic segmentation mask, for example, only the upper body of the pedestrian is scanned, slightly expand the boundary of the point cloud subset outwards to ensure that the shape of the point cloud subset and the dynamic object mask completely overlap, ensuring the object's outline is complete).

[0072] When constructing a dynamic obstacle model, each obstacle records three pieces of information: the specific location of the dynamic object, the smoothed velocity information of the dynamic object, and the object category in the dynamic object mask. If some areas are found not to be covered by a subset of the point cloud, such as when an object is occluded and not detected by the LiDAR, depth images are used to fill in the gaps. (This involves finding points with depth values ​​near the occluded area from the dynamic object mask, calculating the average depth of all points with depth values, and then filling in the missing areas with the average depth value. For example, if the side of a truck is obscured, the depth of the obscured part is inferred from the depth value of the visible part of the truck, and then filling it in to make the dynamic obstacle model look like a complete object.) During the completion process, depth values ​​close to the edge of the occlusion are used as much as possible to avoid imagining the depth of too far away areas.

[0073] S3.5. Utilize a dynamic obstacle model to implement adaptive weight allocation and fusion. Specifically, obstacles are divided into static obstacles (such as walls, fixed equipment, etc.) and dynamic obstacles (such as pedestrians, vehicles). The classification is based on checking the velocity information of each obstacle. If the velocity of each obstacle is consistently below a minimum value (e.g., almost stationary), it is classified as static; otherwise, it is classified as dynamic. For static obstacles, a registered subset of the point cloud is used to assign weights. Specifically, a dominant weight, such as 0.8, is assigned to the registered subset of the point cloud, indicating that positional information is the primary reference. Visual information (depth and image information provided by the RGB-D camera) is assigned a secondary weight, such as 0.2, as an auxiliary reference. The setting of the dominant and secondary weights is based on the performance differences between the LiDAR sensor and the RGB-D camera in static scenes, as well as the actual application scenario.

[0074] After weight allocation, the weight pairs for static obstacles are recorded (i.e., the registered point cloud subset has a weight of 0.8, and the visual information has a weight of 0.2). For dynamic obstacles, during initial allocation, the weight of the visual information is set to 0.6, and the weight of the LiDAR is 0.4 (used to emphasize the motion vector of the visual information). The weights can then be adjusted based on the obstacle's speed. (If the speed increases significantly, such as when an object moves quickly, the visual weight will gradually increase. The adjustment method is to gradually increase the visual weight according to the speed change, with each increase proportional to the speed, and the upper limit is locked at 0.8 to ensure that the registered point cloud subset always retains a certain influence. If the speed decreases, the weight remains unchanged from the initial value.) This weight adjustment process reflects the increasing dependence of the dynamic obstacle's motion characteristics on visual perception.

[0075] S3.6. Based on the weight pairs of each obstacle, the registered point cloud subset and visual information are fused to generate unified environmental perception information. Specifically, for static obstacles, the fusion process primarily relies on the point cloud coordinates of the LiDAR; for example, the 3D coordinates of a wall are directly taken from the average value of the point cloud. Then, fine-tuning is performed using visual information (if there is a slight deviation between the visual depth and the LiDAR position, a weighted average is taken, and the final position is obtained by combining the weight ratio of the registered point cloud subset and the visual information with the weighted average). For dynamic obstacles, the fusion uses the velocity and direction provided by the visual information; that is, the center of the object is first determined using the LiDAR point cloud, and then the boundary details are corrected using visual depth to ensure that the position matches the mask boundary.

[0076] S3.7. Integrate the fused environmental perception information into an environmental perception dataset. During integration, obstacles are arranged in order of ID, and each obstacle is recorded with its category, position, and speed (static obstacles have a speed of zero). For static obstacles, check multi-frame consistency. If the position of a static obstacle changes very little for three consecutive frames, it is identified as a fixed object, and the influence of random noise is eliminated. For dynamic obstacles, verify the matching between speed and object category. For example, if the speed of a pedestrian is too high and exceeds a reasonable range, adjust it to the upper limit of a typical pedestrian walking speed. After optimization, the environmental perception dataset is divided into two layers: the static layer records the precise 3D coordinates of walls, equipment, etc., and the dynamic layer records the trajectory information (i.e., position and speed) of pedestrians, forklifts, etc.

[0077] S4. Using the fused environmental perception dataset, the dynamic environment is updated in real time, a dynamic environment map is generated, and a binocular vision device is used to track the trajectory of moving targets. The real-time motion trajectory of each moving target is calculated to obtain an environmental heat map.

[0078] Includes the following steps,

[0079] S4.1. Improve the dynamic environment based on the environmental perception dataset to obtain an initialized dynamic environment map. Specifically, create a two-dimensional grid structure with a resolution of 0.1m × 0.1m, covering the robot's perception range (assumed to be 20m × 20m). The grid is divided into two layers: a static layer for recording fixed obstacles and a dynamic layer for recording moving obstacles. In the initial state, the static layer extracts all obstacles marked as static from the input environmental perception dataset, directly fills in the position coordinates of the static obstacles, and marks them as immovable areas. The dynamic layer, on the other hand, fills in the initial position and velocity information of all dynamic obstacles in the current frame and marks them as variable areas.

[0080] S4.2 Update the initialized dynamic environment map to obtain the updated dynamic environment map. Specifically, first, the static layer needs to be processed. The position of static obstacles is read and compared with the static layer of the previous moment. If the position is the same, it remains unchanged. If a new static obstacle is added (e.g., a newly detected fixed device), the position coordinates of the new obstacle are added to the corresponding cell of the grid and marked as occupied. Then, the dynamic layer is updated. The position and velocity of the dynamic obstacles in the current frame are read and overwritten with the dynamic layer of the previous moment. For example, if a pedestrian's position moves from grid (5,7) to (6,5), the old record of (5,7) is cleared, and then the new position and velocity direction are written to (6,5). If a dynamic obstacle is outside the perception range, such as leaving the 20-meter boundary, the record is removed from the dynamic layer.

[0081] S4.3. Use binocular vision equipment to track the trajectory of moving targets (within a 20-meter range) in the updated dynamic environment map. Specifically, a ZED2 binocular camera (resolution set to 1280×720) is used to capture stereo images from both the left and right sides (the left and right lenses of the binocular camera simultaneously capture the same scene, generating two images with slightly different perspectives, which can be understood as similar to the stereoscopic vision effect of the human eye). First, find matching points in the left and right images. The search for matching points starts from the left image. Select a random pixel in the left image, such as a pixel located on a pedestrian's head, and then search for the corresponding feature point in the right image. During the search, check the pixels in the right image along the same horizontal line (the binocular camera lenses are horizontally aligned), comparing them one by one from left to right. During the comparison, check the color and texture features of a small area centered on a pixel in the left image (e.g., a 5×5 pixel range). If the color distribution and texture details of a certain area in the right image are highly similar to those in the left image, the two points are considered to be a successful match (for example, if the red hat on a pedestrian's head appears in a similar position in both images and the surrounding pixel color changes are consistent, it is confirmed as a pair of matching points). To improve accuracy, if the features of a certain area are not obvious (such as a solid color background), the search range is expanded to 7×7 pixels until a reliable matching point is found or the matching point is abandoned.

[0082] S4.4 After finding matching points in the left and right images, calculate the depth of each matching point. Since the left and right lenses of a stereo camera have a fixed distance (baseline), the pixel position of the same object in the two images will differ depending on the distance. The closer the distance, the greater the difference in the horizontal position of the matching point in the left and right images; the farther the distance, the smaller the difference. Therefore, during calculation, first record the horizontal coordinates of the matching point in the left image and the horizontal coordinates in the right image, and find the pixel difference. For example, the pixel of a pedestrian's head is located in column 600 in the left image and column 580 in the right image, with a difference of 20 pixels. This pixel difference reflects the distance between the object and the camera; that is, the larger the difference, the closer the object is to the camera.

[0083] Next, by calibrating the camera parameters of the binocular cameras (including lens spacing and focal length), the pixel differences are converted into actual 3D depth values. For example, a difference of 20 pixels may correspond to a depth of 2 meters, while a difference of 10 pixels corresponds to a depth of 4 meters. After the calculation is completed, the depth value of each matching point is recorded in the depth map, forming a depth distribution map with the same resolution as the image pixels.

[0084] S4.5. Locate moving targets from the dynamic environment map based on the depth distribution map and the speed of dynamic obstacles, and output the preliminary motion trajectory (i.e., 3D coordinate sequence) of each moving target. Specifically, during localization, assign a unique ID to each moving target. For example, ID 1 represents a pedestrian. Extract the pedestrian's position from the dynamic layer in the previous frame (e.g., (5,2,0.1)). Then, using the depth distribution map and stereo image of the current frame, update the real-time position of the moving target. That is, find the area corresponding to the moving target ID from the depth distribution map, and calculate the average depth and coordinates of all points in the corresponding area (e.g., if the average depth value of the pedestrian area is concentrated around 2 meters, then the coordinates are updated to (5.5,2,0.1)).

[0085] S4.6. The initial movement trajectory of each moving target is categorized according to its ID. Each trajectory records its position points over a consecutive 5-second period, forming a time series. From this time series, movement direction, speed, and group density are selected as core parameters. Movement direction is determined by comparing the position of the moving target in two consecutive frames; for example, from (5,2,0.1) to (5.5,2,0.1), the direction is forward. Speed ​​is calculated based on position changes and time intervals; for example, if a moving target moves 0.5 meters between two frames with a time interval of 66.7 milliseconds, the resulting speed is the average. Group density is calculated by statistically analyzing the number and distribution of targets within a 20-meter range (dividing the 20-meter area into 1-meter × 1-meter grids and counting the number of targets in each grid; for example, if a grid contains 3 people, the density is recorded as 3 people / square meter).

[0086] S4.7. Define the motion rules for each moving target based on its set of motion parameters. Assume the moving target is a pedestrian. The pedestrian's movement is influenced by two main factors: one is the driving force propelling itself towards the target, and the other is the pushing effect, i.e., the repulsive force, generated by surrounding crowds or obstacles.

[0087] The driving force of a moving target is calculated based on its motion rules. Specifically, taking a pedestrian as an example, the pedestrian moves continuously in the current direction at a stable speed. Taking the pedestrian's current position as the starting point, a straight path is drawn along the direction of movement from the current position. For example, if the pedestrian is currently at (5,2,0.1), moving straight ahead at a speed of 1 meter per second, then the predicted path for this pedestrian over the next 3 seconds is to start from (5,2,0.1), move forward 1 meter per second, reaching (6,2,0.1) in the first second, (7,2,0.1) in the second second, (8,2,0.1) in the third second, and so on.

[0088] S4.8. Identify congested areas forming in the next few seconds based on real-time motion trajectories and generate an environmental heat map. Divide a 20m x 20m area into 400 1m x 1m grids. Then analyze the real-time motion trajectory of each target and count which grids the real-time motion trajectories of all targets will appear in in the next few seconds. The specific statistical method is to check the time coordinates of the real-time motion trajectory of each target and see which grid the real-time motion trajectory of each target falls into. For example, if a pedestrian's trajectory point (5.7, 2.3, 0.1) falls into grid (5, 2) at 1 second, increment the count of that grid by 1. Traverse the trajectories of all targets and record the landing point of each time point into the corresponding grid to form a distribution table.

[0089] The distribution table is used to check the number of targets in each grid at six time points (0.5 seconds to 3 seconds) over the next 3 seconds. For example, grid (5,2) has 1 person at 0.5 seconds, 3 people at 1 second, 4 people at 1.5 seconds, and 3 people at 2 seconds. If the number of targets exceeds 2 at any given time point, the grid is considered to be congested. For example, if grid (5,2) has 4 people at 1.5 seconds, it is marked as a high-density area. During marking, if a grid has more than 2 people at multiple time points, it is marked with higher priority; if it only exceeds 2 at a single moment, it is checked whether surrounding grids also have a similar trend to avoid isolated misjudgments. After marking, all high-density grids are colored red, indicating that these high-density grids need to be avoided in the next 3 seconds.

[0090] S4.9 Overlay the marking results onto the dynamic environment map to form an environmental heat map containing congested area annotations. The overlay process first checks the relationship between the red areas and the static layer. That is, when a red high-density grid overlaps with a wall or fixed equipment, extend 0.5 meters outwards along the outer edge of the grid to find the nearest blank area and re-mark it, ensuring the avoidance zone is passable. For example, if grid (5,5) is red but occupied by a wall, move the marker to the adjacent (6,5). After adjustment, the heat map retains the fixed obstacle positions of the static layer (assumed to be marked in gray) and the moving target positions of the dynamic layer (default color), and adds red avoidance zones, forming a three-layer visualization structure.

[0091] S5. Based on the environmental heat map, use the A* algorithm to generate a global navigation path for the robot, plan detour paths around the hotspot areas, and output the navigation path sequence.

[0092] Includes the following steps,

[0093] S5.1. Based on the environmental heat map, output a global navigation path. Specifically, instruct the robot to start from its current position and end at a preset target point (e.g., the warehouse exit 20 meters away), generating a global navigation path (i.e., a 3D grid coordinate sequence) using the A* algorithm. During planning, each grid within a 20m x 20m area is marked as either passable or impassable (grids containing static obstacles and red avoidance zones are marked as impassable, while the remaining grids are passable). After planning, the A* algorithm starts from the starting point and explores the passable area grid by grid, prioritizing the grid closest to the target in a straight line while avoiding all impassable areas. For example, if the starting point is at (0,0) and the target is at (20,0), the algorithm will attempt to move along the X-axis, but if (5,0) to (7,0) is a red avoidance zone, it may turn to (5,1) to detour.

[0094] S5.2. Based on the global navigation path, plan detour routes for the red avoidance zone (i.e., hotspot area). Specifically, first check each coordinate on the global navigation path. If a point falls into the red grid or is less than 2 meters from the boundary of the red grid (i.e., the minimum safe distance), it is marked as a detour point. For example, if the global navigation path passes through (6,0), and (6,0) to (7,0) is the red avoidance zone, then (6,0) is a detour point.

[0095] The detour direction is determined based on the detour point. Using the red avoidance zone as the center, examine the passable grid above, below, left, and right of the red avoidance zone, and select the side closest to the subsequent direction of the global navigation path (e.g., if the initial global path is to the right, prioritize detouring upwards or downwards). The detour path is designed as an ellipse, extending 2 meters outwards from the detour point to the safety boundary. Draw an ellipse centered on the red avoidance zone, with the major axis along the length of the red avoidance zone and the minor axis at least 2 meters from the safety boundary. For example, if the red avoidance zone is a 2m x 1m rectangle, set the major axis of the ellipse to 4 meters and the minor axis to 3 meters.

[0096] S5.3 Predict and constrain dynamic threats present in the global navigation path coordinate sequence. Specifically, take the velocity information of each moving target (taking a pedestrian as an example) in the dynamic layer and calculate the pedestrian's position in the next second or several seconds. For example, if a pedestrian is at (5,2,0.1) with a velocity of (1,0,0), then the predicted position after 1 second is (6,2,0.1). Compare the predicted position with the global navigation path coordinate sequence. If a point in the global navigation path coordinate sequence coincides with the predicted position or the distance is less than 2 meters, it is marked as a conflict point.

[0097] Start planning the detour 2 meters before the conflict point, check the passable areas to the left and right of the conflict point, and choose the side that is consistent with the global target direction. For example, if the conflict point is at (6,2) and there is an empty space to the left of the conflict point, the path is adjusted to (5.8,2.2) and (5.6,2.4) to bypass (6,2). (When adjusting, calculate the arrival time from the robot's current position to the conflict point. If it is less than 0.5 seconds, then detour further in advance.)

[0098] S5.4. Smoothly optimize the path coordinate sequence of dynamic obstacle avoidance constraints to generate the final navigation path sequence. Specifically, traverse all coordinate points in the path coordinate sequence of dynamic obstacle avoidance constraints. If the turning angle between two adjacent coordinate points is too large (e.g., a right-angle turn), insert intermediate points for a smooth transition. For example, from (5,2) to (5,3) and then to (6,3), insert (5.2,2.8) and (5.5,3) at (5,3) to smooth the curve. Then adjust the spacing of all coordinate points, evenly dividing the path coordinate sequence of dynamic obstacle avoidance constraints into coordinate points at fixed intervals (e.g., 0.1 meters). If the original spacing of coordinate points is too large, supplement intermediate points through linear interpolation; if the spacing is too small, merge adjacent points.

[0099] After adjusting the spacing between coordinate points, check the distance between each coordinate point and the red avoidance zone (it needs to be greater than 2 meters; if it does not meet this requirement, make minor adjustments). The final navigation path is output in the form of a coordinate sequence, such as {(0,0),(0.1,0),(0.2,0.1),…}, with each point spaced 0.1 meters apart.

[0100] This embodiment also provides a robot automatic navigation system based on deep vision fusion, including: a data acquisition module, which acquires environmental data and preprocesses it to obtain RGB-D image frames;

[0101] The segmentation and matching module performs semantic segmentation on RGB-D image frames, dividing them into static backgrounds and dynamic objects, and matching the dynamic object regions with the RGB-D image frames, outputting dynamic object masks and point cloud subsets.

[0102] The fusion module uses the ICP algorithm to register a subset of point clouds from consecutive frames, and assigns weights to static and dynamic obstacles and fuses them to obtain a fused environmental perception dataset.

[0103] The heat map generation module updates the dynamic environment in real time, generates a dynamic environment map, and uses a binocular vision device to track the trajectory of moving targets, calculates the real-time motion trajectory of each moving target, and obtains an environmental heat map.

[0104] The path output module uses the A* algorithm to generate a global navigation path for the robot and plans detour paths around hotspot areas, outputting a navigation path sequence.

[0105] This embodiment also provides a computer device applicable to the robot automatic navigation method based on deep vision fusion, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the robot automatic navigation method based on deep vision fusion as proposed in the above embodiment.

[0106] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0107] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the robot autonomous navigation method based on depth vision fusion as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0108] In summary, this invention optimizes the semantic segmentation process of RGB-D image frames by combining multi-scale feature extraction and dilated convolution techniques, improving the real-time performance of semantic segmentation and overcoming the computational bottleneck of traditional high-complexity segmentation algorithms. Furthermore, the path planning and obstacle avoidance strategies enable intelligent optimization of the robot's motion path, considering not only static obstacles but also dynamic adjustments to avoid moving obstacles. While ensuring safety, this maximizes path efficiency, reducing unnecessary detours and pauses, and improving the robot's task execution efficiency, significantly reducing energy consumption and time costs.

[0109] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A robot automatic navigation method based on deep vision fusion, characterized in that: include, Environmental data is collected and preprocessed to obtain RGB-D image frames; Semantic segmentation is performed on RGB-D image frames, dividing them into static backgrounds and dynamic objects. The dynamic object regions are then matched with the RGB-D image frames, and the dynamic object mask and point cloud subset are output. The ICP algorithm is used to register a subset of point clouds in consecutive frames. At the same time, weights are assigned to static and dynamic obstacles and then fused to obtain a fused environmental perception dataset. The system updates the dynamic environment in real time, generates a dynamic environment map, and uses a binocular vision device to track the trajectory of moving targets, calculates the real-time motion trajectory of each moving target, and obtains an environmental heat map. The A* algorithm is used to generate a global navigation path for the robot, and detour paths are planned around hotspot areas, outputting a sequence of navigation paths.

2. The robot automatic navigation method based on deep vision fusion as described in claim 1, characterized in that: The environmental data includes RGB images and LiDAR point cloud data; The preprocessing includes noise reduction, normalization, and cubic mesh generation.

3. The robot automatic navigation method based on deep vision fusion as described in claim 2, characterized in that: The output dynamic object mask includes selecting MobileNetV2 as the core architecture, constructing a DeepLabv3 network, and inputting RGB-D image frames to obtain a semantic segmentation mask and convert it into a three-dimensional spatial representation to form a dynamic object mask; The dynamic object mask and the lidar point cloud data are matched to obtain a subset of the point cloud.

4. The robot automatic navigation method based on deep vision fusion as described in claim 3, characterized in that: The registration of point cloud subsets in consecutive frames includes capturing changes in the point cloud subsets and organizing them into two-frame point cloud data pairs. The ICP algorithm is used to register each pair of two-frame point cloud data to form matching point pairs and determine the movement mode of the matching point pairs, thus obtaining a subset of continuous frame point clouds after registration.

5. The robot automatic navigation method based on deep vision fusion as described in claim 4, characterized in that: The fused environmental perception dataset refers to a dynamic obstacle model constructed using a subset of registered continuous frame point clouds. By using a dynamic obstacle model, the registered point cloud subset and visual information are fused to generate a fused environmental perception dataset. The visual information refers to the depth and image information provided by the RGB-D camera.

6. The robot automatic navigation method based on deep vision fusion as described in claim 5, characterized in that: The process of obtaining the environmental hotspot map includes creating a two-dimensional grid structure to improve the dynamic environment and generating an initial dynamic environment map. The two-dimensional grid structure includes a static layer and a dynamic layer. By processing static and dynamic layers, the initialized dynamic environment map is updated to obtain the updated dynamic environment map; Using a binocular camera, the trajectory of moving targets in the updated dynamic environment map is tracked, matching points are found in the left and right images, and the depth of each matching point is calculated to generate a depth distribution map. Locate moving targets based on the depth distribution map, assign an ID to each moving target, find the region corresponding to the ID of the moving target from the depth distribution map, obtain the preliminary motion trajectory of each moving target, and classify the moving targets into a set of motion parameters according to their IDs; Based on the set of motion parameters of the moving target, the motion rules of the moving target are defined and the driving force of the target is calculated to obtain the real-time motion trajectory of the moving target; The two-dimensional grid structure is divided into multiple smaller grids. The time points of the real-time motion trajectory of the moving target are checked and recorded in the corresponding smaller grids to form a distribution table. Check the number of moving targets in the distribution table and mark them to form a marking result. Add the marking result to the updated dynamic environment map to form an environment hotspot map.

7. The robot automatic navigation method based on deep vision fusion as described in claim 6, characterized in that: The output navigation path sequence refers to the command robot to generate a global navigation path for the robot using the A* algorithm, starting from the current position and ending at the set target point. Check the coordinates on the global navigation path to determine the robot's detour direction, output the global navigation path coordinate sequence, predict and constrain the dynamic threats existing in the global navigation path coordinate sequence, find the conflict point and adjust the detour path to obtain the path sequence with dynamic obstacle avoidance constraints. The path sequence under dynamic obstacle avoidance constraints is smoothed and optimized to generate a navigation path sequence.

8. A robot automatic navigation system based on deep vision fusion, based on the robot automatic navigation method based on deep vision fusion according to any one of claims 1 to 7, characterized in that: include, The data acquisition module collects environmental data and preprocesses it to obtain RGB-D image frames; The segmentation and matching module performs semantic segmentation on RGB-D image frames, dividing them into static backgrounds and dynamic objects, and matching the dynamic object regions with the RGB-D image frames, outputting dynamic object masks and point cloud subsets. The fusion module uses the ICP algorithm to register a subset of point clouds from consecutive frames, and assigns weights to static and dynamic obstacles and fuses them to obtain a fused environmental perception dataset. The heat map generation module updates the dynamic environment in real time, generates a dynamic environment map, and uses a binocular vision device to track the trajectory of moving targets, calculates the real-time motion trajectory of each moving target, and obtains an environmental heat map. The path output module uses the A* algorithm to generate a global navigation path for the robot and plans detour paths around hotspot areas, outputting a navigation path sequence.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the robot automatic navigation method based on depth vision fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the robot automatic navigation method based on depth vision fusion as described in any one of claims 1 to 7.

Images