Robot navigation method and system based on visual identification

Abstract

The invention provides a robot navigation method and system based on visual identification, and relates to the technical field of computer vision, firstly, a continuous visual information set of the surrounding environment of a robot is obtained, the continuous visual information set comprises environment visual images of different directions of the advancing direction of the robot, and the object appearance features and the spatial arrangement relation are recorded; then establishing association mapping between the continuous visual information set and a preset navigation area to obtain a double-association mapping result; generating a semantic anchor point navigation path based on a double-correlation mapping result, wherein the semantic anchor point navigation path comprises a semantic anchor point sequence from the current position to the target position; real-time visual semantic features in the navigation process are obtained through a real-time visual collection module, and a semantic anchor navigation path is dynamically updated; finally, a final navigation execution path is output according to the updated path, the robot is driven to complete navigation operation, and the navigation capability and the intelligent level of the robot in a complex dynamic environment are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a robot navigation method and system based on visual recognition. Background Art

[0002] In the field of robotics, navigation is a key foundation for robots to achieve autonomous movement and complete various tasks. Currently, common robot navigation methods rely primarily on traditional sensor technologies such as lidar and ultrasonic sensors. While these sensors can perceive the surrounding environment to a certain extent, they have many limitations.

[0003] LiDAR (LiDAR) acquires environmental distance information by emitting laser beams and measuring the reflection time. However, this technology is costly and can significantly reduce measurement accuracy in complex environments, such as those with numerous reflective or light-absorbing objects. Ultrasonic sensors use the reflection of ultrasound waves to detect obstacles, but their detection range is limited and they are susceptible to interference from ambient noise, leading to inaccurate measurements.

[0004] Furthermore, existing navigation methods are mostly based on simple geometric models and fixed path planning algorithms, making them difficult to adapt to dynamically changing environments. When the environment changes, such as the movement of obstacles or the appearance of new objects, the robot cannot adjust its navigation path in a timely manner, which can easily lead to navigation failures or collisions. Furthermore, these methods lack the understanding and utilization of environmental semantics, making them unable to intelligently navigate based on the actual meaning of objects in the environment, thus limiting the robot's application in complex scenarios. Summary of the Invention

[0005] In view of the above-mentioned problems, in combination with the first aspect of the present invention, an embodiment of the present invention provides a robot navigation method based on visual recognition, the method comprising:

[0006] Acquire a continuous visual information set of the robot's surrounding environment, wherein the continuous visual information set includes visual images of the environment at different directions in the robot's travel direction, and the visual images of the environment record the appearance characteristics and spatial arrangement relationship of objects in the environment;

[0007] Establishing an association mapping between the continuous visual information set and a preset navigation area of ​​the robot to obtain a dual association mapping result, wherein the dual association mapping result is used to represent the correspondence between the visual semantic features in the continuous visual information and the spatial topological nodes in the preset navigation area of ​​the robot;

[0008] Generate a semantic anchor navigation path for the robot based on the dual association mapping result, wherein the semantic anchor navigation path includes a sequence of semantic anchor points from the robot's current position to the target position, and each semantic anchor point corresponds to a spatial position with visual semantic features within a preset navigation area;

[0009] Acquire real-time visual semantic features during the navigation process through the robot's real-time visual acquisition module, and dynamically update the semantic anchor navigation path based on a matching relationship between the real-time visual semantic features and the dual association mapping result to obtain an updated semantic anchor navigation path;

[0010] The final navigation execution path of the robot is output according to the updated semantic anchor navigation path, and the robot is driven to complete the navigation operation according to the final navigation execution path, where the final navigation execution path includes motion control instructions corresponding to each semantic anchor point.

[0011] On the other hand, an embodiment of the present invention also provides a robot navigation system based on visual recognition, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0012] Based on the above aspects, the embodiments of the present invention obtain a continuous visual information set of the robot's surrounding environment, comprehensively and meticulously record the appearance characteristics and spatial arrangement relationship of objects in the environment, establish an association mapping between the continuous visual information set and the robot's preset navigation area, and obtain a dual association mapping result. The visual semantic features are cleverly matched with the spatial topological nodes in the preset navigation area, achieving a deep fusion of environmental information and the navigation area. The semantic anchor navigation path generated based on the dual association mapping result uses spatial locations with visual semantic features as semantic anchors, making the robot navigation more semantically understanding and intelligent, and capable of path planning based on the actual meaning of objects in the environment. During the navigation process, real-time visual semantic features are obtained through a real-time visual acquisition module, and the semantic anchor navigation path is dynamically updated based on its matching relationship with the dual association mapping result, enabling the robot to adapt to environmental changes in a timely manner. The final navigation execution path output based on the updated semantic anchor navigation path contains motion control instructions corresponding to each semantic anchor, which can accurately drive the robot to complete navigation operations, greatly improving the robot's navigation capability and intelligence level in complex dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the execution flow of the robot navigation method based on visual recognition provided by an embodiment of the present invention.

[0014] Figure 2 Schematic diagram of exemplary hardware and software components of a robot navigation system based on visual recognition provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0015] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1 This is a flow chart of a robot navigation method based on visual recognition provided by an embodiment of the present invention. The robot navigation method based on visual recognition is introduced in detail below.

[0016] Step S110: Acquire a continuous visual information set of the robot's surrounding environment, where the continuous visual information set includes environmental visual images at different directions in the robot's moving direction, and the environmental visual images record the appearance characteristics and spatial arrangement relationships of objects in the environment.

[0017] In the scenario of modern large-scale e-commerce warehousing and logistics centers, robots undertake important tasks such as cargo handling and sorting. In order to achieve accurate navigation, they need to obtain a continuous set of visual information about their surroundings.

[0018] Step S111: Start multiple visual acquisition devices carried by the robot, and the multiple visual acquisition devices are respectively facing the forward, side and oblique directions of the robot's moving direction to form a multi-view acquisition array.

[0019] When the robot works at the e-commerce warehouse logistics center, it is equipped with multiple visual acquisition devices. The forward visual acquisition device faces the direction of the robot's forward movement and can capture information such as the layout of the shelves in front, whether other robots are working, and whether the aisle in front is unobstructed. The lateral visual acquisition devices are located on the left and right sides of the robot, and are mainly used to obtain information such as the placement of goods on the shelves on both sides and whether any goods have fallen into the aisle. The oblique visual acquisition device is installed at an oblique position on the front side of the robot. It can fill the visual blind spot between the forward and lateral visual acquisition devices. For example, it can see the status of goods at the edge of the shelf in front. The above-mentioned visual acquisition devices with different directions together constitute a multi-perspective acquisition array, ensuring that the robot can collect information about the surrounding environment in all directions.

[0020] Step S112: Control each visual acquisition device in the multi-view acquisition array to synchronously acquire environmental visual data at the same time interval to obtain multiple groups of initial environmental visual images, each group of initial environmental visual images corresponds to a collection time point and contains environmental visual content under different perspectives.

[0021] Each visual acquisition device in the multi-view acquisition array is precisely controlled so that it synchronously collects environmental visual data at the same time interval. In an e-commerce warehousing and logistics center, as the robot moves, all visual acquisition devices will simultaneously start the acquisition operation at specific intervals. For example, at a certain acquisition time point, the forward visual acquisition device captures the label information of the goods on the front shelf and the number of shelves; the side visual acquisition device captures the stacking height and neatness of the goods on the side shelves; the oblique visual acquisition device records the distance between the oblique front shelf and the adjacent shelf. The images collected from these different perspectives together constitute a set of initial environmental visual images at that point in time. As time goes by, multiple sets of the above initial environmental visual images can be obtained, reflecting the state of the robot's surrounding environment at different times.

[0022] Step S113: identifying the perspective overlapping area of ​​each group of initial environmental visual images, and determining the overlapping area between the initial environmental visual images collected by different visual collection devices, where the overlapping area contains the visual content of the same environmental object at different perspectives.

[0023] After acquiring multiple sets of initial environmental visual images, it is necessary to identify the overlapping areas of each set of images. Since the fields of view of different visual acquisition devices in the multi-perspective acquisition array partially overlap, these overlapping areas need to be found. In the e-commerce warehousing and logistics center scenario, both the forward and oblique visual acquisition devices may capture part of the same shelf. By analyzing the features of the image, such as the color, texture, structure and other features of the shelf, the overlapping areas between the initial environmental visual images collected by different visual acquisition devices are determined. The overlapping area contains the visual content of the same environmental object from different perspectives, such as images of different sides of the same shelf from different perspectives.

[0024] Step S114: Based on the overlapping area, multiple groups of initial environmental visual images are subjected to perspective fusion processing, and the environmental visual contents under different perspectives are integrated into an environmental visual image with a unified perspective. Redundant information is removed from the fused environmental visual image, and the repeatedly recorded visual contents of environmental objects are deleted, so as to retain the complete appearance characteristics and spatial arrangement relationship of each environmental object.

[0025] Based on the identified overlapping areas, multiple sets of initial environmental visual images are subjected to perspective fusion processing. In e-commerce warehousing and logistics centers, images collected from forward, side, and oblique perspectives are spliced ​​and integrated based on the overlapping areas, so that the environmental visual content from different perspectives forms a unified environmental visual image. During the fusion process, the environmental visual images can be matched and aligned to ensure that the environmental visual images of the same object from different perspectives are accurately spliced ​​together. Then, redundant information is removed from the fused environmental visual image. Because images from different perspectives may repeatedly record the same object, these duplicate portions need to be deleted. For example, images of the same shelf from different perspectives may contain some duplicate images. These duplicate images are deleted, retaining only the complete appearance characteristics of the shelf, including its color, shape, size, etc., as well as its spatial arrangement in the environment, such as its relative position to other shelves and its distance from the aisle.

[0026] Step S115: Arrange the multiple groups of environmental visual images after removing redundancy in the order of the acquisition time points to form a continuous visual information set containing environmental visual content in different directions in the robot's moving direction. Each environmental visual image in the continuous visual information set records the appearance characteristics and spatial arrangement relationship of objects in the environment.

[0027] After removing redundancy, the multiple sets of environmental visual images are arranged in the order of their acquisition time. In e-commerce warehousing and logistics centers, the robot's position and surrounding environment are constantly changing at different points in time. After arranging them in chronological order, a continuous visual information set is formed, containing the environmental visual content in different directions along the robot's direction of travel. Each environmental visual image in this continuous visual information set records the appearance characteristics and spatial arrangement of objects in the environment. For example, images at different time points can show information such as the increase or decrease of goods on the shelves, the movement trajectories of other robots, and the clearing of aisles.

[0028] Step S120: Establish an association mapping between the continuous visual information set and the robot's preset navigation area to obtain a dual association mapping result, which is used to represent the correspondence between the visual semantic features in the continuous visual information and the spatial topological nodes in the robot's preset navigation area.

[0029] In the e-commerce warehousing and logistics center, a continuous set of visual information of the robot's surrounding environment has been obtained. The next step is to establish an association mapping between it and the robot's preset navigation area to obtain a dual association mapping result.

[0030] Step S121: extracting visual semantic features from each environmental visual image in the continuous visual information set, wherein the visual semantic features include category identification of environmental objects, identification of spatial relative relationships between objects, and description of appearance features of objects.

[0031] Extracting visual semantic features from each environmental visual image in a continuous visual information set. In the scenario of an e-commerce warehousing and logistics center, each environmental visual image must first undergo a series of processing.

[0032] Step S1211: performing image region segmentation on each environmental visual image in the continuous visual information set, dividing the image into a plurality of non-overlapping object image regions, each object image region corresponding to an environmental object.

[0033] In the visual imagery of an e-commerce warehouse logistics center, an image segmentation algorithm is used to divide the image into multiple non-overlapping object image regions. For example, shelves, goods, other robots, and aisle signage can be divided into distinct object image regions, each corresponding to a specific environmental object. This segmentation method can decompose complex environmental images into individual object images, facilitating subsequent feature extraction and analysis.

[0034] Step S1212: performing object category recognition on each object image region, determining the category of the environmental object corresponding to the region, and generating a category identifier of the environmental object.

[0035] Each object image region is identified as a category. In e-commerce warehousing and logistics centers, a deep learning-based image recognition model analyzes each object image region. If an object image region corresponds to a shelf, the model identifies the object as a shelf and generates a category identifier representing the shelf. If the object is cargo, a category identifier representing cargo is generated; if it is another robot, a category identifier representing the robot is generated. These category identifiers can be represented using specific codes to facilitate subsequent processing and management.

[0036] Step S1213: Analyze the position coordinates of each object image area in the environmental visual image, calculate the position distance and azimuth angle between different object image areas, determine the spatial relative relationship between objects, and generate a spatial relative relationship identifier between objects.

[0037] Analyze the position coordinates of each object image region in the environmental visual image. In the e-commerce warehousing and logistics center scenario, determine the center coordinates of each object image region based on the image's coordinate system. Then, by calculating the distances and angles between the center coordinates of different object image regions, determine the spatial relative relationships between objects. For example, calculate the straight-line distance between a shelf and adjacent goods, as well as the orientation angle of the goods relative to the shelf, to determine the spatial relative relationship between them and generate a corresponding spatial relative relationship identifier. This spatial relative relationship identifier can represent the relative positional relationships between objects, such as front-to-back, left-to-right, and up-to-down, as well as the degree of distance between them.

[0038] Step S1214: extracting the appearance features of each object image region, wherein the appearance features include the color distribution features, texture distribution features, and contour shape features of the object, and describing the appearance features to form an appearance feature description of the object.

[0039] Extract the appearance features of each object image region. In e-commerce warehousing and logistics centers, image processing technology is used to extract the appearance features of image regions of objects such as shelves and goods. For shelves, extract their color distribution features, such as whether the overall color is blue or gray, and whether the color is evenly distributed on the shelf surface; extract their texture distribution features, such as whether the shelf surface is smooth or textured; extract their contour shape features, such as whether the shelf is rectangular, square, or other shapes. For goods, similarly extract their color, texture, and contour shape features. Then, describe the above appearance features to form a description of the appearance features of the object. For example, describe the shelf as a "blue, rectangular shelf with a regular surface texture," and describe the goods as "red, cube-shaped goods with a smooth surface."

[0040] Step S1215: Associating the category identifier corresponding to each environmental object, the spatial relative relationship identifier between the environmental object and other objects, and the appearance feature description of the environmental object to form a local visual semantic feature corresponding to the environmental object.

[0041] The system associates each environmental object's corresponding category identifier, its spatial relationship identifier with other objects, and a description of its appearance. For example, in an e-commerce warehousing and logistics center, for a shelf, its category identifier, its spatial relationship identifier with surrounding goods and other shelves (e.g., located a certain distance to the left of a certain item), and its appearance description (e.g., a blue, rectangular shelf with a regular textured surface) are associated to form a local visual semantic feature corresponding to the shelf. This local visual semantic feature comprehensively describes the characteristics and positional relationship of the object in the environment.

[0042] Step S1216: Summarize the local visual semantic features of all environmental objects in each environmental visual image to form an overall visual semantic feature corresponding to the environmental visual image. The overall visual semantic feature is the visual semantic feature extracted from the environmental visual image.

[0043] Aggregate the local visual semantic features of all objects in each environmental visual image. In e-commerce warehousing and logistics centers, the local visual semantic features of all shelves, goods, other robots, and so on in an environmental visual image are integrated and aggregated to form the overall visual semantic features corresponding to the environmental visual image. This overall visual semantic feature contains information such as the category, spatial relationship, and appearance characteristics of all objects in the image, and can fully describe the scene presented by the environmental image. In other words, it is the visual semantic feature extracted from the environmental visual image.

[0044] Step S122: obtaining spatial topological data of a preset navigation area of ​​the robot, where the spatial topological data includes a plurality of spatial topological nodes divided in the preset navigation area, and a connection relationship between each spatial topological node and adjacent spatial topological nodes.

[0045] Obtain the spatial topology data for the robot's preset navigation area. In e-commerce warehousing and logistics centers, the preset navigation area is the range within which the robot needs to navigate. This preset navigation area is divided into multiple spatial topology nodes, which can be the location of shelves, intersections of aisles, storage points for goods, and so on. Each spatial topology node has its own specific location information and records its connection relationship with adjacent spatial topology nodes. For example, a spatial topology node located at an aisle intersection is connected to other spatial topology nodes in adjacent aisles through the aisle. These connection relationships can be represented by information such as direction and distance, indicating which adjacent nodes can be reached from this node and how to reach them.

[0046] Step S123: labeling each spatial topological node with a corresponding reference visual semantic feature, wherein the reference visual semantic feature includes a category identifier of a typical environmental object at the spatial topological node, an identifier of a spatial relative relationship between objects, and a description of an object's appearance feature.

[0047] The corresponding reference visual semantic features are labeled for each spatial topology node. In an e-commerce warehouse logistics center, for each spatial topology node, according to the environmental conditions of the location where it is located, the relevant information of the typical environmental objects is labeled. If a spatial topology node is located next to a shelf, the category identification of the shelf (such as a shelf), the spatial relative relationship identification of the shelf with other surrounding objects (such as adjacent shelves, passages) (such as a certain distance from the adjacent shelves, located on one side of the passage), and the appearance feature description of the shelf (such as color, texture, shape, etc.) are labeled. The above-mentioned reference visual semantic features can provide a reference standard for subsequent correlation mapping, helping to determine the correspondence between visual semantic features and spatial topology nodes.

[0048] Step S124: Correlate the visual semantic features extracted from the continuous visual information set with the reference visual semantic features of each spatial topology node, and construct a correlation relationship model of visual semantic features and spatial topology nodes.

[0049] The visual semantic features extracted from the continuous visual information set are correlated with the reference visual semantic features of each spatial topology node. In the e-commerce warehouse logistics center scenario, by comparing the category identification, spatial relative relationship identification and appearance feature description in the visual semantic features and the reference visual semantic features, the correlation between them is established. For example, if the shelf described in a visual semantic feature is very similar to the shelf described in the reference visual semantic feature of a certain spatial topology node in terms of category, spatial relationship and appearance, then the correlation between the two can be established. Using machine learning algorithms such as neural network algorithms, the above-mentioned correlation information is learned and processed to construct a correlation relationship model of visual semantic features and spatial topology nodes. In the neural network algorithm, the input is the visual semantic features and the reference visual semantic features, which pass through the input layer into the hidden layer. The neurons in the hidden layer process and transform the input information, extract the correlation patterns between the features, and finally output the correlation results through the output layer, i.e. the correspondence between the visual semantic features and the spatial topology nodes.

[0050] Step S125: Optimize the correlation relationship model, adjust the corresponding weight of the visual semantic features and the spatial topology nodes, and make each visual semantic feature accurately match the corresponding spatial topology node.

[0051] Optimize the association relationship model. In e-commerce warehousing and logistics centers, the association relationship model needs to be adjusted and optimized because the environment may change or there may be certain errors in the modeling process. The model is trained and updated by continuously inputting new visual semantic features and reference visual semantic feature data. Adjust the corresponding weights of visual semantic features and spatial topological nodes. For example, if the spatial relative relationship identifier in a certain visual semantic feature is more important in the matching process, increase the weight of this feature in the association calculation. Through multiple iterative training, the model can more accurately match each visual semantic feature to the corresponding spatial topological node, improving the accuracy of the association.

[0052] Step S126: Arrange the correspondence between the optimized visual semantic features and the spatial topological nodes according to a preset format to generate a dual association mapping result for representing the correspondence between the visual semantic features in the continuous visual information and the spatial topological nodes in the preset navigation area.

[0053] The correspondence between the optimized visual semantic features and the spatial topological nodes is organized according to a preset format. In an e-commerce warehousing and logistics center, the preset format can be a structured data format, such as JSON or XML. The information of each visual semantic feature and the corresponding spatial topological node is organized and stored to form a clear mapping relationship table. The mapping relationship table records the identification information of the spatial topological node corresponding to each visual semantic feature, as well as information such as the degree of matching between them. In the above manner, a dual-association mapping result is generated to represent the correspondence between the visual semantic features in the continuous visual information and the spatial topological nodes in the preset navigation area.

[0054] Step S130: Generate a semantic anchor navigation path for the robot based on the dual association mapping result. The semantic anchor navigation path includes a sequence of semantic anchors from the current position of the robot to the target position. Each semantic anchor corresponds to a spatial position with visual semantic features within a preset navigation area.

[0055] In the e-commerce warehousing and logistics center, the dual-association mapping results of the continuous visual information set and the preset navigation area have been obtained. Next, the robot's semantic anchor navigation path is generated based on this.

[0056] Step S131: parse the dual association mapping result, extract the correspondence between the visual semantic features and the spatial topology nodes, determine the visual semantic features corresponding to the current position of the robot, match the visual semantic features to the corresponding spatial topology nodes, and use them as the initial spatial topology nodes of the current position of the robot.

[0057] The bi-association mapping results are parsed to extract the correspondence between visual semantic features and spatial topological nodes. Within the e-commerce warehousing and logistics center, the bi-association mapping results are parsed to extract the corresponding information between the visual semantic features and spatial topological nodes. Next, the visual semantic features corresponding to the robot's current location are determined. Visual semantic features are extracted from the robot's currently captured environmental image, including information such as object categories, spatial relationships, and appearance characteristics. These visual semantic features are compared with the information in the bi-association mapping results to identify a matching spatial topological node. This spatial topological node serves as the initial spatial topological node for the robot's current location. For example, if the robot's currently captured image shows it next to a shelf of a specific color and shape, the corresponding spatial topological node in the bi-association mapping results is identified based on this information in the visual semantic features and determined as the initial spatial topological node.

[0058] Step S132: Obtain the target visual semantic features corresponding to the robot navigation target position, and match the target visual semantic features to the corresponding spatial topological nodes through the dual association mapping results as the target spatial topological nodes of the robot target position.

[0059] Obtain the target visual semantic features corresponding to the robot's navigation target location. In an e-commerce warehousing and logistics center, the target location may be a specific cargo storage point or shelf location. Based on the environmental information of the target location, extract the target visual semantic features, including the category, spatial relationship, and appearance characteristics of the object at the target location. For example, if the target location is a shelf storing specific goods, then extract the category identification of the shelf, its spatial relationship with the surrounding environment, and its appearance characteristics as the target visual semantic features. Then, match the target visual semantic features to the corresponding spatial topological node through the dual association mapping results. Compare and match the target visual semantic features with the information in the dual association mapping results to find the corresponding spatial topological node, which serves as the target spatial topological node for the robot's target location.

[0060] Step S133: planning a spatial topology path from the initial spatial topology node to the target spatial topology node based on the spatial topology data of the preset navigation area, where the spatial topology path includes a plurality of sequentially connected spatial topology nodes.

[0061] Based on the spatial topology data of the preset navigation area, a spatial topology path is planned from the initial spatial topology node to the target spatial topology node. In the e-commerce warehousing and logistics center, the spatial topology data of the preset navigation area records the connection relationship between each spatial topology node. Using a path planning algorithm, such as the A algorithm or the Dijkstra algorithm, an optimal path is searched in the spatial topology data based on the position information of the initial spatial topology node and the target spatial topology node. This path contains multiple spatial topology nodes connected in sequence. These nodes are connected through channels or other passable areas, forming a navigation route from the initial position to the target position. For example, the path may start from the spatial topology node at a channel intersection, pass through the spatial topology nodes on several channels, and finally reach the spatial topology node corresponding to the target position.

[0062] Step S134: For each spatial topological node in the spatial topological path, obtain its corresponding visual semantic feature through the double association mapping result, and determine the spatial topological node with the visual semantic feature as a semantic anchor point.

[0063] For each spatial topological node in the spatial topological path, its corresponding visual semantic features are obtained through the dual association mapping results. In the e-commerce warehousing and logistics center, each spatial topological node on the spatial topological path is traversed, and the visual semantic features corresponding to each node are found according to the dual association mapping results. The above visual semantic features contain information such as the category, spatial relationship and appearance characteristics of the object at the node. Then, the spatial topological node with visual semantic features is determined as a semantic anchor point. If a spatial topological node has clear visual semantic features that can clearly describe the environmental conditions at the node, it will be used as a semantic anchor point. For example, a spatial topological node located next to a specific shelf, whose corresponding visual semantic features clearly describe the relevant information of the shelf, can be determined as a semantic anchor point.

[0064] Step S1341: traverse each spatial topology node in the spatial topology path and record identification information of each spatial topology node.

[0065] Traverse each spatial topological node in the spatial topological path and record the identification information of each node. In an e-commerce warehousing and logistics center, follow the planned spatial topological path and visit each spatial topological node in sequence. Each spatial topological node has its own unique identification information, such as a number or name. This identification information is recorded for subsequent processing and querying. For example, the identification information of the first spatial topological node on the path is recorded as "Node A", the second as "Node B", and so on.

[0066] Step S1342: For each spatial topology node, query the dual-association mapping result to extract the visual semantic features corresponding to the spatial topology node, including the class identifier of the environmental objects at the spatial topology node, the spatial relative relationship identifier between objects, and the object appearance feature description.

[0067] For each spatial topology node, query the dual-association mapping result to extract the visual semantic features corresponding to the spatial topology node. In the e-commerce warehouse logistics center, according to the recorded identification information of the spatial topology node, the corresponding visual semantic features are found in the dual-association mapping result. For each node, the class identifier of the environmental objects at the node is extracted, such as shelves, goods, etc.; the spatial relative relationship identifier between objects, such as the front, back, left, right, distance, etc. relationship between objects; and the object appearance feature description, such as color, shape, texture, etc. For example, for a spatial topology node located near a shelf, the class identifier of the shelf is extracted as "shelf", the spatial relative relationship identifier with the adjacent aisle is "located at a certain distance on the left side of the aisle", and the appearance feature description of the shelf is "blue, multi-layer structure, rectangular shelf".

[0068] Step S1343: Perform integrity analysis on the extracted visual semantic features to confirm whether they contain complete class identifiers, spatial relative relationship identifiers, and appearance feature descriptions.

[0069] Perform integrity analysis on the extracted visual semantic features. In the e-commerce warehouse logistics center, check whether the visual semantic features corresponding to each spatial topology node contain complete class identifiers, spatial relative relationship identifiers, and appearance feature descriptions. If an important piece of information is missing in a visual semantic feature, such as the class identifier of an object or the spatial relationship is not clear, then the visual semantic feature is incomplete. For example, if only the appearance feature of an object is recorded, but its class and spatial relationship with other objects are not specified, then the visual semantic feature is incomplete. Through integrity analysis, spatial topology nodes with complete visual semantic features are screened out.

[0070] Step S1344: Mark the spatial topology nodes with complete visual semantic features as candidate semantic anchor points, and the visual semantic features corresponding to the candidate semantic anchor points can effectively represent the spatial position of the nodes.

[0071] The spatial topology node with complete visual semantic features is marked as a candidate semantic anchor point. In the e-commerce warehouse logistics center, the spatial topology node with complete visual semantic features can accurately describe the environment at the node, thereby effectively representing the spatial position of the node. The node is marked as a candidate semantic anchor point for subsequent further screening and determination of the final semantic anchor point. For example, the visual semantic features corresponding to a spatial topology node completely describe the category of the shelf at the node, the spatial relationship with the surrounding passageway and other shelves, and the appearance features of the shelf. Therefore, the node is marked as a candidate semantic anchor point.

[0072] Step S1345: Distribution density analysis is performed on the candidate semantic anchor points, the topology network distance between adjacent candidate semantic anchor points is calculated, and the topology network distance of adjacent candidate semantic anchor points is within a preset reasonable range.

[0073] The distribution density of the candidate semantic anchor points is analyzed. In the e-commerce warehouse logistics center, the topology network distance between adjacent candidate semantic anchor points is calculated. The topology network distance can be calculated according to the connection relationship and distance information between nodes in the spatial topology data. For example, the topology network distance is determined by calculating the number of spatial topology nodes or the actual distance between two adjacent candidate semantic anchor points. Then, it is judged whether the topology network distance of adjacent candidate semantic anchor points is within a preset reasonable range. The preset reasonable range is determined according to the navigation requirements of the robot and the environmental characteristics. If the distance is too far, the robot may easily lose its way during navigation; if the distance is too close, it may increase the complexity and computational load of navigation. Through the above analysis, the distribution density of the candidate semantic anchor points is ensured to be moderate.

[0074] Step S1346: According to the distribution density analysis result, the candidate semantic anchor points are supplemented or adjusted. Between adjacent candidate semantic anchor points whose topology network distance exceeds the preset reasonable range, a spatial topology node with complete visual semantic features is selected as a supplementary candidate semantic anchor point. The finally determined candidate semantic anchor point is used as a semantic anchor point.

[0075] According to the distribution density analysis result, the candidate semantic anchor points are supplemented or adjusted. In the e-commerce warehouse logistics center, if the topology network distance between adjacent candidate semantic anchor points exceeds the preset reasonable range, a node with complete visual semantic features is selected as a supplementary candidate semantic anchor point from the spatial topology nodes between the two nodes. For example, if the distance between two adjacent candidate semantic anchor points is too far, a spatial topology node with clear visual semantic features is selected as a supplementary candidate semantic anchor point in the passageway between them. After supplementation and adjustment, the finally determined candidate semantic anchor point is used as a semantic anchor point. The above semantic anchor point can provide accurate positioning and guidance information for the robot during navigation.

[0076] Step S135: Arrange the determined semantic anchor points in sequence according to the connection order of the spatial topological nodes in the spatial topological path to form a semantic anchor point sequence from the current position to the target position of the robot.

[0077] The identified semantic anchor points are arranged sequentially according to the order in which the spatial topological nodes are connected within the spatial topological path. In e-commerce warehousing and logistics centers, the semantic anchor points are arranged sequentially according to their location within the path, based on the spatial topological path planning. For example, starting from the initial spatial topological node, the semantic anchor points are arranged sequentially along the path to form an ordered sequence. This sequence represents the key nodes in the robot's navigation path from its current location to its target location. Each semantic anchor point corresponds to a spatial location with clear visual semantic features.

[0078] Step S136: Associating the semantic anchor point sequence with the spatial position information corresponding to each semantic anchor point to generate a semantic anchor point navigation path including the semantic anchor point sequence.

[0079] The semantic anchor sequence is associated with the spatial location information corresponding to each semantic anchor. In e-commerce warehousing and logistics centers, each semantic anchor has its corresponding spatial location information, such as coordinates and relative position to the surrounding environment. This spatial location information is bound and associated with each anchor in the semantic anchor sequence to form a complete semantic anchor navigation path. This semantic anchor navigation path not only contains the semantic anchor sequence information but also the specific spatial location of each anchor. It can provide detailed guidance for robot navigation, enabling the robot to accurately move from its current position to the target position along the path.

[0080] Step S140: The real-time visual semantic features in the navigation process are acquired through the real-time visual acquisition module of the robot, and the semantic anchor navigation path is dynamically updated based on the matching relationship between the real-time visual semantic features and the dual association mapping result to obtain an updated semantic anchor navigation path.

[0081] In e-commerce warehousing and logistics centers, robots begin to navigate according to semantic anchor navigation paths. During the navigation process, they need to obtain environmental information in real time and dynamically update the path.

[0082] Step S141: Start the real-time visual acquisition module of the robot, which acquires the real-time visual image of the environment during the robot navigation process according to a preset acquisition frequency.

[0083] The robot's real-time visual acquisition module is activated. Within the e-commerce warehouse logistics center, this module captures real-time visual images of the robot's surroundings during navigation at a preset acquisition frequency. This frequency is determined based on the robot's movement speed and the likelihood of environmental changes. For example, if the robot moves quickly or the environment changes frequently, the acquisition frequency may be set higher; conversely, it may be set lower. The real-time visual acquisition module continuously collects images of the robot's surroundings to promptly reflect environmental changes.

[0084] Step S142: extracting real-time visual semantic features from the real-time environment visual image, wherein the real-time visual semantic features include category identifiers of real-time environment objects, identifiers of spatial relative relationships between objects, and descriptions of object appearance features.

[0085] Extracting real-time visual semantic features from real-time environmental visual images. In e-commerce warehousing and logistics centers, the processing of real-time environmental visual images is similar to the previous process of extracting visual semantic features from continuous visual information sets.

[0086] Step S1421: performing image region segmentation on the real-time environment visual image, dividing it into a plurality of non-overlapping real-time object image regions, each of which corresponds to a real-time environment object.

[0087] Perform image region segmentation on the real-time visual environment. In e-commerce warehousing and logistics centers, image segmentation algorithms are used to divide the real-time visual environment into multiple non-overlapping real-time object image regions. For example, shelves, goods, and other operating robots can be divided into different real-time object image regions, each corresponding to a real-time object. This segmentation decomposes the complex real-time environment image into individual object images, facilitating subsequent feature extraction and analysis.

[0088] Step S1422: performing real-time object category recognition on each real-time object image region, determining the category of the real-time environment object corresponding to the real-time object image region, and generating a category identifier of the real-time environment object.

[0089] Real-time object category recognition is performed for each real-time object image region. In e-commerce warehousing and logistics centers, a deep learning-based image recognition model analyzes each real-time object image region. If a real-time object image region corresponds to a shelf, the model identifies the object as a shelf and generates a category identifier representing the shelf. If it is cargo, a category identifier representing cargo is generated; if it is another robot, a category identifier representing the robot is generated. These category identifiers can be represented using specific codes to facilitate subsequent processing and management.

[0090] Step S1423: Analyze the position coordinates of each real-time object image area in the real-time environment visual image, calculate the position distance and azimuth angle between different real-time object image areas, determine the spatial relative relationship between real-time objects, and generate a spatial relative relationship identifier between objects.

[0091] Analyze the position coordinates of each real-time object image area in the real-time environment visual image. In the e-commerce warehousing and logistics center scenario, determine the center position coordinates of each real-time object image area based on the image coordinate system. Then, by calculating the distance and angle between the center position coordinates of different real-time object image areas, determine the spatial relative relationship between real-time objects. For example, calculate the straight-line distance between a real-time shelf and the adjacent real-time goods, as well as the azimuth angle of the real-time goods relative to the real-time shelf, so as to determine the spatial relative relationship between them and generate a corresponding spatial relative relationship identifier. This spatial relative relationship identifier can represent the relative position relationship between objects, such as front and back, left and right, up and down, as well as the degree of distance between them.

[0092] Step S1424: extracting the real-time appearance features of each real-time object image area, describing the real-time appearance features, and forming an appearance feature description of the real-time environment object. The real-time appearance features include the color distribution features, texture distribution features, and contour shape features of the real-time environment object.

[0093] Extract the real-time appearance features of each real-time object image region. In e-commerce warehousing and logistics centers, image processing technology is used to extract the appearance features of real-time object image regions such as real-time shelves and real-time goods. For real-time shelves, their color distribution features are extracted, such as whether the overall color is blue or gray, and whether the color distribution on the shelf surface is uniform; their texture distribution features are extracted, such as whether the shelf surface is smooth or textured; and their contour shape features are extracted, such as whether the shelf is rectangular, square, or another shape. For real-time goods, their color, texture, and contour shape features are also extracted. Then, the above real-time appearance features are described to form a description of the appearance features of the real-time environment objects. For example, the real-time shelf is described as a "blue, rectangular real-time shelf with a regular surface texture," and the real-time goods are described as "red, cube-shaped real-time goods with a smooth surface."

[0094] Step S1425: Associating the category identifier corresponding to each real-time environment object, the spatial relative relationship identifier between the real-time environment object and other real-time environment objects, and the appearance feature description of the real-time environment object to form a local real-time visual semantic feature corresponding to the real-time environment object.

[0095] The category identifier corresponding to each real-time environment object, the spatial relative relationship identifier of the real-time environment object with other real-time environment objects, and the appearance feature description of the real-time environment object are associated. In an e-commerce warehousing and logistics center, for a real-time shelf, the category identifier representing the real-time shelf, the spatial relative relationship identifier with the surrounding real-time goods and other real-time shelves (such as being located a certain distance to the left of a real-time product), and the appearance feature description (such as a blue, rectangular real-time shelf with a regular texture on the surface) are associated to form the local real-time visual semantic feature corresponding to the real-time shelf. This local real-time visual semantic feature can comprehensively describe the characteristics and positional relationship of the real-time object in the real-time environment.

[0096] Step S1426: Summarize the local real-time visual semantic features of all real-time environment objects in the real-time environment visual image to form the overall real-time visual semantic features corresponding to the real-time environment visual image. The overall real-time visual semantic features are the real-time visual semantic features extracted from the real-time environment visual image.

[0097] Aggregate the local real-time visual semantic features of all real-time environmental objects in a real-time environmental visual image. In an e-commerce warehousing and logistics center, the local real-time visual semantic features of all real-time shelves, real-time goods, other real-time robots, etc. in a real-time environmental visual image are integrated and aggregated to form the overall real-time visual semantic features corresponding to the real-time environmental visual image. This overall real-time visual semantic feature contains information such as the category, spatial relationship, and appearance characteristics of all real-time objects in the real-time image, and can fully describe the scene presented by the real-time environmental image. In other words, it is the real-time visual semantic feature extracted from the real-time environmental visual image.

[0098] Step S143: querying the dual association mapping result, matching the extracted real-time visual semantic features with the visual semantic features in the dual association mapping result, and determining the spatial topological nodes corresponding to the real-time visual semantic features.

[0099] The dual-association mapping results are queried and the extracted real-time visual semantic features are matched with the visual semantic features in the dual-association mapping results. In the e-commerce warehousing and logistics center, the real-time visual semantic features are compared one by one with the visual semantic features recorded in the dual-association mapping results. By comparing information such as category identification, spatial relative relationship identification, and appearance feature description, the spatial topological node corresponding to the visual semantic feature that best matches the real-time visual semantic feature is found. For example, if the scene described by the real-time visual semantic feature is very similar to the scene described by a visual semantic feature in the dual-association mapping results, the spatial topological node corresponding to the visual semantic feature is determined to be the spatial topological node corresponding to the real-time visual semantic feature.

[0100] Step S144: comparing the spatial topology node with the spatial topology node corresponding to the semantic anchor point that should be currently reached in the semantic anchor point navigation path, and analyzing the difference in visual semantic features between the two.

[0101] This spatial topology node is compared with the spatial topology node corresponding to the semantic anchor point that the robot is currently supposed to reach in the semantic anchor navigation path. In the e-commerce warehousing and logistics center, the spatial topology node corresponding to the real-time visual semantic feature is compared with the spatial topology node corresponding to the semantic anchor point that the robot is currently supposed to reach in the semantic anchor navigation path. The differences in visual semantic features between the two are analyzed, including differences in category identification, spatial relative relationship identification, and appearance feature description. For example, if the shelf category at the real-time spatial topology node is different from the shelf category at the current semantic anchor point, or if the spatial relative relationship between objects has changed, this indicates a visual semantic feature difference. These differences may be caused by environmental changes (such as the movement of goods or the installation of new shelves) or navigation errors.

[0102] Step S145: Based on the visual semantic feature difference, the reference visual semantic features of the corresponding spatial topological nodes in the dual-association mapping result are corrected to make the dual-association mapping result consistent with the real-time environment.

[0103] Based on the difference in visual semantic features, the reference visual semantic features of the corresponding spatial topology nodes in the dual-association mapping results are corrected. In an e-commerce warehousing and logistics center, if a difference in visual semantic features is found between the real-time spatial topology node and the spatial topology node corresponding to the current semantic anchor point, the reference visual semantic features of the spatial topology node in the dual-association mapping results are corrected based on the real-time visual semantic features. For example, if the color of a shelf in the real-time environment changes, the color description information in the reference visual semantic features of the spatial topology node where the shelf is located in the dual-association mapping results is updated. Through the above correction operation, the dual-association mapping results are kept consistent with the real-time environment, ensuring the accuracy of subsequent navigation.

[0104] Step S146: re-evaluate each semantic anchor point in the semantic anchor point navigation path according to the corrected dual-association mapping result, adjust the spatial position of the semantic anchor point, or add a new semantic anchor point.

[0105] Based on the corrected dual-association mapping results, each semantic anchor in the semantic anchor navigation path is re-evaluated. In the e-commerce warehousing and logistics center, since the dual-association mapping results have been updated, each semantic anchor in the semantic anchor navigation path needs to be checked and evaluated again. If the visual semantic features of the spatial topological node corresponding to a semantic anchor have changed significantly, the spatial position of the semantic anchor may need to be adjusted. For example, if the shelf originally corresponding to a semantic anchor is removed, it is necessary to re-determine other spatial topological nodes with visual semantic features near the location as new semantic anchors. At the same time, if it is found that the distribution of semantic anchors in certain areas of the path is unreasonable, such as too large spacing or lack of semantic anchors in certain key positions, new semantic anchors need to be added. In this way, the semantic anchor navigation path can be more adaptable to changes in the real-time environment.

[0106] Step S147: Arrange the adjusted semantic anchor points in a navigation order to form an updated semantic anchor navigation path.

[0107] The adjusted semantic anchors are arranged in navigation order. In the e-commerce warehousing and logistics center, the adjusted semantic anchors are reordered based on the robot's navigation direction and path planning. The semantic anchors are arranged sequentially from the current location to the target location, forming an updated semantic anchor navigation path. This semantic anchor navigation path accounts for real-time environmental changes and provides more accurate navigation guidance for the robot, ensuring it can successfully reach its target location from its current location.

[0108] Step S150: outputting a final navigation execution path of the robot according to the updated semantic anchor navigation path, and driving the robot to complete the navigation operation according to the final navigation execution path, wherein the final navigation execution path includes motion control instructions corresponding to each semantic anchor point.

[0109] In the e-commerce warehousing and logistics center, the updated semantic anchor navigation path has been obtained. Next, the robot's final navigation execution path is output based on this and the robot is driven to complete the navigation operation.

[0110] Step S151: parsing the updated semantic anchor navigation path, extracting the semantic anchor sequence and the spatial position information corresponding to each semantic anchor.

[0111] Parse the updated semantic anchor navigation path to extract the semantic anchor sequence and the spatial location information corresponding to each semantic anchor. Within the e-commerce warehousing and logistics center, the updated semantic anchor navigation path is parsed to extract the recorded semantic anchor sequence and the spatial location information of each semantic anchor. A semantic anchor sequence is a series of semantic anchors arranged in navigation order. Each semantic anchor has its corresponding spatial location information, such as coordinates and relative position to the surrounding environment.

[0112] Step S152: Calculate the spatial distance and relative orientation between two adjacent semantic anchor points. The spatial distance is the straight-line distance between the corresponding spatial positions of the two semantic anchor points, and the relative orientation is the direction of the latter semantic anchor point relative to the former semantic anchor point.

[0113] Calculate the spatial distance and relative orientation between two adjacent semantic anchor points. In the e-commerce warehousing and logistics center, the spatial distance between two adjacent semantic anchor points is calculated based on the spatial position information corresponding to each semantic anchor point. This spatial distance is the straight-line distance between the corresponding spatial positions of the two semantic anchor points and can be calculated through spatial coordinates. At the same time, calculate the relative orientation of the latter semantic anchor point relative to the previous semantic anchor point, that is, the direction of the latter semantic anchor point in space relative to the direction of the previous semantic anchor point. For example, the latter semantic anchor point may be in the front, left front, right rear, etc. of the previous semantic anchor point. The above spatial distance and relative orientation information are crucial for generating motion control instructions for the robot.

[0114] Step S153: According to the motion characteristic parameters of the robot, the spatial distance and relative orientation between adjacent semantic anchor points are converted into motion control instructions, which include the robot's movement speed, turning direction and movement duration.

[0115] Based on the robot's motion characteristic parameters, the spatial distance and relative orientation between adjacent semantic anchor points are converted into motion control instructions. In e-commerce warehousing and logistics centers, the robot's motion characteristic parameters include its maximum movement speed, minimum movement speed, and steering capability. Based on the spatial distance and relative orientation between adjacent semantic anchor points, combined with the robot's motion characteristic parameters, the movement speed, steering direction, and movement duration required for the robot to move from one semantic anchor point to the next are calculated. For example, if the spatial distance is long and the robot's maximum movement speed allows, a higher movement speed can be set; if the relative orientation requires a turn, the appropriate steering direction and steering angle are determined based on the robot's steering capability. This information is combined into motion control instructions to control the robot's movement.

[0116] Step S154: Arrange the motion control instructions corresponding to each semantic anchor point in sequence according to the order of the semantic anchor point sequence to form a motion control instruction sequence.

[0117] The motion control instructions corresponding to each semantic anchor are arranged in sequence according to the order of the semantic anchors. In the e-commerce warehousing and logistics center, the motion control instructions corresponding to each semantic anchor are sorted according to the order of the semantic anchors in the semantic anchor sequence. Starting from the first semantic anchor, the corresponding motion control instructions are arranged in order to form an ordered motion control instruction sequence. This motion control instruction sequence represents the series of motion operations that the robot needs to perform during the entire navigation process from the current position to the target position.

[0118] Step S155: Associating the motion control instruction sequence with the corresponding semantic anchor point identifier to generate a final navigation execution path including the motion control instruction sequence, wherein each motion control instruction in the final navigation execution path corresponds to a semantic anchor point.

[0119] The motion control instruction sequence is associated with the corresponding semantic anchor identifier. In the e-commerce warehousing and logistics center, each motion control instruction is assigned the identifier of its corresponding semantic anchor, indicating which semantic anchor the robot is instructed to move from. This association generates a final navigation execution path containing the motion control instruction sequence. This final navigation execution path not only includes the robot's motion control instructions but also clearly identifies the semantic anchor corresponding to each motion control instruction, enabling the robot to accurately navigate along the path.

[0120] Step S156: Decompose the motion control instruction sequence in the final navigation execution path into individual independent motion control instructions, each independent motion control instruction corresponding to a motion process from the current semantic anchor point to the next semantic anchor point.

[0121] The motion control instruction sequence in the final navigation execution path is broken down into individual independent motion control instructions. In an e-commerce warehousing and logistics center, each instruction in the motion control instruction sequence is separated into individual independent motion control instructions. Each independent motion control instruction corresponds to a motion process from the current semantic anchor point to the next semantic anchor point. For example, the first independent motion control instruction controls the robot to move from the initial semantic anchor point to the second semantic anchor point, the second independent motion control instruction controls the robot to move from the second semantic anchor point to the third semantic anchor point, and so on.

[0122] Step S157: Send the first independent motion control instruction to the motion control module of the robot. The motion control module controls the driving components of the robot to start according to the moving speed, turning direction and moving duration in the motion control instruction, so that the robot moves to the first semantic anchor point.

[0123] The first independent motion control instruction is sent to the robot's motion control module. In the e-commerce warehousing and logistics center, upon receiving the first independent motion control instruction, the motion control module activates the robot's drive components based on the instruction's information, including movement speed, direction, and movement duration. The drive components, such as motors and wheels, are controlled to start moving the robot toward the first semantic anchor point as instructed by the instruction. For example, if the instruction requires the robot to move forward in a straight line at a certain speed, the motion control module will control the motor to rotate at the corresponding speed, driving the wheels to move the robot forward.

[0124] Step S158: During the movement of the robot toward the first semantic anchor point, the real-time visual acquisition module continuously acquires the real-time environment visual image and extracts the real-time visual semantic features. By matching the real-time visual semantic features with the dual association mapping results, it is confirmed that the robot has reached the spatial position corresponding to the first semantic anchor point.

[0125] As the robot moves toward the first semantic anchor point, the real-time visual acquisition module continues to operate. Within the e-commerce warehousing and logistics center, the module continuously acquires real-time visual images of the environment at a preset acquisition frequency and extracts real-time visual semantic features from them. These real-time visual semantic features are then matched with the results of the dual-association mapping. By comparing information such as category identification, spatial relative relationship identification, and appearance feature descriptions, the robot confirms its arrival at the spatial location corresponding to the first semantic anchor point. If the real-time visual semantic features closely match the visual semantic features corresponding to the first semantic anchor point, the robot has arrived at the spatial location corresponding to that semantic anchor point.

[0126] Step S159: After confirming that the robot has reached the first semantic anchor point, a second independent motion control instruction is sent to the motion control module to control the robot to move toward the second semantic anchor point.

[0127] After confirming the robot has reached the first semantic anchor point, a second independent motion control command is sent to the motion control module. In the e-commerce warehousing and logistics center, upon receiving this second independent motion control command, the motion control module controls the robot's drive components to adjust their operating state based on the command's information, including movement speed, turning direction, and movement duration, causing the robot to begin moving toward the second semantic anchor point. For example, if the second semantic anchor point is to the right of the first and requires a certain turning angle, the motion control module controls the robot's steering components to perform the turning operation and then move to the second semantic anchor point at the specified speed.

[0128] Step S1510: Repeat the above process, send each independent motion control instruction in turn, control the robot to pass through each semantic anchor point, when the robot reaches the semantic anchor point corresponding to the target position according to the last motion control instruction, the motion control module stops executing the motion control instruction, and the robot completes the navigation operation from the current position to the target position.

[0129] The above process is repeated, sending each independent motion control instruction in sequence. In the e-commerce warehousing and logistics center, as the robot passes each semantic anchor point, the corresponding independent motion control instructions are continuously sent to the motion control module, which controls the robot to move and turn according to the instructions. When the robot reaches the semantic anchor point corresponding to the target location according to the last motion control instruction, the motion control module stops executing the motion control instruction. At this point, the robot has completed the navigation operation from its current position to the target location and successfully reached its destination, such as a designated cargo storage point or work area, and can continue with subsequent tasks such as cargo handling and sorting.

[0130] Figure 2 A schematic diagram of exemplary hardware and software components of a vision-based robotic navigation system 100 that can implement the concepts of the present application is shown in some embodiments of the present application. For example, a processor 120 can be used in the vision-based robotic navigation system 100 to perform the functions of the present application.

[0131] The vision-based robot navigation system 100 can be a general-purpose server or a special-purpose server, both of which can be used to implement the vision-based robot navigation method of the present application. Although only one server is shown in this application, for convenience, the functions described in this application can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0132] For example, the robot navigation system 100 based on visual recognition may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and different forms of storage media 140, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the robot navigation system 100 based on visual recognition may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present application may be implemented according to these program instructions. The robot navigation system 100 based on visual recognition may also include an I / O interface 150 between the computer and other input and output devices.

[0133] For ease of explanation, only one processor is described in the visual recognition-based robot navigation system 100. However, it should be noted that the visual recognition-based robot navigation system 100 in the present application may also include multiple processors, so the steps performed by one processor described in the present application may also be performed jointly or individually by multiple processors. For example, if the processor of the visual recognition-based robot navigation system 100 performs step A and step B, it should be understood that step A and step B may also be performed jointly by two different processors or individually in one processor. For example, the first processor performs step A, the second processor performs step B, or the first processor and the second processor perform steps A and B together.

[0134] In addition, an embodiment of the present invention further provides a readable storage medium, in which computer-executable instructions are preset. When a processor executes the computer-executable instructions, the robot navigation method based on visual recognition as described above is implemented.

[0135] It should be noted that in order to simplify the description of the present invention and thus help understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, multiple features are sometimes combined into one embodiment, figure or description thereof.

Claims

1. A robot navigation method based on visual recognition, characterized in that: The method comprises: Acquire a continuous visual information set of the robot's surrounding environment, wherein the continuous visual information set includes visual images of the environment at different directions in the robot's travel direction, and the visual images of the environment record the appearance characteristics and spatial arrangement relationship of objects in the environment; Establishing an association mapping between the continuous visual information set and a preset navigation area of ​​the robot to obtain a dual association mapping result, wherein the dual association mapping result is used to represent the correspondence between the visual semantic features in the continuous visual information and the spatial topological nodes in the preset navigation area of ​​the robot; Generate a semantic anchor navigation path for the robot based on the dual association mapping result, wherein the semantic anchor navigation path includes a sequence of semantic anchor points from the robot's current position to the target position, and each semantic anchor point corresponds to a spatial position with visual semantic features within a preset navigation area; Acquire real-time visual semantic features during the navigation process through the robot's real-time visual acquisition module, and dynamically update the semantic anchor navigation path based on a matching relationship between the real-time visual semantic features and the dual association mapping result to obtain an updated semantic anchor navigation path; The final navigation execution path of the robot is output according to the updated semantic anchor navigation path, and the robot is driven to complete the navigation operation according to the final navigation execution path, where the final navigation execution path includes motion control instructions corresponding to each semantic anchor point.

2. The robot navigation method based on visual recognition according to claim 1, characterized in that: The step of obtaining a continuous set of visual information of the robot's surrounding environment includes: Activate multiple visual acquisition devices carried by the robot, wherein the multiple visual acquisition devices are respectively oriented toward the forward, side, and oblique directions of the robot's travel direction to form a multi-view acquisition array; Controlling each visual acquisition device in the multi-view acquisition array to synchronously acquire environmental visual data at the same time interval to obtain multiple groups of initial environmental visual images, each group of initial environmental visual images corresponding to a collection time point and containing environmental visual content at different viewing angles; Identifying the viewpoint overlap region of each set of initial environmental visual images, and determining the overlap region between the initial environmental visual images acquired by different visual acquisition devices, wherein the overlap region contains visual content of the same environmental object at different viewpoints; Performing perspective fusion processing on the multiple sets of initial environmental visual images based on the overlapping areas, integrating the environmental visual contents from different perspectives into an environmental visual image from a unified perspective, removing redundant information from the fused environmental visual image, deleting duplicated recorded visual contents of environmental objects, and retaining the complete appearance features and spatial arrangement relationship of each environmental object; The multiple sets of environmental visual images after redundancy removal are arranged in the order of acquisition time points to form a continuous visual information set containing environmental visual content in different directions in the robot's travel direction. Each environmental visual image in the continuous visual information set records the appearance characteristics and spatial arrangement relationship of objects in the environment.

3. The robot navigation method based on visual recognition according to claim 1, characterized in that: The establishing of the association mapping between the continuous visual information set and the preset navigation area of ​​the robot to obtain a dual association mapping result includes: Extracting visual semantic features from each environmental visual image in the continuous visual information set, wherein the visual semantic features include category identifications of environmental objects, identifications of spatial relative relationships between objects, and descriptions of appearance features of the objects; Acquire spatial topological data of a preset navigation area of ​​the robot, wherein the spatial topological data includes a plurality of spatial topological nodes divided in the preset navigation area, and a connection relationship between each spatial topological node and an adjacent spatial topological node; Label each spatial topological node with a corresponding reference visual semantic feature, wherein the reference visual semantic feature includes a category identifier of a typical environmental object at the spatial topological node, an identifier of the spatial relative relationship between objects, and a description of the object's appearance features; The visual semantic features extracted from the continuous visual information set are associated with the reference visual semantic features of each spatial topological node to build an association relationship model between the visual semantic features and the spatial topological nodes; Optimizing the association relationship model and adjusting the corresponding weights of visual semantic features and spatial topological nodes so that each visual semantic feature can be accurately matched to the corresponding spatial topological node; The correspondence between the optimized visual semantic features and the spatial topological nodes is sorted according to a preset format to generate a dual association mapping result for representing the correspondence between the visual semantic features in the continuous visual information and the spatial topological nodes in the preset navigation area.

4. The robot navigation method based on visual recognition according to claim 3, characterized in that: The extracting of visual semantic features from each environmental visual image in the continuous visual information set includes: Performing image region segmentation on each environmental visual image in the continuous visual information set, dividing the image into a plurality of non-overlapping object image regions, each object image region corresponding to an environmental object; Perform object category recognition on each object image area, determine the category of the environmental object corresponding to the object image area, and generate a category identifier of the environmental object; Analyze the position coordinates of each object image area in the environmental visual image, calculate the position distance and azimuth angle between different object image areas, determine the spatial relative relationship between objects, and generate the spatial relative relationship identification between objects; Extracting appearance features of each object image region, wherein the appearance features include color distribution features, texture distribution features, and contour shape features of the object, and describing the appearance features to form an appearance feature description of the object; Associating the category identifier corresponding to each environmental object, the spatial relative relationship identifier between the environmental object and other objects, and the appearance feature description of the environmental object to form a local visual semantic feature corresponding to the environmental object; The local visual semantic features of all environmental objects in each environmental visual image are summarized to form the overall visual semantic features corresponding to the environmental visual image. The overall visual semantic features are the visual semantic features extracted from the environmental visual image.

5. The robot navigation method based on visual recognition according to claim 1, characterized in that: Generating a semantic anchor navigation path of the robot based on the dual association mapping result includes: Analyzing the dual association mapping result, extracting the correspondence between the visual semantic features and the spatial topological nodes, determining the visual semantic features corresponding to the current position of the robot, and matching the corresponding spatial topological nodes with the visual semantic features as the initial spatial topological nodes of the current position of the robot; Obtaining a target visual semantic feature corresponding to a target position of the robot navigation, and matching the target visual semantic feature to a corresponding spatial topological node through the dual association mapping result as the target spatial topological node of the robot target position; Based on the spatial topology data of the preset navigation area, planning a spatial topology path from the initial spatial topology node to the target spatial topology node, wherein the spatial topology path includes a plurality of sequentially connected spatial topology nodes; For each spatial topological node in the spatial topological path, obtaining its corresponding visual semantic feature through the double association mapping result, and determining the spatial topological node with the visual semantic feature as a semantic anchor point; Arrange the determined semantic anchor points in sequence according to the connection order of the spatial topological nodes in the spatial topological path to form a semantic anchor point sequence from the robot's current position to the target position; The semantic anchor point sequence is associated with the spatial position information corresponding to each semantic anchor point to generate a semantic anchor point navigation path containing the semantic anchor point sequence.

6. The robot navigation method based on visual recognition according to claim 5, characterized in that: The step of obtaining the corresponding visual semantic features of each spatial topological node in the spatial topological path through a double association mapping result, and determining the spatial topological node having the visual semantic features as a semantic anchor point includes: Traversing each spatial topological node in the spatial topological path and recording identification information of each spatial topological node; For each spatial topological node, query the dual association mapping results and extract the visual semantic features corresponding to the spatial topological node, including the category identification of the environmental objects at the spatial topological node, the spatial relative relationship identification between objects, and the description of the object appearance features; Perform integrity analysis on the extracted visual semantic features to confirm whether they contain complete category identification, spatial relative relationship identification, and appearance feature description; Marking a spatial topological node with complete visual semantic features as a candidate semantic anchor point, wherein the visual semantic features corresponding to the candidate semantic anchor point can effectively represent the spatial position of the node; Performing a distribution density analysis on the candidate semantic anchor points, calculating the topological network distance between adjacent candidate semantic anchor points, and ensuring that the topological network distance between adjacent candidate semantic anchor points is within a preset reasonable range; According to the results of distribution density analysis, candidate semantic anchor points are supplemented or adjusted. Between adjacent candidate semantic anchor points whose topological network distance exceeds the preset reasonable range, spatial topological nodes with complete visual semantic features are selected as supplementary candidate semantic anchor points, and the final candidate semantic anchor points are used as semantic anchor points.

7. The robot navigation method based on visual recognition according to claim 1, characterized in that: The real-time visual semantic features acquired during the navigation process by the real-time visual acquisition module of the robot, and the semantic anchor navigation path dynamically updated based on the matching relationship between the real-time visual semantic features and the dual association mapping result to obtain the updated semantic anchor navigation path, comprising: Starting a real-time visual acquisition module of the robot, wherein the real-time visual acquisition module acquires a real-time visual image of the environment during the robot navigation process according to a preset acquisition frequency; Extracting real-time visual semantic features from the real-time environment visual image, wherein the real-time visual semantic features include category identification of real-time environment objects, identification of spatial relative relationships between objects, and description of object appearance features; Querying the dual association mapping result, matching the extracted real-time visual semantic features with the visual semantic features in the dual association mapping result, and determining the spatial topological nodes corresponding to the real-time visual semantic features; Compare the spatial topological node with the spatial topological node corresponding to the semantic anchor point that should be reached in the semantic anchor navigation path, and analyze the difference in visual semantic features between the two; Based on the visual semantic feature differences, the reference visual semantic features of the corresponding spatial topological nodes in the dual association mapping results are corrected to make the dual association mapping results consistent with the real-time environment; According to the corrected dual-association mapping results, each semantic anchor in the semantic anchor navigation path is re-evaluated, and the spatial position of the semantic anchor is adjusted or a new semantic anchor is added; Arrange the adjusted semantic anchors in the navigation order to form an updated semantic anchor navigation path.

8. The robot navigation method based on visual recognition according to claim 7, characterized in that: The extracting of real-time visual semantic features from the real-time environment visual image comprises: Performing image region segmentation on the real-time environment visual image to divide it into a plurality of non-overlapping real-time object image regions, each real-time object image region corresponding to a real-time environment object; Performing real-time object category recognition on each real-time object image region, determining the category of the real-time environment object corresponding to the real-time object image region, and generating a category identifier of the real-time environment object; Analyze the position coordinates of each real-time object image area in the real-time environment visual image, calculate the position distance and azimuth angle between different real-time object image areas, determine the spatial relative relationship between real-time objects, and generate spatial relative relationship identifiers between objects; Extracting real-time appearance features of each real-time object image region, describing the real-time appearance features, and forming an appearance feature description of the real-time environment object, wherein the real-time appearance features include color distribution features, texture distribution features, and contour shape features of the real-time environment object; Associating the category identifier corresponding to each real-time environment object, the spatial relative relationship identifier between the real-time environment object and other real-time environment objects, and the appearance feature description of the real-time environment object to form a local real-time visual semantic feature corresponding to the real-time environment object; The local real-time visual semantic features of all real-time environment objects in the real-time environment visual image are aggregated to form an overall real-time visual semantic feature corresponding to the real-time environment visual image, wherein the overall real-time visual semantic feature is a real-time visual semantic feature extracted from the real-time environment visual image.

9. The robot navigation method based on visual recognition according to claim 1, characterized in that: Outputting a final navigation execution path of the robot according to the updated semantic anchor navigation path, and driving the robot to complete the navigation operation according to the final navigation execution path, includes: Parsing the updated semantic anchor navigation path to extract the semantic anchor sequence and the spatial position information corresponding to each semantic anchor; Calculate the spatial distance and relative orientation between two adjacent semantic anchor points, where the spatial distance is the straight-line distance between the corresponding spatial positions of the two semantic anchor points, and the relative orientation is the direction of the latter semantic anchor point relative to the previous semantic anchor point; According to the robot's motion characteristic parameters, the spatial distance and relative orientation between adjacent semantic anchor points are converted into motion control instructions, wherein the motion control instructions include the robot's movement speed, turning direction and movement duration; According to the order of the semantic anchor point sequence, the motion control instructions corresponding to each semantic anchor point are arranged in sequence to form a motion control instruction sequence; Associating the motion control instruction sequence with the corresponding semantic anchor point identifier to generate a final navigation execution path including the motion control instruction sequence, wherein each motion control instruction in the final navigation execution path corresponds to a semantic anchor point; Decomposing the motion control instruction sequence in the final navigation execution path into individual independent motion control instructions, each independent motion control instruction corresponding to a motion process from a current semantic anchor point to a next semantic anchor point; Sending the first independent motion control instruction to the motion control module of the robot, the motion control module controls the driving components of the robot to start according to the movement speed, turning direction and movement duration in the motion control instruction, so that the robot moves toward the first semantic anchor point; During the robot's movement toward the first semantic anchor point, the real-time visual acquisition module continuously acquires real-time environmental visual images, extracts real-time visual semantic features, and confirms that the robot has reached the spatial position corresponding to the first semantic anchor point by matching the real-time visual semantic features with the dual association mapping results. When it is confirmed that the robot has reached the first semantic anchor point, a second independent motion control instruction is sent to the motion control module to control the robot to move toward the second semantic anchor point; Repeat the above process, send each independent motion control instruction in turn, and control the robot to pass through each semantic anchor point. When the robot reaches the semantic anchor point corresponding to the target position according to the last motion control instruction, the motion control module stops executing the motion control instruction, and the robot completes the navigation operation from the current position to the target position.

10. A robot navigation system based on visual recognition, characterized in that: It includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the robot navigation method based on visual recognition as described in any one of claims 1 to 9.

Images