Robot navigation method and system based on visual recognition

Abstract

The application provides a robot navigation method and system based on visual recognition, and relates to the technical field of computer vision. First, a continuous visual information set of the environment around the robot is acquired, which contains visual images of the environment at different positions of the robot's direction of travel, and records the appearance characteristics and spatial arrangement relationship of objects. Then, an associated mapping of the continuous visual information set and a preset navigation area is established to obtain a double-associated mapping result. Based on the double-associated mapping result, a semantic anchor navigation path is generated, which contains a semantic anchor sequence from the current position to the target position. Real-time visual semantic features in the navigation process are acquired through a real-time visual acquisition module, and the semantic anchor navigation path is dynamically updated. Finally, the final navigation execution path is output according to the updated path to drive the robot to complete the navigation operation, thereby improving the navigation ability and intelligent level of the robot in a complex dynamic environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a robot navigation method and system based on visual recognition. BACKGROUND

[0002] In the field of robot technology, navigation function is the key foundation for robots to realize autonomous action and complete various tasks. Currently, the common robot navigation method mainly relies on traditional sensor technology, such as laser radar, ultrasonic sensor, etc. Although the above sensors can perceive the surrounding environment information to a certain extent, they have many limitations.

[0003] Laser radar obtains environmental distance information by emitting laser beams and measuring reflection time, but its cost is high, and in complex environments, such as the presence of a large number of light-reflecting or light-absorbing objects, the measurement accuracy will be significantly affected. Ultrasonic sensor detects obstacles by using ultrasonic reflection, but its detection range is limited and is easily disturbed by environmental noise, resulting in inaccurate measurement results.

[0004] In addition, most of the existing navigation methods are based on simple geometric models and fixed path planning algorithms, which are difficult to adapt to dynamically changing environments. When the environment changes, such as the movement of obstacles, the appearance of new objects, etc., the robot cannot adjust the navigation path in time, and is prone to navigation failure or collision, etc. Moreover, these methods lack understanding and utilization of environmental semantic information, and cannot intelligently navigate according to the actual meaning of objects in the environment, limiting the application of robots in complex scenarios. SUMMARY

[0005] In view of the above-mentioned problems, in combination with the first aspect of the present application, the embodiments of the present application provide a robot navigation method based on visual recognition, which comprises:

[0006] obtaining a continuous visual information set of the environment around the robot, the continuous visual information set containing environmental visual images of different positions in the direction of the robot's travel, the environmental visual images recording the appearance features and spatial arrangement relationship of objects in the environment;

[0007] establishing an association mapping between the continuous visual information set and the robot's preset navigation area to obtain a double-association mapping result, the double-association mapping result being used to represent the corresponding relationship between the visual semantic features in the continuous visual information and the spatial topology nodes in the robot's preset navigation area;

[0008] generating a semantic anchor point navigation path of the robot based on the double-association mapping result, the semantic anchor point navigation path containing a semantic anchor point sequence of the robot from the current position to the target position, each semantic anchor point corresponding to a spatial position with visual semantic features in the preset navigation area.

[0009] The robot's real-time visual acquisition module acquires real-time visual semantic features during the navigation process. Based on the matching relationship between the real-time visual semantic features and the double association mapping results, the semantic anchor navigation path is dynamically updated to obtain the updated semantic anchor navigation path.

[0010] The robot's final navigation execution path is output based on the updated semantic anchor navigation path, and the robot is driven to complete the navigation operation according to the final navigation execution path. The final navigation execution path contains motion control instructions corresponding to each semantic anchor.

[0011] In another aspect, embodiments of the present invention also provide a vision-based robot navigation system, including a processor and a machine-readable storage medium connected to the processor. The machine-readable storage medium is used to store programs, instructions, or code, and the processor is used to execute the programs, instructions, or code in the machine-readable storage medium to implement the above-described method.

[0012] Based on the above, this embodiment of the invention acquires a continuous set of visual information about the robot's surrounding environment, comprehensively and meticulously recording the appearance features and spatial arrangement of objects in the environment. It establishes an association mapping between the continuous visual information set and the robot's preset navigation area, obtaining a dual association mapping result. This cleverly corresponds visual semantic features with spatial topological nodes within the preset navigation area, achieving deep integration of environmental information and the navigation area. The semantic anchor navigation path generated based on the dual association mapping result uses spatial locations with visual semantic features as semantic anchors, making robot navigation more semantically understanding and intelligent, enabling path planning based on the actual meaning of objects in the environment. During navigation, real-time visual semantic features are acquired through a real-time visual acquisition module, and the semantic anchor navigation path is dynamically updated based on its matching relationship with the dual association mapping result. This allows the robot to adapt to changes in the environment in a timely manner. Finally, the final navigation execution path output based on the updated semantic anchor navigation path includes motion control commands corresponding to each semantic anchor, accurately driving the robot to complete navigation operations, greatly improving the robot's navigation capabilities and intelligence level in complex dynamic environments. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the execution flow of the robot navigation method based on vision recognition provided in an embodiment of the present invention.

[0014] Figure 2 This is a schematic diagram of exemplary hardware and software components of a vision-based robot navigation system provided in an embodiment of the present invention. Detailed Implementation

[0015] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1 This is a flowchart illustrating a vision-based robot navigation method according to an embodiment of the present invention. The vision-based robot navigation method will be described in detail below.

[0016] Step S110: Obtain a continuous visual information set of the robot's surrounding environment. This continuous visual information set includes environmental visual images from different directions along the robot's travel direction. These environmental visual images record the appearance features and spatial arrangement of objects in the environment.

[0017] In modern large-scale e-commerce warehousing and logistics centers, robots undertake important tasks such as cargo handling and sorting. In order to achieve accurate navigation, they need to acquire a continuous set of visual information about their surrounding environment.

[0018] Step S111: Activate the multiple visual acquisition devices mounted on the robot. These multiple visual acquisition devices are respectively oriented towards the forward, lateral, and oblique directions of the robot's movement direction to form a multi-view acquisition array.

[0019] When the robot operates in this e-commerce warehousing and logistics center, it is equipped with multiple visual acquisition devices. The forward-facing visual acquisition devices face the direction the robot is moving, capturing information such as the layout of the shelves ahead, whether other robots are operating, and whether the aisles ahead are clear. The lateral visual acquisition devices are located on the left and right sides of the robot, primarily used to obtain information such as the placement of goods on the shelves on either side and whether any goods have fallen into the aisles. The diagonal visual acquisition devices are installed at an angle on the front of the robot, supplementing the blind spots between the forward and lateral visual acquisition devices; for example, they can see the status of goods at the edge of the shelves diagonally ahead. These visual acquisition devices with different orientations together form a multi-view acquisition array, ensuring that the robot can collect information about its surrounding environment from all directions.

[0020] Step S112: Control each visual acquisition device in the multi-view acquisition array to synchronously acquire environmental visual data at the same time interval, and obtain multiple sets of initial environmental visual images. Each set of initial environmental visual images corresponds to an acquisition time point and contains environmental visual content from different viewpoints.

[0021] Each visual acquisition device in the multi-view acquisition array is precisely controlled to synchronously acquire environmental visual data at the same time intervals. In an e-commerce warehousing and logistics center, as the robot moves, all visual acquisition devices simultaneously initiate acquisition operations at specific time intervals. For example, at a certain acquisition point, the forward-facing visual acquisition device captures information such as the label information and shelf number of the goods on the shelf in front; the lateral visual acquisition device captures the stacking height and neatness of the goods on the side shelf; and the oblique visual acquisition device records the distance between the shelf diagonally in front and the adjacent shelf. The images acquired from these different perspectives together constitute a set of initial environmental visual images at that point in time. As time progresses, multiple sets of the above initial environmental visual images can be obtained, reflecting the state of the robot's surrounding environment at different times.

[0022] Step S113: Identify the overlapping regions of each set of initial environmental visual images to determine the overlapping regions between the initial environmental visual images acquired by different visual acquisition devices. The overlapping regions contain the visual content of the same environmental objects from different perspectives.

[0023] After acquiring multiple sets of initial environmental visual images, it is necessary to identify overlapping regions in each set of images. Since the fields of view of different visual acquisition devices in a multi-view acquisition array partially overlap, it is crucial to identify these overlapping regions. In an e-commerce warehousing and logistics center scenario, both forward and oblique visual acquisition devices may capture a portion of the same shelf. By analyzing the features of the images, such as the color, texture, and structure of the shelf, the overlapping regions between the initial environmental visual images acquired by different visual acquisition devices can be determined. These overlapping regions contain the visual content of the same environmental object from different perspectives, such as images of the same shelf showing different sides from different viewpoints.

[0024] Step S114: Based on the overlapping area, perform viewpoint fusion processing on multiple sets of initial environmental visual images, integrate the environmental visual content from different viewpoints into an environmental visual image with a unified viewpoint, remove redundant information from the fused environmental visual image, delete the visual content of duplicate environmental objects, and retain the complete appearance features and spatial arrangement relationship of each environmental object.

[0025] Based on the identified overlapping areas, multiple initial environmental visual images undergo perspective fusion processing. In an e-commerce warehousing and logistics center, images acquired from forward, lateral, and oblique perspectives are stitched and integrated, using the overlapping areas as a reference, to form a unified environmental visual image from different perspectives. During the fusion process, matching and alignment operations are performed on the environmental visual images to ensure that the environmental visual images of the same object from different perspectives can be accurately stitched together. Then, redundant information is removed from the fused environmental visual image. Because images from different perspectives may record the same object repeatedly, these duplicate parts need to be deleted. For example, the same shelf may have some duplicate images in images from different perspectives. These duplicate images are deleted, retaining only the complete appearance features of the shelf, including its color, shape, size, etc., as well as its spatial arrangement in the environment, such as its relative position to other shelves and its distance from aisles.

[0026] Step S115: Arrange the multiple sets of environmental visual images after removing redundancy in chronological order of acquisition time to form a continuous visual information set containing environmental visual content from different directions along the robot's movement direction. Each environmental visual image in this continuous visual information set records the appearance features and spatial arrangement of objects in the environment.

[0027] Multiple sets of environmental visual images, after redundancy removal, are arranged chronologically according to their acquisition time. In e-commerce warehousing and logistics centers, the robot's position and surrounding environment are constantly changing at different times. Arranging them chronologically creates a continuous set of visual information containing visual content of the environment from different directions along the robot's movement. Each environmental visual image in this continuous set records the appearance features and spatial arrangement of objects in the environment. For example, images from different time points can show information such as the increase or decrease of goods on shelves, the movement trajectories of other robots, and the unobstructedness of passageways.

[0028] Step S120: Establish the association mapping between the continuous visual information set and the robot's preset navigation area to obtain the double association mapping result. The double association mapping result is used to represent the correspondence between the visual semantic features in the continuous visual information and the spatial topological nodes in the robot's preset navigation area.

[0029] In the e-commerce warehousing and logistics center, a continuous set of visual information about the robot's surrounding environment has been acquired. The next step is to establish an association mapping between this information and the robot's preset navigation area to obtain a double association mapping result.

[0030] Step S121: Extract visual semantic features from each environmental visual image in the continuous visual information set. The visual semantic features include the category identifier of environmental objects, the spatial relative relationship identifier between objects, and the appearance feature description of objects.

[0031] Visual semantic features are extracted from each environmental visual image in a continuous set of visual information. In the context of an e-commerce warehousing and logistics center, each environmental visual image first undergoes a series of processing steps.

[0032] Step S1211: Perform image region segmentation on each environmental visual image in the continuous visual information set, dividing the image into multiple non-overlapping object image regions, with each object image region corresponding to an environmental object.

[0033] In the environmental visual images of e-commerce warehousing and logistics centers, image segmentation algorithms are used to divide the images into multiple non-overlapping object image regions. For example, shelves, goods, other robots, and aisle signs are divided into different object image regions, with each region corresponding to an environmental object. Through this segmentation, complex environmental images can be decomposed into independent object images, facilitating subsequent feature extraction and analysis.

[0034] Step S1212: Perform object category recognition on each object image region, determine the category of the environmental object corresponding to the region, and generate the category identifier of the environmental object.

[0035] Object category identification is performed for each object image region. In e-commerce warehousing and logistics centers, a deep learning-based image recognition model is used to analyze each object image region. If an object image region corresponds to a shelf, the model will identify the object as a shelf and generate a category identifier representing the shelf; if it is goods, a category identifier representing the goods will be generated; if it is another type of robot, a category identifier representing the robot will be generated. These category identifiers can be represented by specific codes for convenient subsequent processing and management.

[0036] Step S1213: Analyze the position coordinates of each object image region in the environmental visual image, calculate the position distance and orientation angle between different object image regions, determine the spatial relative relationship between objects, and generate spatial relative relationship identifiers between objects.

[0037] The system analyzes the positional coordinates of each object image region within the environmental visual image. In an e-commerce warehousing and logistics center scenario, the center position coordinates of each object image region are determined based on the image's coordinate system. Then, the spatial relative relationships between objects are determined by calculating the distance and angle between the center position coordinates of different object image regions. For example, the straight-line distance between a shelf and adjacent goods, as well as the orientation angle of the goods relative to the shelf, are calculated to determine their spatial relative relationships and generate corresponding spatial relative relationship identifiers. These identifiers can represent the relative positional relationships between objects, such as front-back, left-right, and up-down, as well as the degree of distance between them.

[0038] Step S1214: Extract the appearance features of each object image region. These appearance features include the object's color distribution features, texture distribution features, and contour shape features. Describe these appearance features to form an object appearance feature description.

[0039] Extract the appearance features of each object image region. In e-commerce warehousing and logistics centers, image processing techniques are used to extract the appearance features of object image regions such as shelves and goods. For shelves, color distribution features are extracted, such as whether the overall color is blue or gray, and whether the color distribution on the shelf surface is uniform; texture distribution features are extracted, such as whether the shelf surface is smooth or textured; and outline shape features are extracted, such as whether the shelf is rectangular, square, or other shapes. For goods, color, texture, and outline shape features are extracted similarly. Then, these appearance features are described to form an object appearance feature description. For example, the shelf is described as "a blue, rectangular shelf with a regular textured surface," and the goods are described as "red, cube-shaped goods with a smooth surface."

[0040] Step S1215: Associate the category identifier corresponding to each environmental object, the spatial relative relationship identifier between the environmental object and other objects, and the appearance feature description of the environmental object to form the local visual semantic feature corresponding to the environmental object.

[0041] The system associates the category identifier of each environmental object, its spatial relationship with other objects, and its appearance description. In an e-commerce warehousing and logistics center, for a shelf, its category identifier, its spatial relationship with surrounding goods and other shelves (e.g., located a certain distance to the left of a certain item), and its appearance description (e.g., a blue, rectangular shelf with a regularly textured surface) are associated to form the local visual semantic feature corresponding to that shelf. This local visual semantic feature comprehensively describes the object's characteristics and positional relationships within the environment.

[0042] Step S1216: Summarize the local visual semantic features of all environmental objects in each environmental visual image to form the overall visual semantic features corresponding to the environmental visual image. The overall visual semantic features are the visual semantic features extracted from the environmental visual image.

[0043] The local visual semantic features of all objects in each environmental visual image are summarized. In an e-commerce warehousing and logistics center, the local visual semantic features of all shelves, goods, other robots, etc., in an environmental visual image are integrated and summarized to form the overall visual semantic features corresponding to that environmental visual image. This overall visual semantic feature includes information such as the category, spatial relationship, and appearance features of all objects in the image, and can completely describe the scene presented by the environmental image, that is, the visual semantic features extracted from the environmental visual image.

[0044] Step S122: Obtain the spatial topology data of the robot's preset navigation area. The spatial topology data includes multiple spatial topology nodes divided within the preset navigation area, and the connection relationship between each spatial topology node and its adjacent spatial topology nodes.

[0045] Acquire spatial topology data for the robot's preset navigation area. In e-commerce warehousing and logistics centers, the preset navigation area is the range within which the robot needs to perform navigation operations. This preset navigation area is divided into multiple spatial topology nodes, which can be locations of shelves, intersections of aisles, storage points for goods, etc. Each spatial topology node has its specific location information and records its connection relationships with adjacent spatial topology nodes. For example, a spatial topology node located at an aisle intersection is connected to other spatial topology nodes on adjacent aisles via the aisles. These connections can be represented by information such as direction and distance, indicating which adjacent nodes can be reached from this node and how to reach them.

[0046] Step S123: Label each spatial topology node with corresponding reference visual semantic features. The reference visual semantic features include the category identifier of typical environmental objects at the spatial topology node, the spatial relative relationship identifier between objects, and the description of the appearance features of the objects.

[0047] Each spatial topology node is labeled with corresponding reference visual semantic features. In an e-commerce warehousing and logistics center, for each spatial topology node, relevant information about typical environmental objects is labeled based on its location and environment. If a spatial topology node is located next to a shelf, then the category identifier of the shelf (e.g., shelf), the spatial relative identifier of the shelf with other surrounding objects (e.g., adjacent shelves, aisles) (e.g., a certain distance from adjacent shelves, located on one side of the aisle), and the appearance characteristics description of the shelf (e.g., color, texture, shape, etc.) are labeled. These reference visual semantic features can provide a reference standard for subsequent association mapping, helping to determine the correspondence between visual semantic features and spatial topology nodes.

[0048] Step S124: The visual semantic features extracted from the continuous visual information set are associated with the reference visual semantic features of each spatial topology node to build a model of the association between visual semantic features and spatial topology nodes.

[0049] Visual semantic features extracted from a continuous set of visual information are correlated with reference visual semantic features for each spatial topological node. In the context of an e-commerce warehousing and logistics center, associations are established by comparing information such as category identifiers, spatial relationship identifiers, and appearance descriptions in the visual semantic features and reference visual semantic features. For example, if a shelf described in a visual semantic feature is very similar to a shelf described in the reference visual semantic feature of a spatial topological node in terms of category, spatial relationship, and appearance, then an association can be established between the two. Machine learning algorithms, such as neural network algorithms, are used to learn and process the above association information to construct an association model between visual semantic features and spatial topological nodes. In the neural network algorithm, the input is the visual semantic feature and the reference visual semantic feature. After passing through the input layer, the input is passed to the hidden layer. The neurons in the hidden layer process and transform the input information, extract the association patterns between features, and finally output the association result through the output layer, i.e., the correspondence between visual semantic features and spatial topological nodes.

[0050] Step S125: Optimize the association model by adjusting the corresponding weights of visual semantic features and spatial topology nodes so that each visual semantic feature can be accurately matched to the corresponding spatial topology node.

[0051] Optimize the association model. In e-commerce warehousing and logistics centers, the environment may change, or there may be errors in the modeling process, so the association model needs to be adjusted and optimized. The model is trained and updated by continuously inputting new visual semantic features and reference visual semantic feature data. The weights corresponding to visual semantic features and spatial topological nodes are adjusted; for example, if the spatial relative relationship identifier in a certain visual semantic feature is more important in the matching process, the weight of that feature in the association calculation is increased. Through multiple iterations of training, the model can more accurately match each visual semantic feature to its corresponding spatial topological node, improving the accuracy of the association.

[0052] Step S126: Organize the optimized visual semantic features and spatial topology nodes according to a preset format to generate a double association mapping result that represents the correspondence between visual semantic features in continuous visual information and spatial topology nodes in a preset navigation area.

[0053] The optimized visual semantic features and their correspondence with spatial topology nodes are organized according to a preset format. In e-commerce warehousing and logistics centers, this preset format can be a structured data format such as JSON or XML. The information of each visual semantic feature and its corresponding spatial topology node is organized and stored to form a clear mapping table. This mapping table records the identification information of the spatial topology node corresponding to each visual semantic feature, as well as the degree of matching between them. Through this method, a bi-association mapping result is generated to represent the correspondence between visual semantic features in continuous visual information and spatial topology nodes within a preset navigation area.

[0054] Step S130: Generate a semantic anchor navigation path for the robot based on the double association mapping result. The semantic anchor navigation path contains a sequence of semantic anchors from the robot's current position to the target position. Each semantic anchor corresponds to a spatial position with visual semantic features within a preset navigation area.

[0055] In the e-commerce warehousing and logistics center, the dual association mapping results of continuous visual information set and preset navigation area have been obtained. Next, the semantic anchor point navigation path of the robot will be generated based on this.

[0056] Step S131: Analyze the double association mapping result, extract the correspondence between visual semantic features and spatial topology nodes, determine the visual semantic features corresponding to the robot's current position, and match the corresponding spatial topology nodes through the visual semantic features as the initial spatial topology nodes for the robot's current position.

[0057] The bi-association mapping results are analyzed to extract the correspondence between visual semantic features and spatial topology nodes. In e-commerce warehousing and logistics centers, the bi-association mapping results are analyzed to extract the recorded visual semantic features and spatial topology node correspondences. Then, the visual semantic features corresponding to the robot's current position are determined. Visual semantic features, including object categories, spatial relationships, and appearance features, are extracted from the environmental visual images currently captured by the robot. These visual semantic features are compared with the information in the bi-association mapping results to find a matching spatial topology node, which serves as the initial spatial topology node for the robot's current position. For example, if the robot's currently captured image shows it is located next to a shelf of a specific color and shape, the corresponding spatial topology node is found in the bi-association mapping results based on this information in the visual semantic features and is determined as the initial spatial topology node.

[0058] Step S132: Obtain the target visual semantic features corresponding to the robot navigation target position, and match the target visual semantic features to the corresponding spatial topology nodes through the double association mapping result, which serve as the target spatial topology nodes for the robot target position.

[0059] The process involves acquiring the target visual semantic features corresponding to the robot's navigation target location. In an e-commerce warehousing and logistics center, the target location might be a specific goods storage point or shelf location. Based on the environmental information of the target location, the target visual semantic features are extracted, including the category, spatial relationship, and appearance features of objects at the target location. For example, if the target location is a shelf storing specific goods, then the category identifier of the shelf, its spatial relationship with the surrounding environment, and its appearance features are extracted as target visual semantic features. Then, the target visual semantic features are matched to the corresponding spatial topology nodes through a bi-association mapping result. By comparing and matching the target visual semantic features with the information in the bi-association mapping result, the corresponding spatial topology node is found, and this spatial topology node serves as the target spatial topology node for the robot's target location.

[0060] Step S133: Based on the spatial topology data of the preset navigation area, plan a spatial topology path from the initial spatial topology node to the target spatial topology node. This spatial topology path contains multiple spatial topology nodes connected in sequence.

[0061] Based on the spatial topology data of a pre-defined navigation area, a spatial topology path is planned from the initial spatial topology node to the target spatial topology node. In an e-commerce warehousing and logistics center, the spatial topology data of the pre-defined navigation area records the connection relationships between various spatial topology nodes. Using path planning algorithms, such as A-Solution or Dijkstra's algorithm, an optimal path is searched in the spatial topology data based on the positional information of the initial and target spatial topology nodes. This path contains multiple sequentially connected spatial topology nodes, which are connected by passages or other traversable areas, forming a navigation route from the initial position to the target position. For example, the path might start from a spatial topology node at a passage intersection, pass through several spatial topology nodes on several passages, and finally reach the spatial topology node corresponding to the target position.

[0062] Step S134: For each spatial topology node in the spatial topology path, obtain its corresponding visual semantic features through the double association mapping result, and determine the spatial topology nodes with visual semantic features as semantic anchors.

[0063] For each spatial topology node in the spatial topology path, its corresponding visual semantic features are obtained through the bi-association mapping result. In an e-commerce warehousing and logistics center, each spatial topology node on the spatial topology path is traversed, and the visual semantic features corresponding to each node are found according to the bi-association mapping result. These visual semantic features include information such as the category, spatial relationship, and appearance features of the object at the node. Then, spatial topology nodes with visual semantic features are determined as semantic anchors. If a spatial topology node has clear visual semantic features that can clearly describe the environmental situation at the node, it is used as a semantic anchor. For example, a spatial topology node located next to a specific shelf, whose corresponding visual semantic features clearly describe the relevant information of the shelf, can be determined as a semantic anchor.

[0064] Step S1341: Traverse each spatial topology node in the spatial topology path and record the identification information of each spatial topology node.

[0065] Traverse each spatial topology node along the planned spatial topology path and record the identification information of each node. In an e-commerce warehousing and logistics center, visit each spatial topology node sequentially along the planned path. Each spatial topology node has unique identification information, such as a number or name. This identification information is recorded for subsequent processing and querying. For example, the identification information of the first spatial topology node on the path is recorded as "Node A", the second as "Node B", and so on.

[0066] Step S1342: For each spatial topology node, query the double association mapping result and extract the visual semantic features corresponding to the spatial topology node, including the category identifier of the environmental objects at the spatial topology node, the spatial relative relationship identifier between objects, and the description of the appearance features of the objects.

[0067] For each spatial topology node, the bi-association mapping result is queried to extract its corresponding visual semantic features. In the e-commerce warehousing and logistics center, based on the recorded identification information of the spatial topology nodes, the corresponding visual semantic features are searched in the bi-association mapping result. For each node, the category identifier of the objects in its environment, such as shelves, goods, etc.; the spatial relative relationship identifier between objects, such as front-back, left-right, distance, etc.; and the appearance feature description of the objects, such as color, shape, texture, etc. For example, for a spatial topology node located near a shelf, the category identifier of the shelf is extracted as "shelf", its spatial relative relationship identifier with the adjacent aisle is "located at a certain distance to the left of the aisle", and the appearance feature description of the shelf is "blue, multi-layered rectangular shelf".

[0068] Step S1343: Perform a completeness analysis on the extracted visual semantic features to confirm whether they contain complete category identifiers, spatial relative relationship identifiers, and appearance feature descriptions.

[0069] Completeness analysis is performed on the extracted visual semantic features. In the e-commerce warehousing and logistics center, the visual semantic features corresponding to each spatial topology node are checked to see if they contain complete category identifiers, spatial relationship identifiers, and appearance feature descriptions. If a visual semantic feature lacks important information, such as the absence of an object category identifier or unclear spatial relationship, then the visual semantic feature is incomplete. For example, if only the appearance features of an object are recorded without specifying its category and spatial relationship with other objects, then the visual semantic feature is incomplete. Through completeness analysis, spatial topology nodes with complete visual semantic features are selected.

[0070] Step S1344: Mark the spatial topological node with complete visual semantic features as a candidate semantic anchor point. The visual semantic features corresponding to the candidate semantic anchor point can effectively characterize the spatial position of the node.

[0071] Spatial topological nodes with complete visual semantic features are marked as candidate semantic anchors. In e-commerce warehousing and logistics centers, spatial topological nodes with complete visual semantic features can accurately describe the environmental conditions at the node, thus effectively representing the node's spatial location. These nodes are marked as candidate semantic anchors for further screening and determination of the final semantic anchors. For example, if the visual semantic features of a spatial topological node completely describe the type of shelf at that node, its spatial relationship with surrounding aisles and other shelves, and the appearance features of the shelves, then that node is marked as a candidate semantic anchor.

[0072] Step S1345: Perform distribution density analysis on the candidate semantic anchor point, calculate the topological network distance between adjacent candidate semantic anchor points, and ensure that the topological network distance between adjacent candidate semantic anchor points is within a preset reasonable range.

[0073] Distribution density analysis is performed on candidate semantic anchors. In an e-commerce warehousing and logistics center, the topological network distance between adjacent candidate semantic anchors is calculated. This distance can be determined based on the connection relationships and distance information between nodes in the spatial topology data. For example, the topological network distance can be determined by calculating the number of spatial topological nodes traversed between two adjacent candidate semantic anchors or the actual distance. Then, it is determined whether the topological network distance between adjacent candidate semantic anchors is within a preset reasonable range. The preset reasonable range is determined based on the robot's navigation needs and environmental characteristics. If the distance is too far, the robot may easily lose its way during navigation; if the distance is too close, it can increase the complexity and computational load of navigation. Through the above analysis, the distribution density of candidate semantic anchors is ensured to be moderate.

[0074] Step S1346: Based on the distribution density analysis results, supplement or adjust the candidate semantic anchors. Among the adjacent candidate semantic anchors whose topological network distance exceeds the preset reasonable range, select spatial topological nodes with complete visual semantic features as supplementary candidate semantic anchors, and use the finally determined candidate semantic anchors as semantic anchors.

[0075] Based on the distribution density analysis results, candidate semantic anchors are supplemented or adjusted. In e-commerce warehousing and logistics centers, if the topological network distance between adjacent candidate semantic anchors exceeds a preset reasonable range, a node with complete visual semantic features is selected from the spatial topological nodes between these two nodes as a supplementary candidate semantic anchor. For example, if the distance between two adjacent candidate semantic anchors is too far, and there is a spatial topological node with clear visual semantic features in the passage between them, it is selected as a supplementary candidate semantic anchor. After supplementation and adjustment, the finally determined candidate semantic anchors are used as semantic anchors. These semantic anchors can provide accurate positioning and guidance information for the robot during navigation.

[0076] Step S135: Arrange the determined semantic anchors in sequence according to the connection order of the spatial topology nodes in the spatial topology path to form a semantic anchor sequence for the robot from the current position to the target position.

[0077] The determined semantic anchors are arranged sequentially according to the connection order of the spatial topology nodes in the spatial topology path. In an e-commerce warehousing and logistics center, based on the planning of the spatial topology path, the semantic anchors are arranged according to their order in the path. For example, starting from the initial spatial topology node, the semantic anchors are arranged sequentially along the path, forming an ordered sequence. This sequence represents the key nodes in the robot's navigation path from its current position to the target position, and each semantic anchor corresponds to a spatial location with clear visual semantic features.

[0078] Step S136: Associate the semantic anchor sequence with the spatial location information corresponding to each semantic anchor to generate a semantic anchor navigation path containing the semantic anchor sequence.

[0079] The semantic anchor sequence is associated with the spatial location information corresponding to each semantic anchor. In an e-commerce warehousing and logistics center, each semantic anchor has its corresponding spatial location information, such as coordinates and its relative position to the surrounding environment. This spatial location information is bound and associated with each anchor in the semantic anchor sequence to form a complete semantic anchor navigation path. This semantic anchor navigation path not only contains the sequence information of the semantic anchors but also the specific spatial location of each anchor, providing detailed guidance for the robot's navigation and enabling the robot to accurately move from its current position to its target position along the path.

[0080] Step S140: Obtain real-time visual semantic features during the navigation process through the robot's real-time visual acquisition module. Based on the matching relationship between the real-time visual semantic features and the double association mapping result, dynamically update the semantic anchor navigation path to obtain the updated semantic anchor navigation path.

[0081] In an e-commerce warehousing and logistics center, the robot begins to navigate according to the semantic anchor navigation path. During the navigation process, it needs to acquire environmental information in real time and dynamically update the path.

[0082] Step S141: Start the robot's real-time vision acquisition module, which acquires real-time environmental visual images during the robot's navigation process according to a preset acquisition frequency.

[0083] The robot's real-time vision acquisition module is activated. In the e-commerce warehousing and logistics center, this module acquires real-time environmental visual images during the robot's navigation process at a preset acquisition frequency. The preset acquisition frequency is determined based on the robot's moving speed and the likelihood of environmental changes. For example, if the robot moves quickly or the environment changes frequently, the acquisition frequency will be set higher; conversely, it can be set lower. The real-time vision acquisition module will continuously operate, constantly acquiring images of the robot's surrounding environment to promptly reflect changes in the environment.

[0084] Step S142: Extract real-time visual semantic features from the real-time environment visual image. The real-time visual semantic features include the category identifier of objects in the real-time environment, the spatial relative relationship identifier between objects, and the description of the appearance features of objects.

[0085] Extracting real-time visual semantic features from real-time environmental visual images. In e-commerce warehousing and logistics centers, the real-time acquired environmental visual images are processed in a manner similar to the previous process of extracting visual semantic features from continuous sets of visual information.

[0086] Step S1421: Perform image region segmentation on the real-time environment visual image, dividing it into multiple non-overlapping real-time object image regions, with each real-time object image region corresponding to a real-time environment object.

[0087] Image region segmentation is performed on real-time environmental visual images. In e-commerce warehousing and logistics centers, image segmentation algorithms are used to divide real-time environmental visual images into multiple non-overlapping real-time object image regions. For example, shelves, goods, and other operating robots are divided into different real-time object image regions, with each region corresponding to a real-time environmental object. Through this segmentation, the complex real-time environmental image is decomposed into independent object images, facilitating subsequent feature extraction and analysis.

[0088] Step S1422: Perform real-time object category recognition on each real-time object image region, determine the category of the real-time environment object corresponding to the real-time object image region, and generate a category identifier for the real-time environment object.

[0089] Real-time object category recognition is performed on each real-time object image region. In an e-commerce warehousing and logistics center, a deep learning-based image recognition model is used to analyze each real-time object image region. If a real-time object image region corresponds to a shelf, the model will identify the object as a shelf and generate a category identifier representing the shelf; if it is goods, a category identifier representing the goods will be generated; if it is another type of robot, a category identifier representing the robot will be generated. These category identifiers can be represented by specific codes for convenient subsequent processing and management.

[0090] Step S1423: Analyze the position coordinates of each real-time object image region in the real-time environment visual image, calculate the position distance and orientation angle between different real-time object image regions, determine the spatial relative relationship between real-time objects, and generate spatial relative relationship identifiers between objects.

[0091] The system analyzes the positional coordinates of each real-time object image region within the real-time environmental visual image. In an e-commerce warehousing and logistics center scenario, the center position coordinates of each real-time object image region are determined based on the image's coordinate system. Then, the spatial relative relationships between real-time objects are determined by calculating the distance and angle between the center position coordinates of different real-time object image regions. For example, the straight-line distance between a real-time shelf and adjacent real-time goods, as well as the orientation angle of the real-time goods relative to the real-time shelf, are calculated to determine their spatial relative relationships and generate corresponding spatial relative relationship identifiers. These identifiers can represent the relative positional relationships between objects, such as front-back, left-right, and up-down, as well as the degree of distance between them.

[0092] Step S1424: Extract the real-time appearance features of each real-time object image region, describe the real-time appearance features, and form a description of the appearance features of the real-time environment object. The real-time appearance features include the color distribution features, texture distribution features, and contour shape features of the real-time environment object.

[0093] Real-time appearance features are extracted for each real-time object image region. In e-commerce warehousing and logistics centers, image processing techniques are used to extract appearance features for real-time object image regions such as real-time shelves and real-time goods. For real-time shelves, color distribution features are extracted, such as whether the overall color is blue or gray, and whether the color distribution on the shelf surface is uniform; texture distribution features are extracted, such as whether the shelf surface is smooth or textured; and outline shape features are extracted, such as whether the shelf is rectangular, square, or other shapes. For real-time goods, color, texture, and outline shape features are also extracted. Then, the above real-time appearance features are described to form a description of the appearance features of real-time environmental objects. For example, a real-time shelf is described as "a blue, rectangular real-time shelf with a regular textured surface," and real-time goods are described as "a red, cube-shaped real-time goods with a smooth surface."

[0094] Step S1425: Associate the category identifier corresponding to each real-time environmental object, the spatial relative relationship identifier between the real-time environmental object and other real-time environmental objects, and the appearance feature description of the real-time environmental object to form the local real-time visual semantic feature corresponding to the real-time environmental object.

[0095] The system associates the category identifier of each real-time environmental object, its spatial relationship with other real-time environmental objects, and its appearance description. In an e-commerce warehousing and logistics center, for a real-time shelf, its category identifier, its spatial relationship with surrounding real-time goods and other real-time shelves (e.g., located a certain distance to the left of a piece of real-time goods), and its appearance description (e.g., a blue, rectangular real-time shelf with a regularly textured surface) are associated to form the local real-time visual semantic features corresponding to that shelf. These local real-time visual semantic features comprehensively describe the characteristics and positional relationships of the real-time object in the real-time environment.

[0096] Step S1426: Summarize the local real-time visual semantic features of all real-time objects in the real-time environment visual image to form the overall real-time visual semantic features corresponding to the real-time environment visual image. The overall real-time visual semantic features are the real-time visual semantic features extracted from the real-time environment visual image.

[0097] This involves summarizing the local real-time visual semantic features of all real-time objects in a real-time environmental visual image. In an e-commerce warehousing and logistics center, the local real-time visual semantic features of all real-time shelves, goods, and other robots in a real-time environmental visual image are integrated and summarized to form the overall real-time visual semantic features corresponding to that image. These overall real-time visual semantic features include information such as the category, spatial relationships, and appearance features of all real-time objects in the image, and can completely describe the scene presented by the real-time environmental image; that is, the real-time visual semantic features extracted from the real-time environmental visual image.

[0098] Step S143: Query the double association mapping result, match the extracted real-time visual semantic features with the visual semantic features in the double association mapping result, and determine the spatial topology node corresponding to the real-time visual semantic features.

[0099] The process involves querying the bi-association mapping results and matching the extracted real-time visual semantic features with the visual semantic features in the bi-association mapping results. Within the e-commerce warehousing and logistics center, each real-time visual semantic feature is compared with the visual semantic features recorded in the bi-association mapping results. By comparing information such as category identifiers, spatial relative relationship identifiers, and appearance feature descriptions, the spatial topology node corresponding to the visual semantic feature that best matches the real-time visual semantic feature is identified. For example, if the scene described by the real-time visual semantic feature is very similar to the scene described by a certain visual semantic feature in the bi-association mapping results, the spatial topology node corresponding to that visual semantic feature is determined to be the spatial topology node corresponding to the real-time visual semantic feature.

[0100] Step S144: Compare the spatial topology node with the spatial topology node corresponding to the semantic anchor point that should be reached in the navigation path of the semantic anchor point, and analyze the differences in visual semantic features between the two.

[0101] The spatial topology node is compared with the spatial topology node corresponding to the semantic anchor point that the robot should reach in the semantic anchor point navigation path. In an e-commerce warehousing and logistics center, the spatial topology node corresponding to the real-time visual semantic features is compared with the spatial topology node corresponding to the semantic anchor point that the robot should reach in the semantic anchor point navigation path. The differences in visual semantic features between the two are analyzed, including differences in category identification, spatial relative relationship identification, and appearance feature description. For example, if the shelf category at the real-time spatial topology node is different from the shelf category at the current semantic anchor point, or if the spatial relative relationship between objects has changed, it indicates a difference in visual semantic features. The above differences may be due to environmental changes (such as the movement of goods, the installation of new shelves, etc.) or navigation errors.

[0102] Step S145: Based on the difference in visual semantic features, correct the reference visual semantic features of the corresponding spatial topology nodes in the double association mapping result so that the double association mapping result is consistent with the real-time environment.

[0103] Based on differences in visual semantic features, the reference visual semantic features of the corresponding spatial topology nodes in the dual-association mapping results are corrected. In e-commerce warehousing and logistics centers, if a difference in visual semantic features is found between a real-time spatial topology node and the spatial topology node corresponding to the current semantic anchor point, the reference visual semantic features of that spatial topology node in the dual-association mapping results are corrected according to the real-time visual semantic features. For example, if the color of a shelf changes in the real-time environment, the color description information in the reference visual semantic features of the spatial topology node where that shelf is located in the dual-association mapping results is updated. Through the above correction operations, the dual-association mapping results are kept consistent with the real-time environment, ensuring the accuracy of subsequent navigation.

[0104] Step S146: Based on the corrected double association mapping results, re-evaluate each semantic anchor in the semantic anchor navigation path, adjust the spatial position of the semantic anchor or add new semantic anchors.

[0105] Based on the revised bi-association mapping results, each semantic anchor in the semantic anchor navigation path is re-evaluated. In e-commerce warehousing and logistics centers, since the bi-association mapping results have been updated, each semantic anchor in the semantic anchor navigation path needs to be re-examined and evaluated. If the visual semantic features of the spatial topology node corresponding to a semantic anchor have changed significantly, the spatial position of that semantic anchor may need to be adjusted. For example, if the shelf originally corresponding to a semantic anchor is removed, other spatial topology nodes with visual semantic features near that location need to be re-identified as new semantic anchors. Simultaneously, if the distribution of semantic anchors in certain areas of the path is found to be unreasonable, such as excessive spacing or a lack of semantic anchors at certain key locations, new semantic anchors need to be added. Through these methods, the semantic anchor navigation path becomes more adaptable to changes in the real-time environment.

[0106] Step S147: Arrange the adjusted semantic anchors according to the navigation order to form the updated semantic anchor navigation path.

[0107] The adjusted semantic anchors are arranged according to the navigation sequence. In the e-commerce warehousing and logistics center, the adjusted semantic anchors are reordered based on the robot's navigation direction and path planning. The semantic anchors are arranged sequentially from the current position to the target position, forming an updated semantic anchor navigation path. This semantic anchor navigation path takes into account changes in the real-time environment, providing the robot with more accurate navigation guidance and ensuring that the robot can successfully reach the target position from its current position.

[0108] Step S150: Output the robot's final navigation execution path according to the updated semantic anchor navigation path, and drive the robot to complete the navigation operation according to the final navigation execution path. The final navigation execution path contains motion control instructions corresponding to each semantic anchor.

[0109] In the e-commerce warehousing and logistics center, the updated semantic anchor navigation path has been obtained. Next, the robot's final navigation execution path will be output based on this and the robot will be driven to complete the navigation operation.

[0110] Step S151: Parse the updated semantic anchor navigation path and extract the semantic anchor sequence and the spatial location information corresponding to each semantic anchor.

[0111] The updated semantic anchor navigation path is parsed to extract the semantic anchor sequence and the spatial location information of each semantic anchor. In an e-commerce warehousing and logistics center, the updated semantic anchor navigation path is parsed to extract the recorded semantic anchor sequence and the spatial location information of each semantic anchor. The semantic anchor sequence is a series of semantic anchors arranged in navigation order, and each semantic anchor has its corresponding spatial location information, such as coordinates and its relative position to the surrounding environment.

[0112] Step S152: Calculate the spatial distance and relative orientation between two adjacent semantic anchors. The spatial distance is the straight-line distance between the corresponding spatial positions of the two semantic anchors, and the relative orientation is the direction of the latter semantic anchor relative to the former semantic anchor.

[0113] Calculate the spatial distance and relative orientation between two adjacent semantic anchors. In an e-commerce warehousing and logistics center, the spatial distance between two adjacent semantic anchors is calculated based on the spatial location information corresponding to each semantic anchor. This spatial distance is the straight-line distance between the corresponding spatial locations of the two semantic anchors and can be calculated using spatial coordinates. Simultaneously, the relative orientation of the latter semantic anchor relative to the former semantic anchor is calculated, that is, the direction of the latter semantic anchor in space relative to the direction of the former semantic anchor. For example, the latter semantic anchor may be directly in front of, to the left front of, or to the right rear of the former semantic anchor. This spatial distance and relative orientation information is crucial for generating motion control commands for the robot.

[0114] Step S153: Based on the robot's motion characteristic parameters, convert the spatial distance and relative orientation between adjacent semantic anchor points into motion control commands. These motion control commands include the robot's moving speed, turning direction, and movement duration.

[0115] Based on the robot's motion characteristic parameters, the spatial distance and relative orientation between adjacent semantic anchor points are converted into motion control commands. In e-commerce warehousing and logistics centers, the robot's motion characteristic parameters include its maximum speed, minimum speed, and turning capability. Based on the spatial distance and relative orientation between adjacent semantic anchor points, combined with the robot's motion characteristic parameters, the required speed, turning direction, and movement time for the robot to move from one semantic anchor point to the next are calculated. For example, if the spatial distance is large and the robot's maximum speed allows, a higher speed can be set; if a turn is required in the relative orientation, the appropriate turning direction and angle are determined based on the robot's turning capability. This information is then combined into motion control commands to control the robot's movement.

[0116] Step S154: Arrange the motion control instructions corresponding to each semantic anchor point in sequence according to the order of the semantic anchor point sequence to form a motion control instruction sequence.

[0117] Following the sequence of semantic anchors, the motion control instructions corresponding to each semantic anchor are arranged sequentially. In an e-commerce warehousing and logistics center, the motion control instructions corresponding to each semantic anchor are ordered according to the order of the semantic anchors in the sequence. Starting from the first semantic anchor, its corresponding motion control instructions are arranged sequentially, forming an ordered sequence of motion control instructions. This sequence of motion control instructions represents a series of motion operations that the robot needs to perform during the entire navigation process from its current position to the target position.

[0118] Step S155: Associate the motion control command sequence with the corresponding semantic anchor point identifier to generate a final navigation execution path containing the motion control command sequence. Each motion control command in the final navigation execution path corresponds to a semantic anchor point.

[0119] The motion control command sequence is associated with its corresponding semantic anchor point identifier. In an e-commerce warehousing and logistics center, each motion control command is added with its corresponding semantic anchor point identifier, indicating which semantic anchor point the motion control command is set up to move the robot from to the next semantic anchor point. Through this association, a final navigation execution path containing the motion control command sequence is generated. This final navigation execution path not only includes the robot's motion control commands but also clearly defines the semantic anchor point corresponding to each motion control command, enabling the robot to navigate accurately according to the path.

[0120] Step S156: Decompose the motion control instruction sequence in the final navigation execution path into a single independent motion control instruction. Each independent motion control instruction corresponds to a motion process from the current semantic anchor point to the next semantic anchor point.

[0121] The motion control command sequence in the final navigation execution path is broken down into individual motion control commands. In e-commerce warehousing and logistics centers, each command in the motion control command sequence is separated into an independent motion control command. Each independent motion control command corresponds to a motion process from the current semantic anchor point to the next semantic anchor point. For example, the first independent motion control command controls the robot to move from the initial semantic anchor point to the second semantic anchor point, the second independent motion control command controls the robot to move from the second semantic anchor point to the third semantic anchor point, and so on.

[0122] Step S157: Send the first independent motion control command to the robot's motion control module. The motion control module controls the robot's drive components to start according to the movement speed, turning direction and movement duration in the motion control command, so that the robot moves towards the first semantic anchor point.

[0123] The first independent motion control command is sent to the robot's motion control module. In an e-commerce warehousing and logistics center, after receiving the first independent motion control command, the motion control module, based on information such as the movement speed, turning direction, and movement duration in the command, controls the robot's drive components to start. These drive components can be motors, wheels, etc. By controlling the operation of these components, the robot begins to move towards the first semantic anchor point according to the command's requirements. For example, if the command requires the robot to move forward in a straight line at a certain speed, the motion control module will control the motors to operate at the corresponding speed, driving the wheels to rotate and making the robot move forward.

[0124] Step S158: During the process of the robot moving towards the first semantic anchor point, the real-time vision acquisition module continuously acquires real-time environmental visual images, extracts real-time visual semantic features, and confirms the spatial position corresponding to the first semantic anchor point by matching the real-time visual semantic features with the double association mapping results.

[0125] As the robot moves toward the first semantic anchor point, the real-time visual acquisition module continues to operate. In the e-commerce warehousing and logistics center, the real-time visual acquisition module continuously acquires real-time environmental visual images at a preset acquisition frequency and extracts real-time visual semantic features from them. Then, these real-time visual semantic features are matched with the results of the dual-association mapping. By comparing information such as category labels, spatial relative relationship labels, and appearance feature descriptions, it is confirmed whether the robot has reached the spatial location corresponding to the first semantic anchor point. If the real-time visual semantic features have a high degree of matching with the visual semantic features corresponding to the first semantic anchor point, it indicates that the robot has reached the spatial location corresponding to that semantic anchor point.

[0126] Step S159: After confirming that the robot has reached the first semantic anchor point, send the second independent motion control command to the motion control module to control the robot to move towards the second semantic anchor point.

[0127] Once the robot has reached the first semantic anchor point, a second independent motion control command is sent to the motion control module. In the e-commerce warehousing and logistics center, upon receiving the second independent motion control command, the motion control module, based on information such as the movement speed, turning direction, and movement duration in the command, controls the robot's drive components to adjust their operating state, causing the robot to begin moving towards the second semantic anchor point. For example, if the second semantic anchor point is to the right of the first semantic anchor point and requires a certain turning angle, the motion control module will control the robot's turning components to perform a turning operation, and then move to the second semantic anchor point at the specified speed.

[0128] Step S1510: Repeat the above process, sending each independent motion control command in sequence to control the robot to pass through each semantic anchor point. When the robot reaches the semantic anchor point corresponding to the target position according to the last motion control command, the motion control module stops executing the motion control command, and the robot completes the navigation operation from the current position to the target position.

[0129] Repeat the above process, sending each independent motion control command sequentially. In the e-commerce warehousing and logistics center, as the robot passes each semantic anchor point in turn, the corresponding independent motion control command is continuously sent to the motion control module, controlling the robot to move and turn according to the command. When the robot reaches the semantic anchor point corresponding to the target location according to the last motion control command, the motion control module stops executing the motion control command. At this point, the robot has completed the navigation operation from the current position to the target position and successfully reached its destination, such as the designated goods storage point or work area, and can continue to perform subsequent tasks such as goods handling and sorting.

[0130] Figure 2 Schematic diagrams are shown of exemplary hardware and software components of a vision-based robot navigation system 100 that can implement the ideas of this application, according to some embodiments of this application. For example, a processor 120 can be used in the vision-based robot navigation system 100 and to perform the functions described in this application.

[0131] The vision-based robot navigation system 100 can be a general-purpose server or a special-purpose server, both of which can be used to implement the vision-based robot navigation method of this application. Although only one server is shown in this application, for convenience, the functions described in this application can be implemented in a distributed manner on multiple similar platforms to balance the load.

[0132] For example, the vision-based robot navigation system 100 may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and various forms of storage media 140, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the vision-based robot navigation system 100 may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The methods of this application can be implemented according to these program instructions. The vision-based robot navigation system 100 also includes an I / O interface 150 between the computer and other input / output devices.

[0133] For ease of explanation, only one processor is described in the vision-based robot navigation system 100. However, it should be noted that the vision-based robot navigation system 100 of this application may also include multiple processors, and therefore the steps performed by one processor as described in this application may also be performed jointly or individually by multiple processors. For example, if the processor of the vision-based robot navigation system 100 performs steps A and B, it should be understood that steps A and B may also be performed jointly by two different processors or individually by one processor. For example, the first processor performs step A, the second processor performs step B, or the first processor and the second processor jointly perform steps A and B.

[0134] Furthermore, this embodiment of the invention also provides a readable storage medium, wherein computer-executable instructions are preset in the readable storage medium, and when the processor executes the computer-executable instructions, the above-mentioned vision-based robot navigation method is implemented.

[0135] It should be noted that, in order to simplify the description of the present invention and thus help to understand one or more embodiments of the invention, multiple features may sometimes be grouped into one embodiment, drawing or description thereof in the foregoing description of the embodiments of the present invention.

Claims

1. A robot navigation method based on vision recognition, characterized in that, The method includes: Acquire a continuous set of visual information about the robot's surrounding environment. The continuous set of visual information includes environmental visual images from different directions along the robot's movement direction. The environmental visual images record the appearance features and spatial arrangement of objects in the environment. Establish an association mapping between the continuous visual information set and the robot's preset navigation area to obtain a double association mapping result. The double association mapping result is used to represent the correspondence between the visual semantic features in the continuous visual information and the spatial topological nodes in the robot's preset navigation area. Based on the dual-association mapping result, a semantic anchor navigation path for the robot is generated. The semantic anchor navigation path contains a sequence of semantic anchors from the robot's current position to the target position. Each semantic anchor corresponds to a spatial position with visual semantic features within a preset navigation area. The robot's real-time visual acquisition module acquires real-time visual semantic features during the navigation process. Based on the matching relationship between the real-time visual semantic features and the double association mapping results, the semantic anchor navigation path is dynamically updated to obtain the updated semantic anchor navigation path. The robot's final navigation execution path is output based on the updated semantic anchor navigation path, and the robot is driven to complete the navigation operation according to the final navigation execution path. The final navigation execution path contains motion control instructions corresponding to each semantic anchor. The process of establishing the association mapping between the continuous visual information set and the robot's preset navigation area to obtain a double association mapping result includes: Visual semantic features are extracted from each environmental visual image in the continuous visual information set. The visual semantic features include category identifiers of environmental objects, spatial relative relationship identifiers between objects, and descriptions of the appearance features of objects. Acquire spatial topology data of the robot's preset navigation area. The spatial topology data includes multiple spatial topology nodes divided within the preset navigation area, and the connection relationship between each spatial topology node and its adjacent spatial topology nodes. Each spatial topology node is labeled with corresponding reference visual semantic features, which include the category identifier of typical environmental objects at the spatial topology node, the spatial relative relationship identifier between objects, and the description of the object's appearance features. The visual semantic features extracted from the continuous visual information set are associated with the reference visual semantic features of each spatial topology node to build a model of the relationship between visual semantic features and spatial topology nodes. The association model is optimized by adjusting the corresponding weights of visual semantic features and spatial topology nodes, so that each visual semantic feature can be accurately matched to the corresponding spatial topology node. The optimized visual semantic features and their correspondence with spatial topology nodes are organized according to a preset format to generate a double association mapping result that represents the correspondence between visual semantic features in continuous visual information and spatial topology nodes in a preset navigation area.

2. The robot navigation method based on vision recognition according to claim 1, characterized in that, The acquisition of the continuous visual information set of the robot's surrounding environment includes: The robot is equipped with multiple visual acquisition devices, which are respectively oriented towards the forward, lateral, and oblique directions of the robot's movement direction to form a multi-view acquisition array. Each visual acquisition device in the multi-view acquisition array is controlled to synchronously acquire environmental visual data at the same time interval to obtain multiple sets of initial environmental visual images. Each set of initial environmental visual images corresponds to an acquisition time point and contains environmental visual content from different viewpoints. For each set of initial environmental visual images, the overlapping area of ​​the viewpoint is identified to determine the overlapping area between the initial environmental visual images acquired by different visual acquisition devices. The overlapping area contains the visual content of the same environmental object from different viewpoints. Based on the overlapping region, multiple initial environmental visual images are subjected to viewpoint fusion processing to integrate environmental visual content from different viewpoints into an environmental visual image with a unified viewpoint. Redundant information is removed from the fused environmental visual image, duplicate environmental object visual content is deleted, and the complete appearance features and spatial arrangement relationship of each environmental object are preserved. Multiple sets of environmental visual images, after removing redundancy, are arranged in chronological order of acquisition time to form a continuous visual information set containing environmental visual content from different directions along the robot's movement. Each environmental visual image in the continuous visual information set records the appearance features and spatial arrangement of objects in the environment.

3. The robot navigation method based on vision recognition according to claim 1, characterized in that, Extracting visual semantic features from each environmental visual image in the continuous visual information set includes: For each environmental visual image in the continuous visual information set, image region segmentation is performed to divide the image into multiple non-overlapping object image regions, and each object image region corresponds to an environmental object. For each object image region, perform object category recognition to determine the category of the environmental object corresponding to that object image region and generate the category label of the environmental object; Analyze the position coordinates of each object image region in the environmental visual image, calculate the positional distance and orientation angle between different object image regions, determine the spatial relative relationship between objects, and generate spatial relative relationship identifiers between objects. Extract the appearance features of each object image region. The appearance features include the object's color distribution features, texture distribution features, and contour shape features. Describe the appearance features to form an object appearance feature description. The category identifier corresponding to each environmental object, the spatial relative relationship identifier between the environmental object and other objects, and the appearance feature description of the environmental object are associated to form the local visual semantic features corresponding to the environmental object. The local visual semantic features of all environmental objects in each environmental visual image are summarized to form the overall visual semantic features corresponding to that environmental visual image. The overall visual semantic features are the visual semantic features extracted from that environmental visual image.

4. The robot navigation method based on vision recognition according to claim 1, characterized in that, The generation of the robot's semantic anchor navigation path based on the dual-association mapping result includes: The results of the dual association mapping are analyzed, and the correspondence between visual semantic features and spatial topology nodes is extracted. The visual semantic features corresponding to the current position of the robot are determined, and the corresponding spatial topology nodes are matched through the visual semantic features, which are used as the initial spatial topology nodes of the current position of the robot. Obtain the target visual semantic features corresponding to the robot navigation target position, and match the target visual semantic features to the corresponding spatial topology nodes through the double association mapping results, which serve as the target spatial topology nodes for the robot target position; Based on the spatial topology data of the preset navigation area, a spatial topology path is planned from the initial spatial topology node to the target spatial topology node. The spatial topology path contains multiple spatial topology nodes connected in sequence. For each spatial topology node in the spatial topology path, its corresponding visual semantic features are obtained through the double association mapping result, and the spatial topology nodes with visual semantic features are determined as semantic anchors. According to the connection order of the spatial topology nodes in the spatial topology path, the determined semantic anchors are arranged in sequence to form a semantic anchor sequence for the robot from the current position to the target position; The semantic anchor sequence is associated with the spatial location information corresponding to each semantic anchor to generate a semantic anchor navigation path containing the semantic anchor sequence.

5. The robot navigation method based on vision recognition according to claim 4, characterized in that, For each spatial topology node in the spatial topology path, its corresponding visual semantic features are obtained through a double association mapping result. Spatial topology nodes with visual semantic features are then identified as semantic anchor points, including: Traverse each spatial topology node in the spatial topology path and record the identification information of each spatial topology node; For each spatial topology node, query the double association mapping results and extract the visual semantic features corresponding to the spatial topology node, including the category identifier of the environmental objects at the spatial topology node, the spatial relative relationship identifier between objects, and the description of the object appearance features. Perform a completeness analysis on the extracted visual semantic features to confirm whether they contain complete category identifiers, spatial relationship identifiers, and appearance feature descriptions; Spatial topological nodes with complete visual semantic features are marked as candidate semantic anchors, and the visual semantic features corresponding to the candidate semantic anchors can effectively characterize the spatial position of the node. The distribution density analysis of the candidate semantic anchors is performed to calculate the topological network distance between adjacent candidate semantic anchors, and the topological network distance between adjacent candidate semantic anchors is kept within a preset reasonable range. Based on the distribution density analysis results, candidate semantic anchors are supplemented or adjusted. Among adjacent candidate semantic anchors whose topological network distance exceeds the preset reasonable range, spatial topological nodes with complete visual semantic features are selected as supplementary candidate semantic anchors, and the finally determined candidate semantic anchors are used as semantic anchors.

6. The robot navigation method based on vision recognition according to claim 1, characterized in that, The process involves acquiring real-time visual semantic features during navigation through the robot's real-time visual acquisition module, and dynamically updating the semantic anchor navigation path based on the matching relationship between the real-time visual semantic features and the dual-association mapping results to obtain the updated semantic anchor navigation path, including: The robot's real-time vision acquisition module is activated, and the real-time vision acquisition module acquires real-time environmental visual images during the robot's navigation process according to a preset acquisition frequency. Real-time visual semantic features are extracted from the real-time environment visual image. The real-time visual semantic features include the category identifier of objects in the real-time environment, the spatial relative relationship identifier between objects, and the description of the appearance features of objects. Query the double association mapping result, match the extracted real-time visual semantic features with the visual semantic features in the double association mapping result, and determine the spatial topology node corresponding to the real-time visual semantic features; Compare the spatial topology node with the spatial topology node corresponding to the semantic anchor point that should be reached in the navigation path, and analyze the differences in visual semantic features between the two. Based on the differences in visual semantic features, the reference visual semantic features of the corresponding spatial topology nodes in the double association mapping results are corrected so that the double association mapping results are consistent with the real-time environment. Based on the corrected double association mapping results, each semantic anchor in the semantic anchor navigation path is re-evaluated, and the spatial position of the semantic anchor is adjusted or new semantic anchors are added. The adjusted semantic anchors are arranged in navigation order to form the updated semantic anchor navigation path.

7. The robot navigation method based on vision recognition according to claim 6, characterized in that, Extracting real-time visual semantic features from the real-time environmental visual image includes: The real-time environment visual image is segmented into multiple non-overlapping real-time object image regions, with each real-time object image region corresponding to a real-time environment object. For each real-time object image region, perform real-time object category recognition to determine the category of the real-time environment object corresponding to the real-time object image region and generate a category identifier for the real-time environment object. Analyze the position coordinates of each real-time object image region in the real-time environment visual image, calculate the position distance and orientation angle between different real-time object image regions, determine the spatial relative relationship between real-time objects, and generate spatial relative relationship identifiers between objects. Extract the real-time appearance features of each real-time object image region, describe the real-time appearance features, and form a description of the appearance features of the real-time environment object. The real-time appearance features include the color distribution features, texture distribution features, and contour shape features of the real-time environment object. The category identifier corresponding to each real-time environmental object, the spatial relative relationship identifier between the real-time environmental object and other real-time environmental objects, and the appearance feature description of the real-time environmental object are associated to form the local real-time visual semantic feature corresponding to the real-time environmental object. The local real-time visual semantic features of all real-time objects in the real-time environment visual image are summarized to form the overall real-time visual semantic features corresponding to the real-time environment visual image. The overall real-time visual semantic features are the real-time visual semantic features extracted from the real-time environment visual image.

8. The robot navigation method based on vision recognition according to claim 1, characterized in that, The step of outputting the robot's final navigation execution path based on the updated semantic anchor navigation path, and driving the robot to complete the navigation operation according to the final navigation execution path, includes: The updated semantic anchor navigation path is parsed, and the semantic anchor sequence and the spatial location information corresponding to each semantic anchor are extracted. Calculate the spatial distance and relative orientation between two adjacent semantic anchors. The spatial distance is the straight-line distance between the corresponding spatial positions of the two semantic anchors, and the relative orientation is the direction of the latter semantic anchor relative to the former semantic anchor. Based on the robot's motion characteristic parameters, the spatial distance and relative orientation between adjacent semantic anchor points are converted into motion control commands, which include the robot's moving speed, turning direction, and movement duration. According to the order of the semantic anchor sequence, the motion control instructions corresponding to each semantic anchor are arranged sequentially to form a motion control instruction sequence; The motion control command sequence is associated with the corresponding semantic anchor point identifier to generate a final navigation execution path containing the motion control command sequence. Each motion control command in the final navigation execution path corresponds to a semantic anchor point. The motion control instruction sequence in the final navigation execution path is decomposed into a single independent motion control instruction, and each independent motion control instruction corresponds to a motion process from the current semantic anchor point to the next semantic anchor point. The first independent motion control command is sent to the robot's motion control module. The motion control module controls the robot's drive components to start according to the movement speed, turning direction and movement duration in the motion control command, so that the robot moves towards the first semantic anchor point. As the robot moves toward the first semantic anchor point, the real-time visual acquisition module continuously acquires real-time environmental visual images and extracts real-time visual semantic features. By matching the real-time visual semantic features with the double association mapping results, the robot confirms that it has reached the spatial position corresponding to the first semantic anchor point. Once it is confirmed that the robot has reached the first semantic anchor point, a second independent motion control command is sent to the motion control module to control the robot to move towards the second semantic anchor point. Repeat the above process, sending each independent motion control command in sequence to control the robot to pass through each semantic anchor point. When the robot reaches the semantic anchor point corresponding to the target position according to the last motion control command, the motion control module stops executing the motion control command, and the robot completes the navigation operation from the current position to the target position.

9. A robot navigation system based on vision recognition, characterized in that, The system includes a processor and a memory, the memory being connected to the processor. The memory is used to store programs, instructions, or code, and the processor is used to execute the programs, instructions, or code in the memory to implement the vision-based robot navigation method according to any one of claims 1-8.

Images