Shelf identification system and method of mobile robot
Through deep learning technology combined with camera and lidar data, the problem of positioning errors in shelf recognition is solved, and the accurate shelf recognition and positioning of mobile robots in complex environments is achieved, thereby improving the stability and efficiency of the system.
Patent Information
- Application Number
- CN202411185583.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing shelf identification technology has positioning errors in complex environments, especially errors caused by interference points on legs of metal shelves, which affects the accurate identification and operation of mobile robots.
Combining the environmental images collected by the camera and the point cloud data collected by the lidar, feature vectors are extracted and correlated analysis is performed through deep learning technology, and a classifier is used to determine whether the mobile robot is located in the center of the shelf, improving the accuracy and efficiency of identification and positioning.
It significantly improves the accuracy and efficiency of mobile robots in shelf identification and positioning, enhances the stability and reliability of the system, and can accurately identify the characteristic points of the shelf and adapt to complex environment changes.
Smart Images

Figure CN119091408A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of shelf identification, and more specifically, to a shelf identification system and method for a mobile robot. Background Art
[0002] In recent years, mobile robots have developed rapidly in the industrial field, making great contributions to the release of productivity and the improvement of efficiency. Compared with traditional industrial robots, mobile robots have greater flexibility and can operate autonomously in different environments, especially in the manufacturing and logistics industries, showing extremely high application value.
[0003] When mobile robots are performing tasks, such as moving objects in a warehouse, accurate positioning technology is particularly critical. Mobile robots need to accurately identify their own position in a changing environment to ensure accurate execution of tasks. At the same time, it is also crucial to determine the position and posture of the shelf, which helps the robot to accurately calculate and process data during operation.
[0004] Currently, the technical solutions for determining the position and posture of shelves usually rely on reflecting lasers to obtain feature points from the shelf legs. However, due to the large number of interference points that often exist on metal shelf legs, these points may be mistaken for non-shelf features, resulting in positioning errors. This error may prevent the robot from accurately grasping the actual center of the shelf, affecting its ability to perform related operations on the side of the shelf.
[0005] Therefore, a shelf recognition system and method for a mobile robot is desired. Summary of the invention
[0006] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a mobile robot shelf recognition system and method, which first obtains the shelf recognition environment image collected by the camera and the environment shelf point cloud data collected by the laser radar, and then uses deep learning technology to perform feature extraction and association analysis on the two, and finally obtains the classification result through the classifier, and the classification result is used to determine whether the mobile robot is located at the center of the shelf, thereby effectively identifying the real shelf feature points, improving the stability and reliability of the system, and thus significantly improving the accuracy and efficiency of the mobile robot in shelf recognition and positioning.
[0007] According to one aspect of the present application, a shelf recognition system for a mobile robot is provided, comprising:
[0008] The mobile robot shelf data acquisition module is used to obtain shelf recognition environment images collected by the camera and environment shelf point cloud data collected by the laser radar;
[0009] A mobile robot shelf data extraction module is used to extract a mobile robot shelf recognition environment associated feature vector and an optimized environment shelf point cloud coordinate associated feature vector from the shelf recognition environment image collected by the camera and the environment shelf point cloud data collected by the laser radar;
[0010] The shelf center position judgment module is used to judge whether the mobile robot is located at the center of the shelf based on the mobile robot shelf recognition environment associated feature vector and the optimized environment shelf point cloud coordinate associated feature vector.
[0011] According to another aspect of the present application, a shelf recognition method of a mobile robot is provided, comprising:
[0012] Obtain shelf recognition environment images collected by the camera and environment shelf point cloud data collected by the laser radar;
[0013] Extracting a mobile robot shelf recognition environment associated feature vector and an optimized environment shelf point cloud coordinate associated feature vector from the shelf recognition environment image collected by the camera and the environment shelf point cloud data collected by the laser radar;
[0014] Based on the mobile robot shelf recognition environment associated feature vector and the optimized environment shelf point cloud coordinate associated feature vector, it is determined whether the mobile robot is located at the center of the shelf.
[0015] Compared with the prior art, the present application provides a mobile robot shelf recognition system and method, which first obtains the shelf recognition environment image collected by the camera and the environment shelf point cloud data collected by the lidar, and then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally obtains the classification result through the classifier. The classification result is used to determine whether the mobile robot is located at the center of the shelf, and then effectively identify the real shelf feature points, improve the stability and reliability of the system, and thus significantly improve the accuracy and efficiency of the mobile robot in shelf recognition and positioning. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.
[0017] Figure 1 4 is a block diagram of a shelf recognition system of a mobile robot according to an embodiment of the present application.
[0018] Figure 2It is a block diagram of a mobile robot shelf data extraction module in a mobile robot shelf recognition system according to an embodiment of the present application.
[0019] Figure 3 It is a block diagram of a shelf recognition environment feature extraction unit in a shelf recognition system of a mobile robot according to an embodiment of the present application.
[0020] Figure 4 It is a block diagram of a shelf center position determination module in a shelf recognition system of a mobile robot according to an embodiment of the present application.
[0021] Figure 5 The present invention is a flowchart of a method for identifying a shelf of a mobile robot according to an embodiment of the present application.
[0022] Figure 6 is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0023] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.
[0024] Figure 1 Schematic diagram of the shelf recognition system of the mobile robot in the embodiment of the present application. Figure 1 As shown, the shelf recognition system 100 of the mobile robot according to the embodiment of the present application includes: a mobile robot shelf data acquisition module 110, which is used to obtain the shelf recognition environment image collected by the camera and the environment shelf point cloud data collected by the laser radar; a mobile robot shelf data extraction module 120, which is used to extract the mobile robot shelf recognition environment associated feature vector and the optimized environment shelf point cloud coordinate associated feature vector from the shelf recognition environment image collected by the camera and the environment shelf point cloud data collected by the laser radar; a shelf center position judgment module 130, which is used to judge whether the mobile robot is located at the center of the shelf based on the mobile robot shelf recognition environment associated feature vector and the optimized environment shelf point cloud coordinate associated feature vector.
[0025] In the above-mentioned mobile robot shelf recognition system 100, the mobile robot shelf data acquisition module 110 is used to obtain the shelf recognition environment image collected by the camera and the environment shelf point cloud data collected by the laser radar. It should be understood that in recent years, mobile robots have rapidly emerged in the industrial field, greatly promoting the improvement of productivity and efficiency. Compared with traditional fixed robots, mobile robots show greater flexibility and adaptability, and can run and operate autonomously in different working environments. Especially in the manufacturing and logistics fields, they show extremely high application potential and value. When mobile robots perform tasks, such as handling objects in warehouses, accurate positioning technology is crucial. Mobile robots need to be able to accurately identify their own positions in complex and changing environments to ensure that tasks can be completed accurately and efficiently. In addition, accurately determining the position and posture of the shelf is also a key challenge, which helps the robot to effectively calculate and process relevant data during operation.
[0026] At present, the technical methods used to determine the position and posture of shelves usually rely on reflecting lasers to obtain feature points from the legs of the shelves. However, since there are often a large number of interference points on the legs of metal shelves, these points may be mistakenly regarded as non-shelf features, resulting in positioning errors. This error may make it impossible for the robot to accurately grasp the true center position of the shelf, thereby affecting its ability to perform related operations around the shelf. Therefore, in the technical solution of the present application, by acquiring the shelf recognition environment image collected by the camera and the environmental shelf point cloud data collected by the lidar, and combining deep learning technology, it is determined whether the mobile robot is in the center of the shelf, so as to accurately identify the real feature points of the shelf, which is a key step to improve the stability and credibility of the system. This method can significantly enhance the accuracy and operating efficiency of the mobile robot in identifying and locating shelves.
[0027] Specifically, the camera provides high-definition visual information, which can capture the appearance characteristics of the shelf and the surrounding environment, and convert it into machine-recognizable data through image processing and feature extraction. The lidar can obtain the precise three-dimensional structure and distance information of the environment, including details such as the spatial coordinates and shape of the shelf. By combining these two data sources, the system can accurately understand the spatial layout of the surrounding environment, identify the location, shape and state of the shelf, and thus achieve autonomous navigation and target positioning. This multi-sensor data fusion method not only improves the robot's perception ability in complex environments, but also enhances its adaptability to environmental changes, bringing important technical support and innovative solutions to applications in fields such as automated warehouse management.
[0028] In the above-mentioned mobile robot shelf recognition system 100, the mobile robot shelf data extraction module 120 is used to extract the mobile robot shelf recognition environment-related feature vector and the optimized environment shelf point cloud coordinate-related feature vector from the shelf recognition environment image collected by the camera and the environment shelf point cloud data collected by the laser radar. It should be understood that in this way, the robot can accurately locate the position and posture of the shelf, realize intelligent navigation and operation, and thus improve the efficiency and accuracy of warehouse management. This method of comprehensive utilization of vision and laser radar data can not only effectively cope with complex changes and obstacles in the environment, but also bring higher autonomy and adaptability to the automated warehousing system, and promote the development of the logistics industry in the direction of intelligence.
[0029] Figure 2 FIG. 4 is a block diagram of a mobile robot shelf data extraction module in a mobile robot shelf recognition system according to an embodiment of the present application. Figure 2 As shown, in a specific embodiment of the present application, the mobile robot shelf data extraction module 120 includes: a shelf recognition environment feature extraction unit 121, which is used to extract features from the shelf recognition environment image collected by the camera to obtain the mobile robot shelf recognition environment associated feature vector; an environment shelf point cloud feature encoding unit 122, which is used to feature encode the environment shelf point cloud data collected by the laser radar to obtain an environment shelf point cloud coordinate joint feature vector; an environment shelf point cloud semantic encoding unit 123, which is used to semantically encode the environment shelf point cloud data collected by the laser radar to obtain an environment shelf point cloud text understanding feature vector; an environment shelf point cloud coordinate feature association unit 124, which is used to associate the environment shelf point cloud coordinate joint feature vector with the environment shelf point cloud text understanding feature vector to obtain an environment shelf point cloud coordinate associated feature vector; an environment shelf feature optimization unit 125, which is used to perform viewpoint metric perception optimization based on reverse error perception on the environment shelf point cloud coordinate associated feature vector based on the mobile robot shelf recognition environment associated feature vector to obtain an optimized environment shelf point cloud coordinate associated feature vector.
[0030] It should be understood that through preprocessing and image analysis technology, key features in the image, such as color, texture, shape, etc., are extracted, which can help the robot distinguish different types of shelves and identify their locations. Secondly, deep learning methods, such as convolutional neural networks (CNNs), are used to map the processed image data to a high-dimensional feature space to form feature vectors related to the shelf environment. These feature vectors not only contain the visual features of the shelf, but also reflect its position and layout information in space, providing accurate navigation and operation basis for the robot. The process of feature extraction is to convert complex visual information into concise and representative mathematical descriptions, so that the robot can quickly and effectively identify and respond to changes in shelf layout in different scenarios. The application of this technology not only improves the automation level of the warehouse management system, but also optimizes the efficiency and accuracy of logistics operations. By continuously optimizing and updating the feature extraction algorithm, the robot's ability to adapt to environmental changes can be further improved, ensuring the stable operation of the system in a complex warehouse environment.
[0031] Furthermore, feature encoding is performed on the environmental shelf point cloud data collected by the LiDAR, aiming to convert complex three-dimensional spatial information into a form that can be processed and understood by the machine, so as to provide support for the precise positioning and navigation of mobile robots in the warehouse environment. The LiDAR obtains accurate three-dimensional coordinate data of the environment by emitting a laser beam and measuring its return time. These data contain detailed information such as the location, height, and shape of the shelf. First, by filtering and preprocessing the point cloud data, noise and irrelevant information are removed to improve data quality and reliability. Secondly, feature extraction and encoding techniques, such as feature point extraction and descriptor calculation, are used to map the point cloud data into a high-dimensional feature space to form coordinate-related feature vectors related to the shelf environment. These feature vectors not only include the precise location and spatial relationship of the shelf, but also reflect the structural and layout characteristics of the surrounding environment, providing the robot with accurate environmental perception capabilities. The feature encoding process is to convert a large amount of point cloud data into a compact and representative mathematical description, so that the robot can quickly and effectively identify and locate the target shelf, and then perform precise operations and path planning. The application of this technology not only improves the operational efficiency and accuracy of the automated warehouse management system, but also enhances the robot's ability to cope with complex environments. By continuously optimizing and improving the feature encoding algorithm, the stability and adaptability of the system can be further improved to ensure its reliable operation in various warehousing scenarios.
[0032] Furthermore, semantic coding is performed on the environmental shelf point cloud data collected by the LiDAR, aiming to transform the complex three-dimensional spatial information into a higher-level, semantically understood feature vector to support the intelligent perception and decision-making of mobile robots in the warehouse environment. LiDAR obtains point cloud data through high-precision ranging and scanning technology, including the precise location, shape and structural information of the shelf and the surrounding environment. The role of semantic coding is to transform low-level point cloud data into a semantic representation that can be directly applied to robot decision-making and operation. This high-level understanding feature vector can not only provide accurate environmental perception capabilities, but also help robots identify the spatial relationship between different shelves, understand the dynamic changes in the environment and the location of obstacles, thereby supporting their autonomous navigation, path planning and operation execution in complex warehouse environments. Through semantic coding, robots can more effectively understand and respond to different scene changes, and improve the intelligence level and work efficiency of warehouse operations. The application of this technology can not only optimize the management and operation of existing logistics systems, but also lay an important technical foundation and innovation direction for the development of intelligent warehousing and automated logistics in the future.
[0033] In particular, the point cloud coordinates of the environment shelf are associated with the point cloud text understanding feature vector of the environment shelf, aiming to combine the spatial information and semantic features of the point cloud data to provide comprehensive perception and understanding capabilities for mobile robots in the warehouse environment. The point cloud coordinates associated feature vector contains the accurate position and spatial layout of each shelf in the environment, and is a high-dimensional mathematical description generated based on the detailed three-dimensional data collected by the lidar. These vectors can accurately indicate the specific location of the robot shelf and the spatial relationship around it, supporting the robot to accurately navigate and locate. At the same time, the point cloud text understanding feature vector converts the point cloud data into a high-level description with semantic information through semantic encoding. These vectors include the classification and recognition of each object in the environment and the understanding of the relationship between them, such as the type of shelf, the category of stored items, etc. This semantic representation enables the robot to have a deeper understanding of the warehouse environment, accurately identify different shelves and their contents, and provide more intelligent support for logistics operations and management. By associating these two feature vectors, the robot can comprehensively utilize the spatial information and semantic information of the point cloud data to achieve a higher level of environmental perception and decision-making capabilities.
[0034] More specifically, mobile robots need to understand the environment accurately when performing tasks, such as automatically identifying shelves and moving goods in a warehouse. Among them, the associated feature vector refers to describing and identifying objects or structures by analyzing different features in the environment (such as the shape, color, position, etc. of the shelf). Optimizing the associated feature vector of the point cloud coordinates of the shelves in the environment means improving the robot's understanding and recognition of the environment structure through optimization algorithms. Specifically, viewpoint metric perception optimization based on reverse error perception is an optimization method. It adjusts the parameters or model of the shelf recognition algorithm through reverse error perception, that is, by observing the errors or uncertainties generated when the robot performs tasks in the actual environment, so as to make the recognition process more accurate and reliable. The goal of this optimization process is to ensure that the robot can accurately identify and locate the shelves at different angles and positions, so as to perform tasks more effectively. By optimizing the associated feature vector of the point cloud coordinates of the environment shelves, the robot can more accurately build a three-dimensional model of the environment, thereby improving the overall task execution efficiency and accuracy.
[0035] Figure 3 FIG. 1 is a block diagram of a shelf recognition environment feature extraction unit in a shelf recognition system of a mobile robot according to an embodiment of the present application. Figure 3 As shown, in a specific embodiment of the present application, the shelf recognition environment feature extraction unit 121 includes: a shelf recognition environment image preprocessing subunit 1211, which is used to preprocess the shelf recognition environment image collected by the camera to obtain a mobile robot shelf recognition environment input tensor; a shelf recognition environment three-dimensional convolutional encoding subunit 1212, which is used to pass the mobile robot shelf recognition environment three-dimensional convolutional neural network through the mobile robot shelf recognition environment input tensor to obtain the mobile robot shelf recognition environment associated feature vector.
[0036] It should be understood that preprocessing includes image denoising and enhancement to eliminate the effects of noise, blur or illumination changes that may exist in the image on recognition accuracy. Preprocessing also involves image resizing and cropping, unifying images of different resolutions and scales into fixed-size input tensors. This standardization process helps improve the training efficiency and recognition accuracy of machine learning models while reducing the consumption of computing resources. In addition, preprocessing may also include color space conversion, converting images from RGB to grayscale or other color representations suitable for specific tasks to better capture key features. The preprocessing stage can also include image annotation and segmentation, that is, marking and extracting different objects or regions in the image. This annotation information can be used to supervise the training of learning models, enabling robots to identify and locate shelves, goods or other important objects, thereby achieving more accurate logistics operations and management. Through reasonable preprocessing steps, the perception and decision-making capabilities of mobile robots in warehousing environments can be improved, and more efficient and accurate warehousing management operations can be achieved. The application of this technology not only improves the automation level of logistics systems, but also lays a solid foundation for the development of future intelligent warehousing technologies.
[0037] Furthermore, the input tensor of the shelf recognition environment collected by the mobile robot from the camera is processed by the mobile robot shelf recognition environment 3D convolutional neural network (CNN), which aims to extract high-level features from complex image data to support accurate shelf recognition and environmental perception. 3D CNN is a deep learning model designed for processing stereoscopic images or video data, which can effectively capture the temporal nature of spatial information and features. The 3D CNN constructed by convolutional layers, pooling layers, and normalization layers can perform multi-level and multi-scale feature extraction on the input 3D image data. These hierarchical feature tables help to identify different parts of the shelf and the background information of the surrounding environment, thereby improving the accuracy and robustness of recognition. When processing stereoscopic data, the 3D CNN can effectively capture the spatial relationship and complex interactions between objects. This is crucial for mobile robots to locate and identify the location, direction, and stored commodity information of shelves in a warehouse environment. The associated feature vector learned by the deep learning network not only includes the location and shape of the object, but also understands its semantic association in the overall scene, such as which shelves store which commodities or materials. The three-dimensional CNN can also process image data under dynamic changes and complex backgrounds, such as lighting changes, occlusions, and object rotations, thereby improving the robot's ability to identify and respond to different environmental conditions. Through end-to-end learning and optimization, the three-dimensional CNN can automatically learn and extract the most discriminative features, so that the mobile robot can accurately understand and respond to changes and challenges in the warehouse environment. Specifically, each layer of the mobile robot shelf recognition environment three-dimensional convolutional neural network is used to perform convolution processing, mean pooling processing based on the local feature matrix, and nonlinear activation processing on the input data in the forward pass of the layer, so that the last layer of the mobile robot shelf recognition environment three-dimensional convolutional neural network outputs the mobile robot shelf recognition environment associated feature vector, wherein the input of the mobile robot shelf recognition environment three-dimensional convolutional neural network is the mobile robot shelf recognition environment input tensor.
[0038] In a specific embodiment of the present application, the shelf recognition environment image preprocessing subunit 1211 includes: performing image segmentation on the shelf recognition environment image captured by the camera to obtain a plurality of mobile robot shelf recognition environment area images; arranging the plurality of mobile robot shelf recognition environment area images into the mobile robot shelf recognition environment input tensor.
[0039] It should be understood that image segmentation can improve processing efficiency and parallelism. Shelves in large warehouse environments are usually widely distributed and densely packed, and a single camera may not be able to capture all the details of the entire scene. By dividing the image into multiple regions, each region can be assigned to a different mobile robot or processing unit to handle the recognition tasks of multiple regions simultaneously, thereby speeding up the overall object recognition speed and response time. Image segmentation helps to reduce processing complexity and improve the scalability of the algorithm. Processing small blocks of images can reduce the amount of computation and memory required for a single processing compared to the overall large image, making the algorithm easier to implement and optimize. This is especially important for mobile robot applications that have high real-time requirements and are sensitive to processing speed, and can more effectively cope with challenges and changes in dynamic environments. Image segmentation also helps to improve recognition accuracy and robustness. The small size of each small block image helps to reduce background interference and visual confusion, making the object recognition in each region more accurate and reliable. The mobile robot can extract and identify local features for each small region, further improving the performance and performance of the overall system.
[0040] Furthermore, integrating the regional images acquired by multiple mobile robots into one input tensor can facilitate data processing and management. In a warehouse environment, there are usually multiple robots working at the same time, and they may acquire images of different parts of the environment at the same time. Integrating these images into one tensor can reduce the complexity of data transmission and processing, and improve the response speed and overall efficiency of the system.
[0041] In a specific embodiment of the present application, the environmental shelf point cloud feature encoding unit 122 includes: constructing the environmental shelf point cloud data collected by the laser radar into multiple environmental shelf point cloud coordinate feature vectors; passing the multiple environmental shelf point cloud coordinate feature vectors through the environmental shelf point cloud feature encoder to obtain the environmental shelf point cloud coordinate joint feature vector.
[0042] It should be understood that LiDAR can obtain the three-dimensional position and shape information of objects in the environment with high frequency and high precision. Constructing point cloud data as coordinate feature vectors allows robots to accurately measure and locate shelves, items and their relative positions in three-dimensional space, thereby supporting refined operations and path planning. By converting point cloud data into feature vectors, global understanding and local perception of the environment can be achieved. Each feature vector can contain the coordinate information of multiple points. The distribution and density of these points in space reflect the specific shape and arrangement of the shelves and the items on them, providing robots with more detailed environmental cognition capabilities.
[0043] Furthermore, point cloud data consists of a large number of three-dimensional coordinate points, which is challenging to process directly and computationally expensive. Through the feature encoder, the feature vector of each environmental shelf point cloud coordinate can be mapped to a lower-dimensional feature space, retaining important spatial structure and object distribution information while reducing data complexity and redundancy. The feature encoder can learn abstract representations of point cloud data that are more discriminative and interpretable, helping to identify and classify different types of shelves and the items on them. Through the encoded feature vectors, the mobile robot can more effectively understand the spatial layout and relationships of objects in the environment, supporting precise positioning, path planning, and operation execution. Specifically, each layer of the environment shelf point cloud feature encoder is used to perform the following on the input data in the forward pass of the layer: based on the convolution kernel, the input data is convolved to generate a convolution feature map; the convolution feature map is subjected to global mean pooling processing based on the feature matrix to generate a pooling feature map; and the feature values at each position in the pooling feature map are nonlinearly activated to generate an activated feature augmentation map; wherein the output of the last layer of the environment shelf point cloud feature encoder is the optimized environment shelf point cloud coordinate associated feature vector, the input from the second layer to the last layer of the environment shelf point cloud feature encoder is the output of the previous layer, and the input of the environment shelf point cloud feature encoder is the multiple environment shelf point cloud coordinate feature vectors.
[0044] In a specific embodiment of the present application, the environment shelf point cloud semantic encoding unit 123 includes: passing the environment shelf point cloud data collected by the laser radar through the environment shelf point cloud word embedding model to obtain a sequence of multiple environment shelf point cloud word embedding vectors; constructing the sequence of multiple environment shelf point cloud word embedding vectors into an environment shelf point cloud semantic feature matrix; passing the environment shelf point cloud semantic feature matrix through the environment shelf point cloud semantic feature extractor to obtain the environment shelf point cloud text understanding feature vector.
[0045] It should be understood that each point in the point cloud data can be regarded as a "word" in the data set, and the point cloud data sequence corresponds to a "sentence" in the overall data set. Through the environmental shelf point cloud word embedding model, the representation of each point cloud point can be learned, so that each point can be expressed in the form of a vector with semantic meaning. This conversion can greatly simplify the processing and analysis of point cloud data while retaining important spatial information and object relationships. By generating a sequence of point cloud word embedding vectors, a semantic understanding of the shelves and items in the entire environment can be achieved. These vectors can not only capture the geometric features and spatial positions in the point cloud data, but also contain deep information about the type of items, arrangement patterns, and environmental structures. This semantic representation helps the robot to perform more advanced object recognition, scene understanding, and path planning. Specifically, the environmental shelf point cloud data collected by the laser radar is segmented to obtain a word sequence; the embedding layer of the environmental shelf point cloud word embedding model is used to map each word in the word sequence to a word embedding vector to obtain a sequence of multiple environmental shelf point cloud word embedding vectors.
[0046] Furthermore, the semantic feature matrix of the environmental shelf point cloud can arrange and organize the word embedding vector of each point cloud point according to its position and semantic relevance in space. This structured representation allows the robot to intuitively understand and analyze the layout of items, spatial relationships, and possible implicit patterns in the environment, thereby supporting more intelligent and precise warehouse management operations. Constructing the semantic feature matrix of the environmental shelf point cloud helps to improve the interpretability and application flexibility of the data. Each word embedding vector represents a feature point in the point cloud data. Through the arrangement and organization of the matrix, it is easy to compare features, analyze similarities, and detect patterns. This structured representation not only simplifies the data processing process, but also improves the efficiency of data utilization and the depth of analysis.
[0047] Furthermore, the semantic feature matrix of the environmental shelf point cloud is processed by the environmental shelf point cloud semantic feature extractor, aiming to further refine and abstract the high-level semantic information in the point cloud data, and transform the complex environmental scene into a more representative and interpretable text understanding feature vector to support the intelligent decision-making and operation execution of the robot in warehouse management. The environmental shelf point cloud semantic feature extractor can extract the key features of the point cloud data from the semantic feature matrix, such as shelf layout, item type and location distribution. Through this process, each environmental shelf point cloud can be represented as a higher-level and more abstract text understanding feature vector, making the data easier to understand and apply to the robot's intelligent decision-making and behavior planning. By extracting the environmental shelf point cloud text understanding feature vector, the semantic understanding and reasoning of the environmental scene can be achieved. These feature vectors not only contain the geometric information of the point cloud data, but also integrate the semantic relationship and spatial distribution of the items, which helps the robot to achieve more accurate and intelligent object recognition, path planning and task execution, and improve the efficiency and reliability of the warehouse management system. Specifically, each layer of the environment shelf point cloud semantic feature extractor is used to perform the following operations on the input data in the forward pass of the layer: performing convolution processing based on the convolution kernel on the input data to obtain a convolution feature map; performing pooling processing based on the feature matrix on the convolution feature map to obtain a pooling feature map; performing nonlinear activation on the pooling feature map to obtain an activation feature map; wherein the input of the first layer of the environment shelf point cloud semantic feature extractor is the environment shelf point cloud semantic feature matrix, and the output of the last layer of the environment shelf point cloud semantic feature extractor is the environment shelf point cloud text understanding feature vector.
[0048] In the above-mentioned mobile robot shelf recognition system 100, the environment shelf feature optimization unit 125 is used to perform viewpoint metric perception optimization based on reverse error perception on the environment shelf point cloud coordinate associated feature vector based on the mobile robot shelf recognition environment associated feature vector to obtain an optimized environment shelf point cloud coordinate associated feature vector.
[0049] It should be understood that in the technical solution of the present application, it is considered that the environmental shelf point cloud data comes from the point cloud information collected by the laser radar, while the mobile robot shelf recognition environment-related feature vector comes from the image information collected by the camera. These two data sources provide different types of information, which may lead to essential differences between feature vectors. Point cloud data extracts features through a point cloud feature encoder and a word embedding model, while image data extracts features through a three-dimensional convolutional neural network. Different feature extraction methods may lead to differences in the expression of feature vectors. Point cloud data provides precise spatial location information, while image data provides visual information. If there is inconsistency between the two in spatial alignment or scale, the causal association between the feature vectors may be missing. When fusing the two feature vectors, if the fusion strategy does not fully consider the spatial relationship and information complementarity between them, the environmental shelf point cloud coordinate-related feature vector may have a posterior-prior causal association missing relative to the mobile robot shelf recognition environment-related feature vector, resulting in a posterior-prior causal association missing. In order to solve this problem, in the technical solution of the present application, based on the mobile robot shelf recognition environment-related feature vector, the environmental shelf point cloud coordinate-related feature vector is optimized based on the viewpoint metric perception of reverse error perception.
[0050] Among them, based on the mobile robot shelf recognition environment associated feature vector, the environment shelf point cloud coordinate associated feature vector is subjected to viewpoint metric perception optimization based on reverse error perception to obtain the optimized environment shelf point cloud coordinate associated feature vector, including: calculating the position difference between the environment shelf point cloud coordinate associated feature vector and the mobile robot shelf recognition environment associated feature vector to obtain a first viewpoint metric representation vector; calculating the position point multiplication between the environment shelf point cloud coordinate associated feature vector and the mobile robot shelf recognition environment associated feature vector to obtain a second viewpoint metric representation vector; calculating the position point multiplication between the environment shelf point cloud coordinate associated feature vector and the mobile robot shelf recognition environment associated feature vector The first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector are added to obtain a third viewpoint metric representation vector; the gradient amplitude of each position in the first viewpoint metric representation vector, the second viewpoint metric representation vector and the third viewpoint metric representation vector are calculated to obtain a first viewpoint metric perception vector, a second viewpoint metric perception vector and a third viewpoint metric perception vector; the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector are input into a backward machine learning reverse multi-viewpoint perception optimization module to obtain a reverse multi-viewpoint perception optimization representation vector; the position point multiplication between the reverse multi-viewpoint perception optimization representation vector and the environment shelf point cloud coordinate associated feature vector is calculated to obtain the optimized environment shelf point cloud coordinate associated feature vector.
[0051] The backward machine learning reverse multi-view perception optimization module is expressed as follows:
[0052]
[0053] in, and represents the eigenvalues of the i-th position of the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector, respectively, and μ represents the three eigenvalues of the corresponding positions in the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector , σ is the standard deviation of the three eigenvalues at corresponding positions in the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector, represents the maximum value of the three eigenvalues at the corresponding positions in the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector, ReLU represents a linear rectification function, softmax represents a normalized exponential function, and v ri Represents the eigenvalue of the i-th position of the inverse multi-view perceptual optimization representation vector.
[0054] Specifically, in the technical solution of the present application, based on the mobile robot shelf recognition environment associated feature vector, the environment shelf point cloud coordinate associated feature vector is optimized based on reverse error perception. The process includes first calculating the position difference, dot multiplication and sum between the environment shelf point cloud coordinate associated feature vector and the mobile robot shelf recognition environment associated feature vector through multi-view analysis (Multi-View Analysis) to obtain three viewpoint metric representation vectors, and quantify the relationship between features from different angles. Then, the gradient amplitude of each position in these viewpoint metric representation vectors is calculated through reverse error perception (Backward Error Perception) to identify key change points to provide a basis for feature optimization.
[0055] Then, the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector are input into the backward machine learning reverse multi-viewpoint perception optimization module, wherein the backward machine learning reverse multi-viewpoint perception optimization module captures the significant comprehensive ability factor of the feature importance perception (Feature Importance Perception) of the feature sub-dimension distribution of the corresponding position in the first to third viewpoint metric perception vectors by adopting a squeeze-incentive mechanism composed of -functions. Then, the position point multiplication between the reverse multi-viewpoint perception optimization representation vector and the environment shelf point cloud coordinate associated feature vector is calculated to perform backward optimization perception (Backward Optimization Perception) on the environment shelf point cloud coordinate associated feature vector to adjust the environment shelf point cloud coordinate associated feature vector so as to better associate with the mobile robot shelf recognition environment associated feature vector, thereby enhancing the causal relationship between the environment shelf point cloud coordinate associated feature vector and the mobile robot shelf recognition environment associated feature vector.
[0056] In the above-mentioned mobile robot shelf recognition system 100, the shelf center position judgment module 130 is used to judge whether the mobile robot is located at the center of the shelf based on the mobile robot shelf recognition environment associated feature vector and the optimized environment shelf point cloud coordinate associated feature vector. It should be understood that the mobile robot can obtain important information about the shape, size and surrounding environment characteristics of the shelf through the shelf recognition environment associated feature vector. These feature vectors may include the geometric features of the shelf, the surface structure or the relationship with the surrounding objects, providing the robot with a basis for quickly identifying and locating the shelf in a complex environment. The optimized environment shelf point cloud coordinate associated feature vector provides spatial data and layout information of the specific location of the shelf. These vectors describe the precise coordinates of the shelf, the arrangement of objects and the spatial relationship of the surrounding environment, providing the robot with an accurate position reference and operation benchmark. By fusing these two feature vectors, the robot can comprehensively consider the spatial layout characteristics and specific position information of the shelf, so as to judge whether it is located at the center of the shelf. This judgment process not only includes simple coordinate comparison, but also considers the influencing factors of the shape of the shelf and the surrounding environment to ensure the accuracy and stability of the positioning result.
[0057] Figure 4The block diagram of the shelf center position judgment module in the shelf recognition system of the mobile robot according to the embodiment of the present application. In a specific embodiment of the present application, the shelf center position judgment module 130 includes: a mobile robot shelf data feature fusion unit 131, which is used to fuse the mobile robot shelf recognition environment associated feature vector and the optimized environment shelf point cloud coordinate associated feature vector to obtain a shelf center position judgment feature vector; a shelf center position judgment classification unit 132, which is used to pass the shelf center position judgment feature vector through a classifier to obtain a classification result, and the classification result is used to judge whether the mobile robot is located at the center of the shelf.
[0058] It should be understood that by fusing these two feature vectors, the robot can comprehensively consider the geometric structure characteristics and specific spatial position information of the shelf, thereby inferring the center position feature vector of each shelf. This vector not only includes the coordinates of the center point of the shelf, but may also contain environmental features related to the location, such as the relative position relationship to other important locations or the placement of items. In addition, the fused shelf center position judgment feature vector can also improve the robustness and accuracy of the system. By combining different types of feature vectors, the diversity and complexity of data can be effectively handled, and the robot's adaptability and operating accuracy in different storage environments can be improved.
[0059] Furthermore, by using a classifier to process the feature vector for determining the center position of the shelf, complex feature data can be converted into simple and clear classification results. The classifier can classify the input feature vector into several predefined categories, such as "located in the center" or "not located in the center". This classification result provides the direct feedback needed by the robot when performing tasks, helping it to quickly determine the accuracy of the current position and whether it needs to be adjusted. The results obtained by the classifier can also be used to optimize the automated control and monitoring of the system. For example, the system can automatically adjust the robot's movement trajectory or trigger specific operation processes based on the classification results to achieve more intelligent and efficient warehouse management. This automated feedback closed loop not only reduces the need for manual intervention, but also improves the system's response speed and overall operational efficiency.
[0060] In summary, the embodiment of the present application first obtains the shelf recognition environment image collected by the camera and the environment shelf point cloud data collected by the lidar, and then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally obtains the classification result through the classifier. The classification result is used to determine whether the mobile robot is located at the center of the shelf, and then effectively identify the real shelf feature points, improve the stability and reliability of the system, and thus significantly improve the accuracy and efficiency of the mobile robot in shelf recognition and positioning.
[0061] As described above, the shelf identification system 100 of the mobile robot according to the embodiment of the present application can be implemented in various terminal devices. In one example, the shelf identification system 100 of the mobile robot can be integrated into the terminal device as a software module and / or a hardware module. For example, the shelf identification system 100 of the mobile robot can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the shelf identification system 100 of the mobile robot can also be one of the many hardware modules of the terminal device.
[0062] Alternatively, in another example, the shelf identification system 100 of the mobile robot and the terminal device may also be separate devices, and the shelf identification system 100 of the mobile robot may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.
[0063] Figure 5 FIG. 1 is a flow chart of a shelf identification method of a mobile robot according to an embodiment of the present application. Figure 5 As shown, according to the shelf recognition method of a mobile robot in an embodiment of the present application, it includes: S110, acquiring a shelf recognition environment image collected by a camera and environmental shelf point cloud data collected by a laser radar; S120, extracting a mobile robot shelf recognition environment associated feature vector and an optimized environmental shelf point cloud coordinate associated feature vector from the shelf recognition environment image collected by the camera and the environmental shelf point cloud data collected by the laser radar; S130, judging whether the mobile robot is located at the center of the shelf based on the mobile robot shelf recognition environment associated feature vector and the optimized environmental shelf point cloud coordinate associated feature vector.
[0064] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned mobile robot shelf recognition method have been described in the above reference. Figures 1 to 4 The description of the shelf recognition system of the mobile robot has been described in detail, and therefore, its repeated description will be omitted.
[0065] Below, reference Figure 6 To describe an electronic device according to an embodiment of the present application.
[0066] Figure 6 A block diagram of an electronic device according to an embodiment of the present application is illustrated.
[0067] like Figure 6As shown, the electronic device 10 includes an input device 11, an input interface 12, a central processing unit 13, a memory 14, an output interface 15, an output device 16, and a bus 17. The input interface 12, the central processing unit 13, the memory 14, and the output interface 15 are connected to each other through the bus 17, and the input device 11 and the output device 16 are connected to the bus 17 through the input interface 14 and the output interface 15, respectively, and then connected to other components of the electronic device 10.
[0068] Specifically, the input device 11 receives input information from the outside and transmits the input information to the central processing unit 13 through the input interface 12; the central processing unit 13 processes the input information based on the computer executable instructions stored in the memory 14 to generate output information, temporarily or permanently stores the output information in the memory 14, and then transmits the output information to the output device 16 through the output interface 15; the output device 16 outputs the output information to the outside of the electronic device 10 for user use.
[0069] In one embodiment, Figure 6 The electronic device 10 shown can be implemented as a network device, which may include: a memory configured to store programs; a processor configured to run the programs stored in the memory to execute any one of the mobile robot shelf recognition methods described in the above embodiments.
[0070] According to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed from a removable storage medium.
[0071] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0072] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present application, but the present application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of the present application, and these modifications and improvements are also considered to be within the scope of protection of the present application.
Claims
1. A shelf recognition system for a mobile robot, characterized in that: include: The mobile robot shelf data acquisition module is used to obtain shelf recognition environment images collected by the camera and environment shelf point cloud data collected by the laser radar; A mobile robot shelf data extraction module is used to extract a mobile robot shelf recognition environment associated feature vector and an optimized environment shelf point cloud coordinate associated feature vector from the shelf recognition environment image collected by the camera and the environment shelf point cloud data collected by the laser radar; The shelf center position judgment module is used to judge whether the mobile robot is located at the center of the shelf based on the mobile robot shelf recognition environment associated feature vector and the optimized environment shelf point cloud coordinate associated feature vector.
2. The shelf recognition system of the mobile robot according to claim 1, characterized in that: The mobile robot shelf data extraction module comprises: A shelf recognition environment feature extraction unit, used for performing feature extraction on the shelf recognition environment image collected by the camera to obtain a feature vector associated with the mobile robot shelf recognition environment; An environmental shelf point cloud feature encoding unit, used for performing feature encoding on the environmental shelf point cloud data collected by the laser radar to obtain a joint feature vector of the environmental shelf point cloud coordinates; An environment shelf point cloud semantic encoding unit, used for semantically encoding the environment shelf point cloud data collected by the laser radar to obtain an environment shelf point cloud text understanding feature vector; An environment shelf point cloud coordinate feature association unit, used to associate the environment shelf point cloud coordinate joint feature vector with the environment shelf point cloud text understanding feature vector to obtain an environment shelf point cloud coordinate association feature vector; The environmental shelf feature optimization unit is used to perform viewpoint metric perception optimization based on reverse error perception on the environmental shelf point cloud coordinate associated feature vector based on the mobile robot shelf recognition environment to obtain the optimized environmental shelf point cloud coordinate associated feature vector.
3. The shelf recognition system of the mobile robot according to claim 2, characterized in that: The shelf recognition environment feature extraction unit comprises: A shelf recognition environment image preprocessing subunit, used for preprocessing the shelf recognition environment image collected by the camera to obtain a mobile robot shelf recognition environment input tensor; The shelf recognition environment three-dimensional convolutional encoding subunit is used to pass the mobile robot shelf recognition environment input tensor through the mobile robot shelf recognition environment three-dimensional convolutional neural network to obtain the mobile robot shelf recognition environment associated feature vector.
4. The shelf recognition system of the mobile robot according to claim 3, characterized in that: The shelf recognition environment image preprocessing subunit includes: The shelf recognition environment image captured by the camera is divided into image blocks to obtain a plurality of mobile robot shelf recognition environment area images; The plurality of mobile robot shelf recognition environment area images are arranged as the mobile robot shelf recognition environment input tensor.
5. The shelf recognition system of the mobile robot according to claim 4, characterized in that: The environmental shelf point cloud feature encoding unit includes: The environmental shelf point cloud data collected by the laser radar is constructed into a plurality of environmental shelf point cloud coordinate feature vectors; The plurality of environment shelf point cloud coordinate feature vectors are passed through an environment shelf point cloud feature encoder to obtain the environment shelf point cloud coordinate joint feature vector.
6. The shelf recognition system of the mobile robot according to claim 5, characterized in that: The environment shelf point cloud semantic encoding unit includes: The environmental shelf point cloud data collected by the laser radar is passed through the environmental shelf point cloud word embedding model to obtain a sequence of multiple environmental shelf point cloud word embedding vectors; Constructing a sequence of the plurality of environmental shelf point cloud word embedding vectors into an environmental shelf point cloud semantic feature matrix; The environment shelf point cloud semantic feature matrix is passed through the environment shelf point cloud semantic feature extractor to obtain the environment shelf point cloud text understanding feature vector.
7. The shelf recognition system of the mobile robot according to claim 6, characterized in that: The environmental shelf feature optimization unit comprises: Calculating the position difference between the environment shelf point cloud coordinate associated feature vector and the mobile robot shelf recognition environment associated feature vector to obtain a first viewpoint metric representation vector; Calculate the position point multiplication between the environment shelf point cloud coordinate associated feature vector and the mobile robot shelf recognition environment associated feature vector to obtain a second viewpoint metric representation vector; Calculate the positional sum of the environment shelf point cloud coordinate associated feature vector and the mobile robot shelf recognition environment associated feature vector to obtain a third viewpoint metric representation vector; Calculating the gradient magnitude of each position in the first viewpoint metric representation vector, the second viewpoint metric representation vector, and the third viewpoint metric representation vector to obtain a first viewpoint metric perception vector, a second viewpoint metric perception vector, and a third viewpoint metric perception vector; Inputting the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector into a backward machine learning inverse multi-viewpoint perception optimization module to obtain a reverse multi-viewpoint perception optimization representation vector; The position point multiplication between the reverse multi-view perception optimization representation vector and the environment shelf point cloud coordinate associated feature vector is calculated to obtain the optimized environment shelf point cloud coordinate associated feature vector.
8. The shelf recognition system of the mobile robot according to claim 7, characterized in that: The shelf center position determination module includes: A mobile robot shelf data feature fusion unit, used to fuse the mobile robot shelf recognition environment associated feature vector and the optimized environment shelf point cloud coordinate associated feature vector to obtain a shelf center position judgment feature vector; The shelf center position judgment classification unit is used to pass the shelf center position judgment feature vector through a classifier to obtain a classification result, and the classification result is used to judge whether the mobile robot is located at the center of the shelf.
9. A shelf recognition method for a mobile robot, characterized in that: include: Obtain shelf recognition environment images collected by the camera and environment shelf point cloud data collected by the laser radar; Extracting a mobile robot shelf recognition environment associated feature vector and an optimized environment shelf point cloud coordinate associated feature vector from the shelf recognition environment image collected by the camera and the environment shelf point cloud data collected by the laser radar; Based on the mobile robot shelf recognition environment associated feature vector and the optimized environment shelf point cloud coordinate associated feature vector, it is determined whether the mobile robot is located at the center of the shelf.
10. The shelf identification method of a mobile robot according to claim 9, characterized in that: Extracting the mobile robot shelf recognition environment associated feature vector and optimizing the environment shelf point cloud coordinate associated feature vector from the shelf recognition environment image collected by the camera and the environment shelf point cloud data collected by the laser radar, including: Performing feature extraction on the shelf recognition environment image captured by the camera to obtain a feature vector associated with the mobile robot shelf recognition environment; Performing feature encoding on the environmental shelf point cloud data collected by the laser radar to obtain a joint feature vector of the environmental shelf point cloud coordinates; Performing semantic encoding on the environmental shelf point cloud data collected by the laser radar to obtain an environmental shelf point cloud text understanding feature vector; Associating the environment shelf point cloud coordinate joint feature vector with the environment shelf point cloud text understanding feature vector to obtain an environment shelf point cloud coordinate associated feature vector; Based on the mobile robot shelf recognition environment associated feature vector, the environment shelf point cloud coordinate associated feature vector is optimized based on the viewpoint metric perception based on reverse error perception to obtain the optimized environment shelf point cloud coordinate associated feature vector.
