Goods allocation identification system and method of mobile robot

Abstract

The invention relates to the field of goods allocation identification, and particularly discloses a goods allocation identification system and method for a mobile robot, and the method comprises the steps: firstly obtaining goods electronic tag information, warehouse multi-region goods shelf images collected by a camera, and warehouse environment point cloud data scanned by a laser radar, and then carrying out the deep learning technology, and feature extraction and correlation analysis are performed on the three parts, and finally a classification result is obtained through a classifier so as to obtain a cargo allocation classification label, so that automatic inventory loading management is realized, human errors are reduced, and the use of a warehouse space is optimized.

Description

Technical Field

[0001] The present application relates to the field of cargo location identification, and more specifically, to a cargo location identification system and method for a mobile robot. Background Art

[0002] Logistics refers to the process of ensuring the flow of goods, services and information from suppliers to consumers through effective planning, implementation and control in supply chain management. This includes transportation, warehousing, inventory management, distribution and information processing, aiming to improve efficiency, reduce costs and meet customer needs.

[0003] The traditional manual loading method adopts the "people to goods" mode, that is, the loaders load the goods on the shelves according to the goods information. This method is not only complicated in operation process and inefficient, but also has high labor costs and high skill requirements for loaders. This method no longer meets the needs of modern warehousing. With the advancement of Internet of Things technology, robotics technology and computer technology, the introduction of mobile robot control systems into the loading link of automated warehouses has become a development trend in the field of logistics loading.

[0004] Therefore, a cargo location identification system and method for a mobile robot is desired. Summary of the invention

[0005] In order to solve the above technical problems, this application is proposed. The embodiment of this application provides a mobile robot cargo location recognition system and method, which first obtains cargo electronic tag information, warehouse multi-area shelf images collected by cameras, and warehouse environment point cloud data scanned by laser radar, and then uses deep learning technology to perform feature extraction and association analysis on the three, and finally obtains classification results through a classifier to obtain cargo location classification labels, thereby realizing automated inventory management, reducing human errors, and optimizing the use of warehouse space.

[0006] According to one aspect of the present application, a cargo location identification system for a mobile robot is provided, comprising:

[0007] The mobile robot cargo location data acquisition module is used to obtain cargo electronic tag information, warehouse multi-area shelf images collected by cameras, and warehouse environment point cloud data scanned by laser radar;

[0008] A mobile robot cargo storage location data extraction module is used to extract the cargo electronic tag information text semantic feature vector and the warehouse environment multimodal association feature vector from the cargo electronic tag information, the warehouse multi-area shelf image collected by the camera and the warehouse environment point cloud data scanned by the laser radar;

[0009] The cargo location label judgment module is used to obtain the cargo location classification label based on the semantic feature vector of the cargo electronic label information text and the multimodal association feature vector of the warehouse environment.

[0010] According to another aspect of the present application, a method for identifying cargo locations of a mobile robot is provided, comprising:

[0011] Obtain cargo electronic tag information, warehouse multi-area shelf images captured by cameras, and warehouse environment point cloud data scanned by LiDAR;

[0012] Extracting a text semantic feature vector of the cargo electronic tag information and a multimodal correlation feature vector of the warehouse environment from the cargo electronic tag information, the warehouse multi-area shelf image collected by the camera, and the warehouse environment point cloud data scanned by the laser radar;

[0013] Based on the semantic feature vector of the cargo electronic tag information text and the multimodal association feature vector of the warehouse environment, a cargo location classification label is obtained.

[0014] Compared with the prior art, the present application provides a mobile robot cargo location identification system and method, which first obtains the cargo electronic tag information, the warehouse multi-area shelf images collected by the camera, and the warehouse environment point cloud data scanned by the lidar, and then uses deep learning technology to perform feature extraction and association analysis on the three, and finally obtains the classification result through the classifier to obtain the cargo location classification label, thereby realizing automated inventory management, reducing human errors, and optimizing the use of warehouse space. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 4 is a block diagram of a cargo location identification system for a mobile robot according to an embodiment of the present application.

[0017] Figure 2 4 is a block diagram of a mobile robot cargo location data extraction module in a mobile robot location identification system according to an embodiment of the present application.

[0018] Figure 3 It is a block diagram of a warehouse environment multimodal feature extraction unit in a cargo location recognition system of a mobile robot according to an embodiment of the present application.

[0019] Figure 4 It is a block diagram of a cargo location label determination module in a cargo location identification system of a mobile robot according to an embodiment of the present application.

[0020] Figure 5 The present invention is a flowchart of a method for identifying cargo locations of a mobile robot according to an embodiment of the present application.

[0021] Figure 6 is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0023] Figure 1 FIG. 1 is a block diagram of a cargo location identification system of a mobile robot according to an embodiment of the present application. Figure 1 As shown, according to the embodiment of the present application, the cargo location identification system 100 of the mobile robot includes: a mobile robot cargo location data acquisition module 110, which is used to obtain cargo electronic tag information, warehouse multi-area shelf images collected by a camera, and warehouse environment point cloud data scanned by a laser radar; a mobile robot cargo location data extraction module 120, which is used to extract cargo electronic tag information text semantic feature vectors and warehouse environment multimodal association feature vectors from the cargo electronic tag information, the warehouse multi-area shelf images collected by the camera, and the warehouse environment point cloud data scanned by the laser radar; a cargo location label judgment module 130, which is used to obtain a cargo location classification label based on the cargo electronic tag information text semantic feature vector and the warehouse environment multimodal association feature vector.

[0024] In the above-mentioned mobile robot cargo location identification system 100, the mobile robot cargo location data acquisition module 110 is used to obtain cargo electronic tag information, warehouse multi-area shelf images collected by cameras, and warehouse environment point cloud data scanned by laser radar. It should be understood that logistics in supply chain management refers to the entire process from supplier to consumer, through precise planning, implementation and control to ensure the smooth flow of goods, services and information. It covers transportation, warehousing, inventory management, distribution and information processing, with the goal of improving efficiency, reducing costs and meeting customer needs. The traditional method of loading goods usually adopts the "people to goods" model, that is, the loaders place the goods on the shelves one by one according to the goods information. This method is not only cumbersome and inefficient, but also involves high labor costs and requires high skill levels of the loaders. With the changes in modern warehousing needs, this manual method has become no longer applicable. Therefore, in the technical solution of the present application, by acquiring the electronic tag information of the goods, the multi-area shelf images of the warehouse collected by the camera, and the point cloud data of the warehouse environment scanned by the lidar, and combining with deep learning technology, the cargo location classification labels are obtained to realize the automated management of inventory loading, which not only reduces human errors, but also optimizes the utilization of warehouse space.

[0025] Specifically, the acquisition of goods electronic tag information, warehouse multi-area shelf images captured by cameras, and warehouse environment point cloud data scanned by LiDAR is to comprehensively and accurately identify the location and status of goods in the warehouse. Each data source provides different types of information, which helps to improve the recognition accuracy and reliability of the system. First, the goods electronic tag information provides direct goods identification data. Electronic tags usually contain detailed information about the goods, such as name, quantity, specifications, etc., which can be quickly obtained by reading the tags. The main advantages of this data are its high accuracy and immediacy, but it is limited to the area covered by the tag and cannot provide the spatial location or environmental context of the goods. Secondly, the warehouse multi-area shelf images captured by cameras can capture the visual information of the shelves, including the appearance, placement and relative position of the goods. Image data helps the system understand the layout of the goods on the actual shelves and extract and classify the goods through image recognition technology. Although image information provides rich visual data, image quality and viewing angle may affect the accuracy of the data in complex environments. Finally, the warehouse environment point cloud data scanned by LiDAR provides three-dimensional structural information of the warehouse space. LiDAR can accurately measure the distance between objects and sensors and generate detailed point cloud data, which is crucial for understanding the spatial distribution and environmental layout of goods. Point cloud data can create a three-dimensional model to help the system identify and locate goods and obstacles around them, especially in low-light or visually complex environments. By integrating these data sources, the system can obtain comprehensive cargo information and environmental context, and conduct multi-angle and multi-level data analysis. This comprehensive recognition method not only improves the accuracy of cargo positioning, but also handles complex and dynamic warehouse environments, ultimately achieving efficient warehouse management and operations.

[0026] In the above-mentioned mobile robot cargo location identification system 100, the mobile robot cargo location data extraction module 120 is used to extract the cargo electronic tag information text semantic feature vector and the warehouse environment multimodal association feature vector from the cargo electronic tag information, the warehouse multi-area shelf image collected by the camera, and the warehouse environment point cloud data scanned by the laser radar. In this way, the system can more accurately understand the actual location of the cargo and identify the relationship between the cargo and the environment, thereby improving the accuracy and efficiency of cargo location identification. Such multimodal data fusion not only makes up for the limitations of a single data source, but also can handle complex environmental conditions to achieve more accurate warehouse management and cargo positioning.

[0027] Figure 2 FIG. 1 is a block diagram of a mobile robot cargo location data extraction module in a mobile robot cargo location identification system according to an embodiment of the present application. Figure 2As shown, in a specific embodiment of the present application, the mobile robot cargo storage location data extraction module 120 includes: a cargo electronic tag feature extraction unit 121, used to perform feature extraction on the cargo electronic tag information to obtain a text semantic feature vector of the cargo electronic tag information; a warehouse environment multimodal feature extraction unit 122, used to perform feature extraction on the warehouse multi-area shelf images collected by the camera and the warehouse environment point cloud data scanned by the lidar to obtain the warehouse environment multimodal correlation feature vector.

[0028] It should be understood that the feature extraction process involves converting the original electronic tag text information into semantic feature vectors, which is achieved through natural language processing (NLP) technology. First, the text content in the tag information needs to be segmented and vocabulary mapped to decompose it into basic units that the system can understand. Then, through a context encoding model, such as a Transformer-based model, these segmented texts are converted into high-dimensional semantic feature vectors. These feature vectors can capture the semantic information in the text and represent the meaning and relevance in the text. Extracting text semantic feature vectors helps improve the accuracy of the cargo identification system. Among them, the original electronic tag information is only static text data, and the semantic feature vector can map this text data into a space with mathematical and computational meaning, so that the system can understand and process this information. This semantic conversion can also help the system deal with synonyms, typos, and other problems, and improve the robustness and accuracy of the system.

[0029] Furthermore, feature extraction is performed on the warehouse multi-area shelf images collected by the camera and the warehouse environment point cloud data scanned by the LiDAR. The main purpose is to make full use of the data from different sensors and fully understand the spatial layout and environmental characteristics of the goods in the warehouse environment. The core of this process is to convert image data and point cloud data into feature representations that can be effectively fused and analyzed, thereby improving the intelligence level of the warehouse management system. The image data collected by the camera provides two-dimensional visual information of the shelves and goods. By extracting features from these images, the system can identify the structure of the shelves, the appearance of the goods, and the placement of the goods. These visual features are crucial for understanding the type and state of the goods and the specific arrangement of the shelves. However, relying solely on image data may not accurately obtain the spatial depth information of the goods and the three-dimensional structure of the environment. The point cloud data scanned by the LiDAR provides three-dimensional spatial information of the warehouse environment. Point cloud data can capture the precise spatial coordinates of the shelves and goods, and describe the real geometry of the environment, including the height and width of the shelves, and the actual position and relative relationship of the goods. These data are very important for understanding the specific location of the goods in three-dimensional space and their placement in a complex environment. The features of these two data types are extracted and fused to form a multimodal correlation feature vector of the warehouse environment, which can comprehensively consider the visual information and spatial structure information of the shelves. This multimodal feature vector enables the system to more comprehensively understand the layout and environmental configuration of the goods, thereby achieving more accurate goods positioning, classification and inventory management.

[0030] In a specific embodiment of the present application, the cargo electronic label feature extraction unit 121 includes: segmenting the cargo electronic label information to obtain a cargo electronic label information word sequence; passing the cargo electronic label information word sequence through a cargo electronic label information context text encoder based on a Taming model to obtain a cargo electronic label information text semantic feature vector.

[0031] It should be understood that the electronic label of goods usually contains multiple fields, such as product name, model, specification, production date, etc. These fields exist in the form of natural language text, but when computers process this kind of unstructured data, they need to convert it into structured information. In the word segmentation stage, splitting the continuous text in the label into meaningful words or phrases is the first step to achieve this goal. In system design, the information after word segmentation is easier to integrate and compare with other data sources. Through standardization and structured processing, label information can be more consistently integrated into databases and information systems, improving data consistency and compatibility.

[0032] Furthermore, the word sequence of the cargo electronic label information is passed through the cargo electronic label information context text encoder based on the Taming model in order to achieve a deep understanding and effective representation of the text information. Among them, the Taming model (such as Taming Transformer) is an advanced context text encoder that uses deep learning technology for text modeling. Its main advantage is that it can capture the contextual relationship between words, thereby generating a richer and more accurate text feature representation. It should be understood that traditional word embedding methods such as Word2Vec can usually only capture the local semantics of words, while the Taming model can use a deep neural network architecture to capture the contextual information of words in the entire sentence. This context modeling capability enables the representation of each word to not only reflect its individual meaning, but also consider its context in a specific context. Through the context encoder, each word in the word sequence will be mapped to a high-dimensional semantic space. In this space, the vector of the word not only represents its basic meaning, but also integrates its relationship with other words. The Taming model encodes the entire text sequence into a comprehensive text feature vector, taking into account the order and semantic relationship between words. This feature vector provides a global understanding of the label information, enabling the system to more accurately classify and identify various types of information about the goods. Specifically, the embedding layer of the cargo electronic label information context text encoder based on the Taming model is used to convert the cargo electronic label information word sequence into an embedding vector to obtain a sequence of cargo electronic label embedding vectors; the converter-based Bert model of the cargo electronic label information context text encoder based on the Taming model is used to perform global context semantic encoding on the sequence of cargo electronic label embedding vectors to obtain multiple cargo electronic label feature vectors; and, the multiple cargo electronic label feature vectors are cascaded to obtain the cargo electronic label information text semantic feature vector.

[0033] Figure 3 FIG. 4 is a block diagram of a warehouse environment multimodal feature extraction unit in a cargo location recognition system of a mobile robot according to an embodiment of the present application. Figure 3 As shown, in a specific embodiment of the present application, the warehouse environment multimodal feature extraction unit 122 includes: a multi-area shelf feature encoding subunit 1221, which is used to feature encode the warehouse multi-area shelf image collected by the camera to obtain a warehouse panoramic shelf environment feature vector; an environment point cloud feature encoding subunit 1222, which is used to feature encode the warehouse environment point cloud data scanned by the lidar to obtain a warehouse environment point cloud semantic association feature vector; an environment multimodal feature association subunit 1223, which is used to associate the warehouse panoramic shelf environment feature vector with the warehouse environment point cloud semantic association feature vector to obtain the warehouse environment multimodal association feature vector.

[0034] It should be understood that the key to image feature encoding is to extract and integrate important information from the image. Warehouse images usually contain multiple regions of shelves and the goods on them, and the images of each region show different perspectives and details. Feature encoding first extracts important visual information from the image, such as the layout of the shelves, the location and category of the goods, etc. This information is extracted through deep learning models such as convolutional neural networks (CNNs), which can recognize patterns and structures in images. For example, the model can recognize the outline of the shelf, the label and color of the goods, and thus understand the content of each region. Integrate the image features of different regions into a panoramic shelf environment feature vector. This step involves fusing the image features of multiple regions into a comprehensive feature representation that reflects the structure and content of the entire warehouse environment. This comprehensive feature vector provides a comprehensive view of the entire warehouse shelf layout, which helps the system understand the overall configuration of the shelf and the distribution of items.

[0035] Furthermore, feature encoding of warehouse environment point cloud data scanned by LiDAR is to extract useful information from three-dimensional spatial data and conduct in-depth analysis. The main purpose of this process is to convert the complex geometric and spatial information in point cloud data into easy-to-process feature representations, so as to achieve a comprehensive understanding and intelligent management of the warehouse environment. Point cloud data is collected by LiDAR equipment and records the three-dimensional coordinate (x, y, z) information of each point in the warehouse environment. Point cloud data can provide accurate three-dimensional information about shelf structure, cargo location and other spatial features, but such data is usually primitive, large and difficult to use directly. Geometric features such as point density, surface normal vector, and distance between points are extracted from point cloud data. These geometric features can reflect key information such as the shape, size and spatial distribution of the shelf. For example, by calculating the surface normal vector of point cloud data, the inclination angle and shape of the shelf can be identified. The relationship between points in space also needs to be considered in the process of encoding point cloud data. This includes the spatial relationship between shelves and shelves, and between goods and shelves.

[0036] Furthermore, the warehouse panoramic shelf environment feature vector and the warehouse environment point cloud semantic association feature vector are associated in order to fuse information from different data sources and provide a more comprehensive and accurate understanding of the warehouse environment. This multimodal feature fusion can take advantage of the complementary advantages of different data types, thereby improving the intelligence level of the warehouse management system. Among them, the panoramic shelf environment feature vector is derived from image data, which mainly contains visual information such as shelf layout, location and category of goods. These feature vectors can reflect the two-dimensional plane configuration of the shelf and the appearance characteristics of the items, but they lack deep spatial relationship information. The point cloud semantic association feature vector is derived from LiDAR scanning and provides accurate geometric information in three-dimensional space, such as the shape, size and distribution of the shelf in space. Point cloud data can reveal the three-dimensional structure and spatial layout of the environment, but it is usually limited in semantic understanding. Associating the panoramic image feature vector and the point cloud feature vector can combine visual information with spatial geometric information to obtain a more complete description of the warehouse environment, combining visual features and spatial relationships to improve the understanding of the environment.

[0037] In a specific embodiment of the present application, the multi-area shelf feature encoding subunit 1221 includes: arranging the warehouse multi-area shelf images collected by the camera into a warehouse area shelf input tensor; passing the warehouse area shelf input tensor through a convolutional neural network-based warehouse area shelf feature encoder to obtain the warehouse panoramic shelf environment feature vector.

[0038] It should be understood that the original warehouse multi-area shelf images usually exist in the form of image files, each image showing a different area in the warehouse. Arranging these images into tensors can structure multiple image data into a unified multidimensional array. This structured data format facilitates batch processing and training of deep learning models. Specifically, each image can be represented as a two-dimensional matrix (height × width × color channel), and multiple images are arranged together to form a three-dimensional tensor, which represents the height, width, color channel of the image, and the batch of images. When processing image data, the size, resolution, and color channel of different images may vary. By arranging images into tensors, the images can be uniformly preprocessed and standardized. For example, scaling all images to the same size and converting them to a standard color channel format can ensure that the data input to the model is consistent and avoid processing problems caused by inconsistent data.

[0039] Further, the warehouse area shelf input tensor is processed by a warehouse area shelf feature encoder based on a convolutional neural network (CNN), mainly to efficiently extract and encode key information in the warehouse environment, thereby realizing intelligent warehouse management and analysis. It should be understood that the original warehouse area shelf image data is usually high-dimensional, and directly processing this data may be very complex and inefficient. Through the feature encoder, CNN converts the high-dimensional input tensor into a lower-dimensional feature vector. This feature vector retains important structural and semantic information, but is represented in a more compact and processing-friendly form. The warehouse panoramic shelf environment feature vector is obtained by integrating information from all regions, a process that enables the model to understand the layout and item distribution of the entire warehouse. The feature encoder extracts local features through convolutional layers and integrates these local features through mechanisms such as pooling layers to ultimately form a comprehensive environment feature vector. Specifically, each layer of the warehouse area shelf feature encoder based on a convolutional neural network is used to perform convolution processing, mean pooling processing based on a local feature matrix, and nonlinear activation processing on the input data in the forward pass of the layer, so that the last layer of the warehouse area shelf feature encoder based on a convolutional neural network outputs the warehouse panoramic shelf environment feature vector, wherein the input of the warehouse area shelf feature encoder based on a convolutional neural network is the warehouse area shelf input tensor.

[0040] In a specific embodiment of the present application, the environment point cloud feature encoding subunit 1222 includes: passing the warehouse environment point cloud data scanned by the lidar through a warehouse environment point cloud context encoder including an embedding layer to obtain a plurality of warehouse environment point cloud semantic feature vectors; and arranging the plurality of warehouse environment point cloud semantic feature vectors in two dimensions and passing them through a warehouse environment point cloud feature encoder to obtain the warehouse environment point cloud semantic association feature vector.

[0041] It should be understood that the point cloud data generated by the LiDAR contains the 3D coordinate information of each point in the warehouse environment. This data is usually high-dimensional and sparse, and direct processing can be very complex. The spatial distribution and density variation of point cloud data make it challenging to extract useful semantic information. Therefore, it is necessary to convert these raw point cloud data into more meaningful feature representations. The context encoder with an embedding layer can map each point of the point cloud data into a low-dimensional feature space. By mapping the high-dimensional point cloud data into a dense feature vector space, the embedding layer can capture the local and global information of each point while reducing the redundancy of the data. The point cloud context encoder integrates the spatial distribution and neighborhood information of the point cloud into the feature representation by analyzing the spatial relationship and context information in the point cloud data. In this way, the encoder can extract the structural features and semantic information of the point cloud data, such as the location of shelves, the arrangement of items, and other environmental features. This context encoding capability helps to understand the complex structure in the point cloud data and improve the overall understanding of the environment. This multi-feature representation allows the system to understand and operate on each part of the point cloud data in detail. Specifically, the warehouse environment point cloud data scanned by the lidar is segmented to obtain an environment point cloud word sequence; the embedding layer of the warehouse environment point cloud context encoder including the embedding layer is used to map each environment point cloud word in the environment point cloud word sequence into an environment point cloud word embedding vector to obtain a sequence of environment point cloud word embedding vectors; the converter-based Bert model of the warehouse environment point cloud context encoder including the embedding layer is used to perform global context semantic encoding on the sequence of environment point cloud word embedding vectors to obtain multiple warehouse environment point cloud semantic feature vectors.

[0042] Furthermore, after arranging multiple warehouse environment point cloud semantic feature vectors in two dimensions, they are processed by the warehouse environment point cloud feature encoder in order to effectively capture and encode the spatial relationship and semantic association information in the point cloud data. Point cloud data usually faces the challenge of high dimensionality and sparseness when processing. Arranging multiple point cloud semantic feature vectors in two dimensions, that is, organizing them into a matrix form, helps to structure the processing of these data. In a warehouse environment, different parts of the point cloud data often have different spatial positions and relationships with each other. The two-dimensional arrangement arranges the point cloud features according to their spatial structure, so that the feature encoder can take these spatial relationships into account during the encoding process. For example, the relative position of the shelves, the distribution of items, and the overall layout of the environment can all be reflected in the structure of the two-dimensional matrix, which helps to extract and understand the spatial association in the data. Processing the two-dimensionally arranged feature matrix through the warehouse environment point cloud feature encoder can further extract and integrate the high-level semantic information of the point cloud data. Specifically, the multiple warehouse environment point cloud semantic feature vectors are arranged in two dimensions to obtain a warehouse environment point cloud associated feature matrix; each layer of the warehouse environment point cloud feature encoder is used to perform the following on the input data in the forward pass of the layer: based on the convolution kernel, the input data is convolved to generate a convolution feature map; the convolution feature map is subjected to global mean pooling processing based on the feature matrix to generate a pooled feature map; and the feature values ​​at each position in the pooled feature map are nonlinearly activated to generate an activated feature augmentation map; wherein the output of the last layer of the warehouse environment point cloud feature encoder is the warehouse environment point cloud semantic associated feature vector, the input from the second layer to the last layer of the warehouse environment point cloud feature encoder is the output of the previous layer, and the input of the warehouse environment point cloud feature encoder is the warehouse environment point cloud associated feature matrix.

[0043] In the above-mentioned cargo location identification system 100 of the mobile robot, the cargo location label judgment module 130 is used to obtain the cargo location classification label based on the cargo electronic label information text semantic feature vector and the warehouse environment multimodal association feature vector. It should be understood that the cargo electronic label usually contains detailed text information about the cargo, such as product name, specification, quantity, etc. However, this information alone may not be sufficient to accurately determine the specific location of the cargo. The multimodal feature vector of the warehouse environment, including point cloud data, image information, and spatial layout, can provide rich data about the actual location and environment of the cargo in the warehouse. Combining this information can provide a more comprehensive background and context, which helps to locate the cargo more accurately. The cargo electronic label information text semantic feature vector extracts semantic information from the text through natural language processing technology, which can reveal the type, purpose and storage requirements of the cargo. For example, through the text semantic feature vector, it can be understood whether a cargo is fragile or heavy, which is crucial to determining its optimal storage location. The multimodal association feature vector of the warehouse environment integrates different types of data in the environment, such as point cloud data, images, and environmental descriptions. These feature vectors can accurately describe the layout, spatial distribution, and physical characteristics of the warehouse. For example, the location of shelves, aisle width, and the arrangement of other goods can all affect the storage location of goods. By analyzing these multimodal data, we can understand the best location in the warehouse for storing specific goods. Combining the text semantic feature vector of the goods electronic tag information with the multimodal association feature vector of the warehouse environment can generate accurate cargo location classification labels. This combination process usually involves a machine learning model. By comprehensively analyzing text semantics and environmental features, the model can learn which goods are suitable for placement in which locations in the warehouse. With accurate cargo location classification labels, the warehouse management system can achieve more efficient storage and retrieval operations.

[0044] Figure 4 FIG. 1 is a block diagram of a cargo location label determination module in a cargo location identification system of a mobile robot according to an embodiment of the present application. Figure 4 As shown, in a specific embodiment of the present application, the cargo location label judgment module 130 includes: a cargo location feature fusion unit 131, which is used to fuse the semantic feature vector of the cargo electronic label information text and the multimodal association feature vector of the warehouse environment to obtain a cargo location judgment classification feature vector; a cargo location feature optimization unit 132, which is used to perform fine-grained feature-class label coupling correlation metric optimization on the cargo location judgment classification feature vector to obtain an optimized cargo location judgment classification feature vector; a cargo location label classification judgment unit 133, which is used to pass the optimized cargo location judgment classification feature vector through a classifier to obtain a classification result, and the classification result is used to represent the cargo location classification label.

[0045] It should be understood that the multimodal feature vector of the warehouse environment integrates information such as spatial layout, shelf location, aisle width, etc. These features are crucial to the actual storage location of the goods. By analyzing this data, the system can identify the spatial areas suitable for storing specific goods and evaluate the suitability of the goods in the actual environment.

[0046] In particular, in the technical solution of the present application, data from different data sources, namely, cargo electronic tag information, warehouse multi-area shelf images, and warehouse environment point cloud data, are integrated into a comprehensive cargo location judgment classification feature vector. Considering that each data source has its own unique information and expression, for example, the camera provides visual information, while the lidar provides point cloud data, the complex correlations and noise between different data sources may have a negative impact on the output of the classifier, resulting in misclassification or deviation, that is, it may cause the features of certain categories to be mistakenly emphasized or ignored, resulting in category coherent interference. Therefore, in the technical solution of the present application, the cargo location judgment classification feature vector is optimized by fine-grained feature-class label coupling correlation metric to obtain an optimized cargo location judgment classification feature vector.

[0047] Among them, the cargo location judgment classification feature vector is optimized by fine-grained feature-class label coupling correlation measurement to obtain an optimized cargo location judgment classification feature vector, including: calculating the product between the cargo location judgment classification feature vector and its transposed vector to obtain a cargo location judgment feature autocorrelation expression matrix; performing correlation measurement on the class decoding weight matrix of the classifier and the cargo location judgment feature autocorrelation expression matrix to obtain a cargo location judgment feature-class weight fine-grained coupling representation matrix; performing probabilistic processing on the cargo location judgment feature-class weight fine-grained coupling representation matrix based on the Sigmoid activation function to obtain a cargo location judgment feature-class weight fine-grained coupling probabilistic representation matrix; performing back-propagation expression compensation on the cargo location judgment classification feature vector based on the cargo location judgment feature-class weight fine-grained coupling probabilistic representation matrix to obtain a class-related compensation representation vector; and calculating the position-weighted sum between the class-related compensation representation vector and the cargo location judgment classification feature vector to obtain the optimized cargo location judgment classification feature vector.

[0048] The following correlation measurement formula is used to measure the correlation between the category decoding weight matrix of the classifier and the autocorrelation expression matrix of the cargo location judgment feature to obtain a cargo location judgment feature-category weight fine-grained coupling representation matrix;

[0049] Wherein, the correlation measurement formula is:

[0050]

[0051] Among them, X 1i represents the i-th row vector of the class decoding weight matrix of the classifier, X 2j represents the j-th row vector of the cargo location judgment feature autocorrelation expression matrix, S represents the covariance matrix of the data set to which the i-th row vector of the category decoding weight matrix of the classifier and the j-th row vector of the cargo location judgment feature autocorrelation expression matrix belong, S -1 represents the inverse matrix of the covariance matrix, M i,j It represents the eigenvalue of the (i, j)th position of the cargo location judgment feature-category weight fine-grained coupling probabilistic representation matrix.

[0052] Among them, the cargo location judgment classification feature vector is back-propagated and expressed to obtain a category-related compensation representation vector based on the cargo location judgment feature-category weight fine-grained coupling probabilistic representation matrix using the following compensation formula;

[0053] Wherein, the compensation formula is:

[0054]

[0055] Wherein, V represents the category-related compensation representation vector, M represents the cargo location judgment feature-category weight fine-grained coupling probabilistic representation matrix, V c represents the cargo location judgment classification feature vector, Represents matrix multiplication.

[0056] In the technical solution of the present application, when the cargo location judgment classification feature vector is classified by the classifier, since the weight of the classifier also needs to adapt to the cargo location judgment classification feature vector, the category coherent interference with the cargo location judgment classification feature vector may occur. Based on this, in the technical solution of the present application, the cargo location judgment classification feature vector is optimized by fine-grained feature-class label coupling correlation measurement, which first represents the autocorrelation interference feature spectrum with the autocorrelation matrix of the cargo location judgment classification feature vector, and then simulates the fine-grained feature-class label coherent interference phase between the cargo location judgment feature and the classifier weight matrix by calculating the correlation measurement between the autocorrelation matrix of the cargo location judgment classification feature vector and the class decoding weight matrix of the classifier, and then back-propagates the expression compensation of the cargo location judgment classification feature vector with the cargo location judgment feature-class weight fine-grained coupling representation matrix, and restores the equivalent probability intensity representation of the cargo location judgment classification feature vector without interference by reverse compensation, and obtains the optimized cargo location judgment classification feature vector to improve the accuracy of the classification result.

[0057] Furthermore, the optimized cargo location judgment classification feature vector contains enough information to describe the characteristics of the cargo and its suitable storage location. By classifying these features, the optimal location of the cargo in the warehouse can be effectively determined, thereby improving the accuracy and efficiency of warehouse management. The classifier uses a training data set (i.e., samples of known cargo location and environmental characteristics) to learn how to map feature vectors to corresponding cargo location classification labels. By inputting the optimized cargo location judgment classification feature vector into the classifier, the classifier can output a classification result, which represents the specific classification label of the cargo. This label represents the ideal location or category of the cargo in the warehouse, such as "high shelf" or "near the entrance and exit area". This classification result is directly related to the actual storage location of the cargo. After obtaining the classification results, the warehouse management system can accurately place and schedule the cargo according to these labels. By reasonably classifying the cargo, more efficient inventory management can be achieved, the storage location of the cargo can be optimized, the time for finding and moving the cargo can be reduced, and the overall operational efficiency can be improved. Using a classifier for cargo location classification can greatly improve the automation level of warehouse management. Traditional warehouse management often relies on manual operation and judgment. The application of classifiers can reduce human errors and subjective judgments and improve the intelligence of warehouse management. This automated classification system can process large amounts of data in real time, make decisions quickly, and adapt to the dynamically changing warehouse environment.

[0058] In summary, the embodiment of the present application first obtains the electronic tag information of the goods, the multi-area shelf images of the warehouse collected by the camera, and the warehouse environment point cloud data scanned by the lidar, and then uses deep learning technology to perform feature extraction and association analysis on the three, and finally obtains the classification result through the classifier to obtain the cargo location classification label, thereby realizing automated inventory management, reducing human errors, and optimizing the use of warehouse space.

[0059] As described above, the cargo location identification system 100 of the mobile robot according to the embodiment of the present application can be implemented in various terminal devices. In one example, the cargo location identification system 100 of the mobile robot can be integrated into the terminal device as a software module and / or a hardware module. For example, the cargo location identification system 100 of the mobile robot can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the cargo location identification system 100 of the mobile robot can also be one of the many hardware modules of the terminal device.

[0060] Alternatively, in another example, the cargo location identification system 100 of the mobile robot and the terminal device may be separate devices, and the cargo location identification system 100 of the mobile robot may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in an agreed data format.

[0061] Figure 5 FIG. 1 is a flow chart of a method for identifying cargo locations of a mobile robot according to an embodiment of the present application. Figure 5 As shown, according to the cargo location identification method of a mobile robot of an embodiment of the present application, it includes: S110, acquiring cargo electronic tag information, multi-area shelf images of a warehouse collected by a camera, and warehouse environment point cloud data scanned by a laser radar; S120, extracting a text semantic feature vector of the cargo electronic tag information and a multimodal association feature vector of the warehouse environment from the cargo electronic tag information, the multi-area shelf images of the warehouse collected by the camera, and the multimodal association feature vector of the warehouse environment scanned by the laser radar; S130, obtaining a cargo location classification label based on the cargo electronic tag information text semantic feature vector and the warehouse environment multimodal association feature vector.

[0062] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned mobile robot cargo position identification method have been described in the above reference. Figures 1 to 4 The description of the cargo location recognition system of the mobile robot has been described in detail, and therefore, its repeated description will be omitted.

[0063] Below, reference Figure 6 To describe an electronic device according to an embodiment of the present application.

[0064] like Figure 6 As shown, the electronic device 10 includes an input device 11, an input interface 12, a central processing unit 13, a memory 14, an output interface 15, an output device 16, and a bus 17. The input interface 12, the central processing unit 13, the memory 14, and the output interface 15 are connected to each other through the bus 17, and the input device 11 and the output device 16 are connected to the bus 17 through the input interface 12 and the output interface 15, respectively, and then connected to other components of the electronic device 10.

[0065] Specifically, the input device 11 receives input information from the outside and transmits the input information to the central processing unit 13 through the input interface 12; the central processing unit 13 processes the input information based on the computer executable instructions stored in the memory 14 to generate output information, temporarily or permanently stores the output information in the memory 14, and then transmits the output information to the output device 16 through the output interface 15; the output device 16 outputs the output information to the outside of the electronic device 10 for user use.

[0066] In one embodiment, Figure 6The electronic device 10 shown can be implemented as a network device, which may include: a memory configured to store programs; a processor configured to run the programs stored in the memory to execute any one of the mobile robot cargo location identification methods described in the above embodiments.

[0067] According to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed from a removable storage medium.

[0068] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0069] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present application, but the present application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of the present application, and these modifications and improvements are also considered to be within the scope of protection of the present application.

Claims

1. A cargo location identification system for a mobile robot, characterized in that: include: The mobile robot cargo location data acquisition module is used to obtain cargo electronic tag information, warehouse multi-area shelf images collected by cameras, and warehouse environment point cloud data scanned by laser radar; A mobile robot cargo storage location data extraction module is used to extract the cargo electronic tag information text semantic feature vector and the warehouse environment multimodal association feature vector from the cargo electronic tag information, the warehouse multi-area shelf image collected by the camera and the warehouse environment point cloud data scanned by the laser radar; The cargo location label judgment module is used to obtain the cargo location classification label based on the semantic feature vector of the cargo electronic label information text and the multimodal association feature vector of the warehouse environment.

2. The cargo location identification system of the mobile robot according to claim 1, characterized in that: The mobile robot cargo location data extraction module includes: A cargo electronic label feature extraction unit, used for extracting features from the cargo electronic label information to obtain a text semantic feature vector of the cargo electronic label information; The warehouse environment multimodal feature extraction unit is used to extract features from the warehouse multi-area shelf images collected by the camera and the warehouse environment point cloud data scanned by the laser radar to obtain the warehouse environment multimodal correlation feature vector.

3. The cargo position identification system of the mobile robot according to claim 2, characterized in that: The cargo electronic tag feature extraction unit comprises: Segmenting the cargo electronic label information to obtain a cargo electronic label information word sequence; The cargo electronic label information word sequence is passed through a cargo electronic label information context text encoder based on a Taming model to obtain a cargo electronic label information text semantic feature vector.

4. The cargo location identification system of the mobile robot according to claim 3, characterized in that: The warehouse environment multimodal feature extraction unit comprises: A multi-area shelf feature encoding subunit, used for performing feature encoding on the warehouse multi-area shelf image collected by the camera to obtain a warehouse panoramic shelf environment feature vector; An environmental point cloud feature encoding subunit, used for performing feature encoding on the warehouse environment point cloud data scanned by the laser radar to obtain a semantically associated feature vector of the warehouse environment point cloud; The environment multimodal feature association subunit is used to associate the warehouse panoramic shelf environment feature vector with the warehouse environment point cloud semantic association feature vector to obtain the warehouse environment multimodal association feature vector.

5. The cargo position identification system of the mobile robot according to claim 4, characterized in that: The multi-area shelf feature encoding subunit includes: Arranging the warehouse multi-area shelf images collected by the camera into a warehouse area shelf input tensor; The warehouse area shelf input tensor is passed through a warehouse area shelf feature encoder based on a convolutional neural network to obtain the warehouse panoramic shelf environment feature vector.

6. The cargo location identification system of the mobile robot according to claim 5, characterized in that: The environmental point cloud feature encoding subunit includes: Passing the warehouse environment point cloud data scanned by the laser radar through a warehouse environment point cloud context encoder including an embedding layer to obtain a plurality of warehouse environment point cloud semantic feature vectors; The multiple warehouse environment point cloud semantic feature vectors are arranged in two dimensions and then passed through a warehouse environment point cloud feature encoder to obtain the warehouse environment point cloud semantic association feature vector.

7. The cargo location identification system of the mobile robot according to claim 6, characterized in that: The cargo location label determination module includes: A cargo location feature fusion unit, used to fuse the cargo electronic tag information text semantic feature vector and the warehouse environment multimodal association feature vector to obtain a cargo location judgment classification feature vector; A cargo location feature optimization unit, used for optimizing the correlation metric of fine-grained feature-class label coupling on the cargo location judgment classification feature vector to obtain an optimized cargo location judgment classification feature vector; The cargo storage location label classification judgment unit is used to pass the optimized cargo storage location judgment classification feature vector through a classifier to obtain a classification result, and the classification result is used to represent the cargo storage location classification label.

8. The cargo location identification system of the mobile robot according to claim 7, characterized in that: The cargo location feature optimization unit includes: Calculate the product between the cargo location judgment classification feature vector and its transposed vector to obtain a cargo location judgment feature autocorrelation expression matrix; Performing correlation measurement on the category decoding weight matrix of the classifier and the cargo location judgment feature autocorrelation expression matrix to obtain a cargo location judgment feature-category weight fine-grained coupling representation matrix; The cargo location judgment feature-category weight fine-grained coupling representation matrix is ​​subjected to probabilistic processing based on a Sigmoid activation function to obtain a cargo location judgment feature-category weight fine-grained coupling probabilistic representation matrix; Based on the cargo location judgment feature-category weight fine-grained coupling probabilistic representation matrix, back-propagation expression compensation is performed on the cargo location judgment classification feature vector to obtain a category-related compensation representation vector; The position-weighted sum between the category-related compensation representation vector and the cargo location judgment classification feature vector is calculated to obtain the optimized cargo location judgment classification feature vector.

9. A method for identifying cargo positions of a mobile robot, characterized in that: include: Obtain cargo electronic tag information, warehouse multi-area shelf images captured by cameras, and warehouse environment point cloud data scanned by LiDAR; Extracting a text semantic feature vector of the cargo electronic tag information and a multimodal correlation feature vector of the warehouse environment from the cargo electronic tag information, the warehouse multi-area shelf image collected by the camera, and the warehouse environment point cloud data scanned by the laser radar; Based on the semantic feature vector of the cargo electronic tag information text and the multimodal association feature vector of the warehouse environment, a cargo location classification label is obtained.

10. The method for identifying cargo locations of a mobile robot according to claim 9, characterized in that: Extracting a text semantic feature vector of the goods electronic label information and a multimodal correlation feature vector of the warehouse environment from the goods electronic label information, the warehouse multi-area shelf image collected by the camera, and the warehouse environment point cloud data scanned by the laser radar, including: Extracting features of the cargo electronic label information to obtain a text semantic feature vector of the cargo electronic label information; Feature extraction is performed on the warehouse multi-area shelf images collected by the camera and the warehouse environment point cloud data scanned by the laser radar to obtain the multimodal association feature vector of the warehouse environment.

Images