Material cage identification system and method of mobile robot

Abstract

The invention relates to the field of material cage recognition, and particularly discloses a material cage recognition system and method of a mobile robot, and the method comprises the steps: firstly obtaining a material cage environment panoramic image collected by a camera and material cage space coordinate position data collected by a laser scanner, and then carrying out the feature extraction and correlation analysis of the two images through a deep learning technology, and finally, whether the mobile robot can stack the material cages or not is judged through a classifier, so that manual intervention is reduced, the efficiency is improved, and the operation error rate is reduced.

Description

Technical Field

[0001] The present application relates to the field of cage identification, and more specifically, to a cage identification system and method for a mobile robot. Background Art

[0002] With the development of warehousing and logistics technology, in the process of warehousing operations, material cages are often used to transport and store goods, and by stacking material cages, three-dimensional warehousing can be achieved, reducing the occupation of storage space.

[0003] In traditional warehouse scenarios, every time a driver drives a forklift to stack cages, he needs to stick his head out of the forklift to check the status of the cage to determine whether the cage can be stacked. However, every time a cage is stacked, the driver needs to manually confirm the stacking, which reduces the efficiency of cage stacking.

[0004] Therefore, a cage identification system and method for a mobile robot is desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a cage recognition system and method for a mobile robot, which first obtains a panoramic image of the cage environment collected by a camera and the spatial coordinate position data of the cage collected by a laser scanner, and then uses deep learning technology to perform feature extraction and association analysis on the two, and finally uses a classifier to determine whether the mobile robot can stack the cage, thereby reducing manual intervention, improving efficiency, and reducing the error rate of operation.

[0006] According to one aspect of the present application, a cage identification system for a mobile robot is provided, comprising:

[0007] The robot cage recognition data acquisition module is used to obtain the cage environment panoramic image collected by the camera and the cage space coordinate position data collected by the laser scanner;

[0008] A robot cage recognition data extraction module is used to extract a cage environment panoramic multimodal association feature vector and a cage space coordinate position semantic feature vector from the cage environment panoramic image collected by the camera and the cage space coordinate position data collected by the laser scanner;

[0009] The mobile robot cage stacking judgment module is used to judge whether the mobile robot can stack the cages based on the panoramic multimodal association feature vector of the cage environment and the semantic feature vector of the cage space coordinate position.

[0010] According to another aspect of the present application, a cage identification method for a mobile robot is provided, comprising:

[0011] Acquire the panoramic image of the cage environment collected by the camera and the spatial coordinate position data of the cage collected by the laser scanner;

[0012] Extracting a cage environment panoramic multimodal association feature vector and a cage space coordinate position semantic feature vector from the cage environment panoramic image collected by the camera and the cage space coordinate position data collected by the laser scanner;

[0013] Based on the panoramic multimodal association feature vector of the cage environment and the semantic feature vector of the cage spatial coordinate position, it is determined whether the mobile robot can stack the cages.

[0014] Compared with the prior art, the present application provides a material cage recognition system and method for a mobile robot, which first obtains a panoramic image of the material cage environment captured by a camera and the spatial coordinate position data of the material cage captured by a laser scanner, and then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally uses a classifier to determine whether the mobile robot can stack the material cages, thereby reducing manual intervention, improving efficiency, and reducing the operational error rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 It is a block diagram of a cage identification system of a mobile robot according to an embodiment of the present application.

[0017] Figure 2 It is a block diagram of a robot cage recognition data extraction module in a cage recognition system of a mobile robot according to an embodiment of the present application.

[0018] Figure 3 It is a block diagram of a cage environment panoramic feature extraction unit in a cage recognition system of a mobile robot according to an embodiment of the present application.

[0019] Figure 4 It is a block diagram of a mobile robot cage stacking judgment module in a mobile robot cage recognition system according to an embodiment of the present application.

[0020] Figure 5 This is a flow chart of a cage identification method for a mobile robot according to an embodiment of the present application.

[0021] Figure 6is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0023] Figure 1 Schematic diagram of a cage identification system for a mobile robot according to an embodiment of the present application. Figure 1 As shown, the material cage recognition system 100 of the mobile robot according to the embodiment of the present application includes: a robot material cage recognition data acquisition module 110, which is used to obtain a panoramic image of the material cage environment collected by a camera and a material cage spatial coordinate position data collected by a laser scanner; a robot material cage recognition data extraction module 120, which is used to extract a material cage environment panoramic multimodal association feature vector and a material cage spatial coordinate position semantic feature vector from the material cage environment panoramic image collected by the camera and the material cage spatial coordinate position data collected by the laser scanner; a mobile robot material cage stacking judgment module 130, which is used to judge whether the mobile robot can stack the material cages based on the material cage environment panoramic multimodal association feature vector and the material cage spatial coordinate position semantic feature vector.

[0024] In the above-mentioned mobile robot cage recognition system 100, the robot cage recognition data acquisition module 110 is used to obtain the cage environment panoramic image collected by the camera and the cage space coordinate position data collected by the laser scanner. It should be understood that with the advancement of warehousing and logistics technology, cages are widely used in the transportation and storage of goods. By stacking cages, three-dimensional warehousing can be achieved, thereby effectively saving storage space. In traditional warehousing operations, when stacking cages, forklift drivers usually need to extend their heads out of the forklift to observe the state of the cage to determine whether stacking is possible. This approach is not only time-consuming, but also relies on the driver's manual operation to confirm the feasibility of stacking, thereby reducing the overall efficiency of cage stacking. Therefore, in the technical solution of the present application, by obtaining the cage environment panoramic image collected by the camera and the cage space coordinate position data collected by the laser scanner, and combining deep learning technology, it is determined whether the mobile robot can stack the cage, thereby reducing manual intervention, improving efficiency and reducing operational errors.

[0025] Specifically, obtaining the panoramic image of the cage environment collected by the camera and the spatial coordinate position data of the cage collected by the laser scanner is the key to achieving efficient cage identification and stacking. The panoramic image of the cage environment provided by the camera provides the system with visual information of the cage's surroundings, helping to identify the position, shape, size and other visual features of the cage. Through these image data, the system can capture the appearance characteristics of the cage and the spatial layout of the surrounding environment, which is crucial for understanding the actual situation and environmental conditions of the cage. The spatial coordinate position data of the cage provided by the laser scanner provides accurate spatial coordinate information, depicting the position and geometry of the cage in three-dimensional space. Unlike the two-dimensional image of the camera, the laser scanner can accurately measure the height, width and depth of the cage to generate a detailed three-dimensional model. This accurate spatial data is very important for evaluating the actual size of the cage and the spatial relationship with other cages or objects, which helps to determine whether it can be stacked safely and effectively. By combining these two types of data, the system can comprehensively consider the visual characteristics and spatial position of the cage to make more accurate stacking decisions. Obtaining these two types of data is the basis for realizing an automated cage stacking system, which can improve recognition accuracy and operational safety.

[0026] In the above-mentioned mobile robot cage recognition system 100, the robot cage recognition data extraction module 120 is used to extract the cage environment panoramic multimodal association feature vector and the cage space coordinate position semantic feature vector from the cage environment panoramic image collected by the camera and the cage space coordinate position data collected by the laser scanner. It should be understood that integrating this information together can form a more comprehensive and accurate cage model, helping the system to consider more factors when making stacking decisions. Doing so not only improves the accuracy of recognition, but also ensures the safety and effectiveness of operations in complex environments. Therefore, extracting and integrating these feature vectors is a key step in realizing an automated cage handling system.

[0027] Figure 2 FIG. 1 is a block diagram of a robot cage recognition data extraction module in a cage recognition system of a mobile robot according to an embodiment of the present application. Figure 2 As shown, in a specific embodiment of the present application, the robot cage recognition data extraction module 120 includes: a cage environment panoramic feature extraction unit 121, used to perform feature extraction on the cage environment panoramic image collected by the camera to obtain the cage environment panoramic multimodal association feature vector; a cage space coordinate position feature extraction unit 122, used to perform feature extraction on the cage space coordinate position data collected by the laser scanner to obtain the cage space coordinate position semantic feature vector.

[0028] It should be understood that the panoramic images captured by the camera provide detailed visual information of the cage and its surroundings. These images include the appearance characteristics, color, texture, and spatial layout of the cage. These visual features are very important for identifying the specific type of cage, its location, and its relative relationship with other objects. For example, by analyzing the color and texture in the image, different types of cages can be distinguished, and by identifying the edges and shapes of the cages, its spatial layout can be determined. The purpose of feature extraction is to convert this complex visual data into a more concise form that can be used for subsequent processing, namely feature vectors. These feature vectors contain important key information in the image, such as the geometry of the cage, its relative position, and possible occlusion. The multimodal association feature vector further integrates this information, taking into account the spatial distribution and contextual relationships in the image, thereby providing a more comprehensive description of the environment.

[0029] Furthermore, feature extraction of the cage spatial coordinate position data collected by the laser scanner is performed to accurately describe the position and layout of the cage in three-dimensional space, which is essential for cage identification, processing and operation in the automation system. Among them, the laser scanner can provide high-precision three-dimensional spatial data, which accurately captures the actual spatial coordinates, shape and size of the cage. This fine spatial information can help the system better understand the geometric characteristics of the cage and its specific location in the environment. By extracting features from these spatial data, the system can convert complex three-dimensional coordinate information into a more concise and useful form, namely semantic feature vectors, which extract the core features related to the spatial position of the cage. The feature extraction process involves processing the point cloud data obtained by laser scanning to obtain feature vectors representing the spatial coordinate position of the cage. These feature vectors contain information about the cage's geometry, size, and relative position to other objects in the environment. For example, by extracting feature vectors, the system can understand the exact size, location, and possible spatial gap of the cage, so as to determine whether the cage is suitable for the current operation requirements.

[0030] Figure 3 FIG. 1 is a block diagram of a cage environment panoramic feature extraction unit in a cage recognition system of a mobile robot according to an embodiment of the present application. Figure 3 As shown, in a specific embodiment of the present application, the cage environment panoramic feature extraction unit 121 includes: a cage environment panoramic attention feature encoding subunit 1211, which is used to perform attention feature encoding on the cage environment panoramic image collected by the camera to obtain a cage environment panoramic spatial feature map and a cage environment panoramic channel feature map; a cage environment panoramic multimodal feature aggregation subunit 1212, which is used to perform feature aggregation on the cage environment panoramic spatial feature map and the cage environment panoramic channel feature map to obtain the cage environment panoramic multimodal association feature vector.

[0031] It should be understood that the panoramic images captured by the camera contain a large amount of visual information, including not only the specific appearance of the cage, but also the background of the surrounding environment, other objects, and changes in lighting. This information contains features that are crucial to the recognition and processing of cages. However, image data often contains a large amount of redundant or irrelevant information, so it is particularly important to extract effective features from it. Through attention feature encoding, the system can intelligently focus on key areas in the image, such as the position and shape of the cage, thereby filtering out irrelevant information and improving the accuracy of feature extraction. Attention feature encoding weights important features so that the system can quickly extract useful information from massive image data. This method not only improves the accuracy of recognition, but also significantly improves computational efficiency and reduces unnecessary data processing overhead.

[0032] Furthermore, feature aggregation of the panoramic spatial feature map of the cage environment and the panoramic channel feature map of the cage environment is to effectively integrate different types of feature information, so as to obtain a comprehensive understanding of the cage and its environment. The panoramic spatial feature map of the cage environment mainly describes the spatial layout and positional relationship in the image, such as the position, boundary and relative relationship of the cage with the surrounding environment. The channel feature map focuses on the different color and texture features in the image, which are very important for identifying the appearance, material and possible markings of the cage. By aggregating these two feature maps, spatial information can be combined with visual information to form a comprehensive and multi-dimensional feature representation, which helps to fully understand the actual situation of the cage. Among them, although a single feature map can provide specific information, it often cannot fully capture the complex characteristics of the cage in the environment. The spatial feature map can show the geometric position of the cage, while the channel feature map reveals its visual characteristics. Feature aggregation of the two can provide more accurate cage recognition and positioning capabilities. For example, by combining the features of spatial position and visual appearance, the system can more accurately judge the type, state and relationship of the cage with other objects, thereby improving the overall recognition accuracy.

[0033] In a specific embodiment of the present application, the cage environment panoramic attention feature encoding subunit 1211 includes: passing the cage environment panoramic image captured by the camera through the cage environment panoramic spatial attention mechanism to obtain the cage environment panoramic spatial feature map; passing the cage environment panoramic image captured by the camera through the cage environment panoramic channel attention mechanism to obtain the cage environment panoramic channel feature map.

[0034] It should be understood that in the panoramic image of the cage environment, although it contains rich visual information, not all areas are critical for identifying the cage or its environment. The spatial attention mechanism can automatically identify the most informative areas in the image and focus attention on these key areas. For example, through the attention mechanism, the system can highlight the edges, position and relationship of the cage with other objects in the environment, while ignoring irrelevant details in the background. This ability to focus on key areas improves the accuracy and relevance of feature extraction. The panoramic spatial feature map of the cage environment mainly involves the spatial layout and geometric features of objects in the image. The spatial attention mechanism can effectively capture and represent the position, size and relative position relationship of the cage in the environment by weighting the spatial information of the image. Considering that traditional image processing methods often need to process a large amount of pixel information, this may lead to high computational overhead and inefficiency. The spatial attention mechanism can effectively reduce the amount of unnecessary data processing by weighting the image and emphasizing the features of important areas. This approach enables the system to extract useful information more quickly, thereby improving processing efficiency. Specifically, the panoramic image of the cage environment collected by the camera is subjected to deep convolution encoding using the convolution encoding part of the cage environment panoramic spatial attention mechanism to obtain an initial convolution feature map; the initial convolution feature map is input into the spatial attention part of the cage environment panoramic spatial attention mechanism to obtain a spatial attention map; the spatial attention map is activated by a Softmax function to obtain a spatial attention feature map; and the spatial attention feature map and the initial convolution feature map are multiplied by the position points to obtain the cage environment panoramic spatial feature map. More specifically, the initial convolution feature map is input into the spatial attention part of the cage environment panoramic spatial attention mechanism to obtain the spatial attention map, including: performing average pooling and maximum pooling along the channel dimension on the initial convolution feature map to obtain an average feature matrix and a maximum feature matrix; cascading and channel-adjusting the average feature matrix and the maximum feature matrix to obtain a channel feature matrix; and convolution encoding the channel feature matrix using the convolution layer of the spatial attention feature map to obtain a spatial attention map.

[0035] Furthermore, the panoramic image of the cage environment captured by the camera is processed by the cage environment panoramic channel attention mechanism in order to effectively extract channel feature information in the image, such as color and texture, which is essential for accurately identifying and understanding the cage and its environment. In the panoramic image captured by the camera, different channels (such as red, green, and blue channels) carry different types of information, which is essential for the recognition and classification of the cage. For example, the color channel can provide the appearance characteristics of the cage, while the texture channel reveals the details of the cage surface. The channel attention mechanism weights the features of these different channels to highlight the channel features that are helpful for recognition, thereby improving the effect of feature extraction. The channel attention mechanism can dynamically adjust its attention to features according to the contribution of each channel. Through this mechanism, the system can automatically identify which channel features are most important in the current task and enhance these features accordingly. This dynamic weighting improves the accuracy of feature extraction, so that the generated channel feature map can more accurately reflect the visual features of the cage, such as color and texture. Considering that the panoramic image contains multiple information sources, such as color, brightness, and texture, this information can be effectively fused through the channel attention mechanism. By focusing on the features of different channels, the system can understand the visual features of the cage more comprehensively. Specifically, each layer of the cage environment panoramic channel attention mechanism performs the following operations on the input data in the forward pass of the layer: convolution processing on the input data based on a two-dimensional convolution kernel to generate a convolution feature map; pooling processing on the convolution feature map to generate a pooling feature map; activation processing on the pooling feature map to generate an activation feature map; calculating the quotient of the eigenvalue mean of the feature matrix corresponding to each channel in the activation feature map and the sum of the eigenvalue mean of the feature matrix corresponding to all channels as the weighting coefficient of the feature matrix corresponding to each channel; and weighting the feature matrix of each channel with the weighting coefficient of each channel in the activation feature map to generate a channel attention feature map; wherein the output of the last layer of the cage environment panoramic channel attention mechanism is the cage environment panoramic channel feature map.

[0036] In a specific embodiment of the present application, the cage environment panoramic multimodal feature aggregation subunit 1212 includes: passing the cage environment panoramic spatial feature map and the cage environment panoramic channel feature map through a cage environment panoramic multimodal association convolutional neural network as a feature encoder to obtain a cage environment panoramic multimodal association feature map; pooling the cage environment panoramic multimodal association feature map to obtain the cage environment panoramic multimodal association feature vector.

[0037] It should be understood that although a single spatial feature map or channel feature map can provide a certain perspective, it often cannot fully capture all the key information in the environment. By fusing these two feature maps, the multimodal association convolutional neural network can comprehensively consider the characteristics of space and channels, thereby generating a richer and more expressive multimodal association feature map. Among them, the panoramic spatial feature map of the cage environment mainly focuses on the spatial layout and geometric information in the image, while the panoramic channel feature map of the cage environment focuses on the color and texture features of the image. By combining these two feature maps, the information in the image can be fully understood from multiple angles. The spatial feature map provides the location and shape information of the object, while the channel feature map provides the visual details and color information of the object. This combination can more accurately describe the actual environment of the cage, thereby improving recognition accuracy. Specifically, each layer of the cage environment panoramic multimodal association convolutional neural network used as a feature encoder performs convolution processing, mean pooling processing based on the local feature matrix and nonlinear activation processing on the input data in the forward transfer of the layer, so that the last layer of the cage environment panoramic multimodal association convolutional neural network used as a feature encoder outputs the cage environment panoramic multimodal association feature map, wherein the input of the cage environment panoramic multimodal association convolutional neural network used as a feature encoder is the cage environment panoramic spatial feature map and the cage environment panoramic channel feature map.

[0038] Furthermore, the panoramic multimodal correlation feature map of the cage environment is pooled to achieve dimensionality reduction, extraction and summarization of features, thereby optimizing the performance and efficiency of the model. The panoramic multimodal correlation feature map of the cage environment usually contains a large amount of high-dimensional data, which may lead to large computational workload and high storage requirements. Through pooling, these high-dimensional feature maps can be converted into low-dimensional feature vectors to reduce the computational burden and storage requirements. Pooling effectively compresses data by selecting important areas in the feature map and retaining key feature information. The pooling operation can extract important statistical information from the feature map, such as the maximum value or average value, which can represent the main features of the feature map. In this way, pooling helps summarize and integrate the key information in the image, so that the final feature vector can concisely and comprehensively express the panoramic features of the cage environment.

[0039] In a specific embodiment of the present application, the cage spatial coordinate position feature extraction unit 122 includes: segmenting the cage spatial coordinate position data collected by the laser scanner to obtain a cage spatial coordinate position word sequence; passing the cage spatial coordinate position word sequence through a cage spatial coordinate position semantic encoder to obtain the cage spatial coordinate position semantic feature vector.

[0040] It should be understood that the spatial coordinate data collected by the laser scanner is usually high-dimensional continuous data, which may contain a large number of original coordinate points, and these data are difficult to process directly in their original form. Through word segmentation, that is, dividing these continuous data points into several meaningful units (i.e., "words"), complex data structures can be converted into structured word sequences. This structured data form is easier to analyze, process and understand. After the spatial coordinate data is segmented, the complexity of the data can be greatly reduced. The word sequence after word segmentation usually has a fixed length and clear boundaries, which makes subsequent calculations and processing more efficient. Compared with processing the original continuous coordinate data, the computing resources and time required to operate and analyze word sequences are greatly reduced.

[0041] Furthermore, the cage spatial coordinate position word sequence is processed through the cage spatial coordinate position semantic encoder in order to convert the discrete word sequence of spatial coordinates into a meaningful vector representation with semantic information. The spatial coordinate position word sequence is only a discrete and basic spatial data representation, which is difficult to directly express complex spatial relationships and features. Through the semantic encoder, the word sequence can be converted into a feature vector with rich semantic information, making the expression of spatial data more comprehensive and in-depth. This vectorized representation can capture the implicit patterns and relationships in the spatial data and provide useful information for further analysis. As the output of the semantic encoder, the feature vector can provide a more expressive and structured input for the machine learning model. Compared with the original word sequence, the semantic feature vector has a higher dimension and complexity. Specifically, the embedding layer of the cage spatial coordinate position semantic encoder is used to map each cage spatial coordinate position word in the cage spatial coordinate position word sequence into a cage spatial coordinate position word embedding vector to obtain a sequence of cage spatial coordinate position word embedding vectors; the converter-based Bert model of the cage spatial coordinate position semantic encoder is used to perform global contextual semantic encoding on the sequence of cage spatial coordinate position word embedding vectors to obtain multiple cage spatial coordinate position feature vectors; and the multiple cage spatial coordinate position feature vectors are cascaded to obtain the cage spatial coordinate position semantic feature vector.

[0042] In the above-mentioned mobile robot cage identification system 100, the mobile robot cage stacking judgment module 130 is used to judge whether the mobile robot can stack the cage based on the panoramic multimodal association feature vector of the cage environment and the semantic feature vector of the cage space coordinate position. It should be understood that the panoramic multimodal association feature vector of the cage environment provides comprehensive information about the cage environment, including spatial layout, object distribution and other environmental features. These features help the system understand the positional relationship and spatial configuration of the cage in the environment. The cage space coordinate position semantic feature vector provides detailed position and semantic information of the cage itself, such as size, shape and relative position. Combining these two features, the environmental conditions and cage status can be fully understood, so as to make more accurate judgments. By analyzing the panoramic feature vector and the semantic feature vector, the system can evaluate whether there is enough space for stacking, whether the cage meets the stacking requirements, and whether obstacles will be encountered during the stacking process. This comprehensive analysis can effectively prevent potential collisions and stacking instability.

[0043] Figure 4 FIG. 1 is a block diagram of a mobile robot cage stacking judgment module in a mobile robot cage recognition system according to an embodiment of the present application. Figure 4 As shown, in a specific embodiment of the present application, the mobile robot cage stacking judgment module 130 includes: a mobile robot cage stacking feature fusion unit 131, which is used to fuse the cage environment panoramic multimodal association feature vector and the cage spatial coordinate position semantic feature vector to obtain a cage stacking judgment classification feature vector; a mobile robot cage stacking feature optimization unit 132, which is used to perform weight-adapted fine-grained feature-class label coupling optimization on the cage stacking judgment classification feature vector to obtain an optimized cage stacking judgment classification feature vector; a cage stacking classification judgment unit 133, which is used to pass the optimized cage stacking judgment classification feature vector through a classifier to obtain a classification result, and the classification result is used to judge whether the mobile robot can stack the cages.

[0044] It should be understood that the fusion of the panorama multimodal correlation feature vector of the cage environment and the semantic feature vector of the cage spatial coordinate position to obtain the cage stacking judgment classification feature vector is to comprehensively consider the comprehensive information of the environment and the cage to improve the accuracy and efficiency of stacking decisions. Such fusion not only optimizes the decision-making process, but also enhances the robustness and intelligent decision-making ability of the system, enabling the robot to perform stacking tasks more effectively in complex environments.

[0045] In particular, in the technical solution of the present application, the weight of the classifier determines how to allocate data points of different categories in the feature space. In the cage stacking judgment problem, the classifier needs to be able to distinguish different stacking states, such as whether the cage can be safely stacked. The decision on cage stacking depends on multiple factors, including environmental characteristics and semantic information of spatial coordinates. Due to the complexity and diversity of the cage environment, the classifier cannot correctly understand all the conditions or environmental restrictions for cage stacking during the training process, and therefore will be overly sensitive or insensitive to specific feature vector patterns, resulting in certain categories of features being incorrectly emphasized or ignored, resulting in category coherent interference. Therefore, in the technical solution of the present application, the cage stacking judgment classification feature vector is optimized based on fine-grained feature-class label coupling based on weight adaptation to obtain an optimized cage stacking judgment classification feature vector.

[0046] Among them, the cage stacking judgment classification feature vector is optimized based on fine-grained feature-class label coupling based on weight adaptation to obtain an optimized cage stacking judgment classification feature vector, including: calculating the product between the cage stacking judgment classification feature vector and its transposed vector to obtain a cage stacking judgment feature autocorrelation expression matrix; measuring the correlation between the class decoding weight matrix of the classifier and the cage stacking judgment feature autocorrelation expression matrix to obtain a cage stacking judgment feature-class weight fine-grained coupling representation matrix; performing probabilistic processing on the cage stacking judgment feature-class weight fine-grained coupling representation matrix based on the Sigmoid activation function to obtain a cage stacking judgment feature-class weight fine-grained coupling probabilistic representation matrix; performing back-propagation expression compensation on the cage stacking judgment classification feature vector based on the cage stacking judgment feature-class weight fine-grained coupling probabilistic representation matrix to obtain a class-related compensation representation vector; calculating the position-weighted sum between the class-related compensation representation vector and the cage stacking judgment classification feature vector to obtain the optimized cage stacking judgment classification feature vector.

[0047] The following correlation measurement formula is used to measure the correlation between the category decoding weight matrix of the classifier and the cage stacking judgment feature autocorrelation expression matrix to obtain the cage stacking judgment feature-category weight fine-grained coupling representation matrix;

[0048] Wherein, the correlation measurement formula is:

[0049]

[0050] Among them, X 1i represents the i-th row vector of the class decoding weight matrix of the classifier, X 2jrepresents the j-th row vector of the cage stacking judgment feature autocorrelation expression matrix, S represents the covariance matrix of the data set to which the i-th row vector of the category decoding weight matrix of the classifier and the j-th row vector of the cage stacking judgment feature autocorrelation expression matrix belong, S -1 represents the inverse matrix of the covariance matrix, M i,j Represents the eigenvalue of the (i, j)th position of the cage stacking judgment feature-category weight fine-grained coupling probabilistic representation matrix.

[0051] Among them, the following compensation formula is used to perform back propagation expression compensation on the cage stacking judgment classification feature vector based on the cage stacking judgment feature-category weight fine-grained coupling probabilistic representation matrix to obtain a category-related compensation representation vector;

[0052] Wherein, the compensation formula is:

[0053]

[0054] Among them, V r represents the category-related compensation representation vector, M represents the cage stacking judgment feature-category weight fine-grained coupling probabilistic representation matrix, V c represents the cage stacking judgment classification feature vector, Represents matrix multiplication.

[0055] In the technical solution of the present application, when the cage stacking judgment classification feature vector is classified by a classifier, since the weight of the classifier also needs to be adapted to the cage stacking judgment classification feature vector, category coherent interference with the cage stacking judgment classification feature vector may occur. Based on this, in the technical solution of the present application, the cage stacking judgment classification feature vector is optimized by fine-grained feature-class label coupling based on weight adaptation, which first represents the autocorrelation interference feature spectrum with the autocorrelation matrix of the cage stacking judgment classification feature vector, and then simulates the fine-grained feature-class label coherent interference phase between the cage stacking judgment feature and the weight matrix of the classifier by calculating the correlation measure between the autocorrelation matrix of the cage stacking judgment classification feature vector and the class decoding weight matrix of the classifier, and then uses the cage stacking judgment feature-class weight fine-grained coupling representation matrix to perform back-propagation expression compensation on the cage stacking judgment classification feature vector, and restores the equivalent probability intensity representation of the cage stacking judgment classification feature vector in the absence of interference by reverse compensation, and obtains the optimized cage stacking judgment classification feature vector to improve the accuracy of the classification result.

[0056] Furthermore, the optimized cage stacking judgment classification feature vector contains comprehensive data of environmental information and cage characteristics. These features are analyzed by the classifier, which can map complex input data to specific categories, such as "stackable" or "non-stackable". In practical applications, the stacking task of cages may involve complex environmental factors, such as space limitations, obstacles, and changes in the cage state. The classifier is able to process these complex data and make a comprehensive judgment based on the classification results of the optimized cage stacking judgment classification feature vector. This adaptability allows the robot to remain efficient and flexible in dynamic environments.

[0057] In summary, the embodiment of the present application first obtains a panoramic image of the cage environment captured by a camera and the spatial coordinate position data of the cage captured by a laser scanner, and then uses deep learning technology to perform feature extraction and correlation analysis on the two. Finally, a classifier is used to determine whether the mobile robot can stack the cages, thereby reducing manual intervention, improving efficiency, and reducing operational error rates.

[0058] As described above, the cage identification system 100 of the mobile robot according to the embodiment of the present application can be implemented in various terminal devices. In one example, the cage identification system 100 of the mobile robot can be integrated into the terminal device as a software module and / or a hardware module. For example, the cage identification system 100 of the mobile robot can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the cage identification system 100 of the mobile robot can also be one of the many hardware modules of the terminal device.

[0059] Alternatively, in another example, the cage identification system 100 of the mobile robot and the terminal device may also be separate devices, and the cage identification system 100 of the mobile robot may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0060] Figure 5 Flowchart of the cage identification method of the mobile robot according to the embodiment of the present application. Figure 5 As shown, the material cage recognition method of a mobile robot according to an embodiment of the present application includes: S110, acquiring a panoramic image of the material cage environment captured by a camera and material cage spatial coordinate position data captured by a laser scanner; S120, extracting a panoramic multimodal association feature vector of the material cage environment and a material cage spatial coordinate position semantic feature vector from the panoramic image of the material cage environment captured by the camera and the material cage spatial coordinate position data captured by the laser scanner; S130, judging whether the mobile robot can stack the material cages based on the panoramic multimodal association feature vector of the material cage environment and the material cage spatial coordinate position semantic feature vector.

[0061] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned mobile robot cage identification method have been referred to above. Figures 1 to 4 The description of the cage recognition system of the mobile robot has been introduced in detail, and therefore, its repeated description will be omitted.

[0062] Below, reference Figure 6 To describe an electronic device according to an embodiment of the present application.

[0063] like Figure 6 As shown, the electronic device 10 includes an input device 11, an input interface 12, a central processing unit 13, a memory 14, an output interface 15, an output device 16, and a bus 17. The input interface 12, the central processing unit 13, the memory 14, and the output interface 15 are connected to each other through the bus 17, and the input device 11 and the output device 16 are connected to the bus 17 through the input interface 14 and the output interface 15, respectively, and then connected to other components of the electronic device 10.

[0064] Specifically, the input device 11 receives input information from the outside and transmits the input information to the central processing unit 13 through the input interface 12; the central processing unit 13 processes the input information based on the computer executable instructions stored in the memory 14 to generate output information, temporarily or permanently stores the output information in the memory 14, and then transmits the output information to the output device 16 through the output interface 15; the output device 16 outputs the output information to the outside of the electronic device 10 for user use.

[0065] In one embodiment, Figure 6 The electronic device 10 shown can be implemented as a network device, which may include: a memory configured to store programs; a processor configured to run the programs stored in the memory to execute any one of the mobile robot cage identification methods described in the above embodiments.

[0066] According to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed from a removable storage medium.

[0067] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0068] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present application, but the present application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of the present application, and these modifications and improvements are also considered to be within the scope of protection of the present application.

Claims

1. A cage recognition system for a mobile robot, characterized in that: include: The robot cage recognition data acquisition module is used to obtain the cage environment panoramic image collected by the camera and the cage space coordinate position data collected by the laser scanner; A robot cage recognition data extraction module is used to extract a cage environment panoramic multimodal association feature vector and a cage space coordinate position semantic feature vector from the cage environment panoramic image collected by the camera and the cage space coordinate position data collected by the laser scanner; The mobile robot cage stacking judgment module is used to judge whether the mobile robot can stack the cages based on the panoramic multimodal association feature vector of the cage environment and the semantic feature vector of the cage space coordinate position.

2. The cage identification system for a mobile robot according to claim 1, characterized in that: The robot cage identification data extraction module comprises: A cage environment panoramic feature extraction unit, used for extracting features from the cage environment panoramic image captured by the camera to obtain a cage environment panoramic multimodal correlation feature vector; The cage space coordinate position feature extraction unit is used to extract features from the cage space coordinate position data collected by the laser scanner to obtain the cage space coordinate position semantic feature vector.

3. The cage identification system for a mobile robot according to claim 2, characterized in that: The cage environment panoramic feature extraction unit comprises: A cage environment panoramic attention feature encoding subunit, used for performing attention feature encoding on the cage environment panoramic image acquired by the camera to obtain a cage environment panoramic space feature map and a cage environment panoramic channel feature map; The cage environment panoramic multimodal feature aggregation subunit is used to perform feature aggregation on the cage environment panoramic space feature map and the cage environment panoramic channel feature map to obtain the cage environment panoramic multimodal association feature vector.

4. The cage identification system for a mobile robot according to claim 3, characterized in that: The cage environment panoramic attention feature encoding subunit includes: The cage environment panoramic image captured by the camera is passed through the cage environment panoramic space attention mechanism to obtain the cage environment panoramic space feature map; The cage environment panoramic image captured by the camera is passed through the cage environment panoramic channel attention mechanism to obtain the cage environment panoramic channel feature map.

5. The cage identification system for a mobile robot according to claim 4, characterized in that: The cage environment panoramic multimodal feature aggregation subunit includes: The cage environment panoramic spatial feature map and the cage environment panoramic channel feature map are passed through a cage environment panoramic multimodal association convolutional neural network as a feature encoder to obtain a cage environment panoramic multimodal association feature map; The cage environment panoramic multimodal association feature map is pooled to obtain the cage environment panoramic multimodal association feature vector.

6. The cage identification system for a mobile robot according to claim 5, characterized in that: The cage space coordinate position feature extraction unit comprises: Segmenting the cage space coordinate position data collected by the laser scanner to obtain a cage space coordinate position word sequence; The cage space coordinate position word sequence is passed through a cage space coordinate position semantic encoder to obtain the cage space coordinate position semantic feature vector.

7. The cage identification system for a mobile robot according to claim 6, characterized in that: The mobile robot cage stacking judgment module includes: A mobile robot cage stacking feature fusion unit is used to fuse the cage environment panoramic multimodal association feature vector and the cage space coordinate position semantic feature vector to obtain a cage stacking judgment classification feature vector; A mobile robot cage stacking feature optimization unit is used to perform weight-adapted fine-grained feature-class label coupling optimization on the cage stacking judgment classification feature vector to obtain an optimized cage stacking judgment classification feature vector; The cage stacking classification judgment unit is used to pass the optimized cage stacking judgment classification feature vector through a classifier to obtain a classification result, and the classification result is used to judge whether the mobile robot can stack the cages.

8. The cage identification system for a mobile robot according to claim 7, characterized in that: The mobile robot cage stacking feature optimization unit comprises: Calculating the product between the cage stacking judgment classification feature vector and its transposed vector to obtain a cage stacking judgment feature autocorrelation expression matrix; Performing correlation measurement on the category decoding weight matrix of the classifier and the cage stacking judgment feature autocorrelation expression matrix to obtain a cage stacking judgment feature-category weight fine-grained coupling representation matrix; The cage stacking judgment feature-category weight fine-grained coupling representation matrix is ​​subjected to probabilistic processing based on a Sigmoid activation function to obtain a cage stacking judgment feature-category weight fine-grained coupling probabilistic representation matrix; Based on the cage stacking judgment feature-category weight fine-grained coupling probabilistic representation matrix, back-propagation expression compensation is performed on the cage stacking judgment classification feature vector to obtain a category-related compensation representation vector; The position-weighted sum between the category-related compensation representation vector and the cage stacking judgment classification feature vector is calculated to obtain the optimized cage stacking judgment classification feature vector.

9. A cage identification method for a mobile robot, characterized in that: include: Acquire the panoramic image of the cage environment collected by the camera and the spatial coordinate position data of the cage collected by the laser scanner; Extracting a cage environment panoramic multimodal association feature vector and a cage space coordinate position semantic feature vector from the cage environment panoramic image acquired by the camera and the cage space coordinate position data acquired by the laser scanner; Based on the panoramic multimodal association feature vector of the cage environment and the semantic feature vector of the cage spatial coordinate position, it is determined whether the mobile robot can stack the cages.

10. The method for identifying a cage of a mobile robot according to claim 9, characterized in that: Extracting a panoramic multimodal correlation feature vector of the cage environment and a semantic feature vector of the cage space coordinate position from the cage environment panoramic image collected by the camera and the cage space coordinate position data collected by the laser scanner, including: Performing feature extraction on the panoramic image of the cage environment captured by the camera to obtain a panoramic multimodal correlation feature vector of the cage environment; Feature extraction is performed on the cage space coordinate position data collected by the laser scanner to obtain the cage space coordinate position semantic feature vector.

Images