Method, device, equipment and product for placing articles in storage environment

Abstract

A method, apparatus, apparatus and product for placing items in a storage environment. The method includes acquiring, by a storage robot (102, 502) through a camera (104, 504), a plurality of images (114) of a plurality of storage spaces (112, 514) in a storage environment. The method also includes obtaining a plurality of environmental information for the plurality of storage spaces (116). The method also includes obtaining item information (316) associated with an item (106, 506, 606) to be placed, the item information including a size (108, 608) of the item and an environmental demand (110) for the storage space. The method further includes generating a plurality of match degree scores corresponding to the plurality of storage spaces using a machine learning-based model based on the plurality of images, the plurality of environmental information, and the item information (120). The method further includes determining a target storage space from the plurality of storage spaces based on the plurality of match degree scores (122). In addition, the method further comprises the step of placing the article in the target storage space. Through the mode, various abnormal conditions can be flexibly processed, the operation efficiency of the system is improved, and the robustness of the system is enhanced.

Description

Methods, apparatuses, devices, and products for placing items in a warehouse environment TECHNICAL FIELD

[0001] The present disclosure relates to the field of robotics, and more specifically, to methods, apparatuses, devices, and products for placing items in a warehouse environment. BACKGROUND

[0002] Warehouse robots are a class of robots used for automated operations in warehouse environments such as warehouses or distribution centers, mainly for improving warehouse management and logistics efficiency. These robots can perform a variety of tasks such as storage, picking, carrying, sorting, and inventory management. The development of warehouse robots has greatly promoted the development of automated warehouses, helping enterprises reduce operating costs, improve production efficiency, and reduce human errors.

[0003] Warehouse robots can carry goods from one location to another, usually from a storage area to a picking or shipping area. Some warehouse robots have picking functions and can pick specific items and deliver them to designated locations. Warehouse robots can also work with automated systems to inventory and manage inventory, track the location, quantity, and status of goods in real time, and ensure the accuracy of inventory information. Warehouse robots can sort and pick items according to order requirements, thereby speeding up delivery.

[0004] SUMMARY

[0005] In a first aspect of embodiments of the present disclosure, a method for placing items in a warehouse environment is provided. The method includes obtaining, by a warehouse robot, a plurality of images of a plurality of storage spaces in the warehouse environment through a camera. The method further includes obtaining, by the warehouse robot, a plurality of environmental information of the plurality of storage spaces. The method further includes obtaining, by the warehouse robot, item information associated with an item to be placed, the item information including a size of the item and an environmental requirement of the storage space. The method further includes generating, by the warehouse robot, a plurality of matching score corresponding to the plurality of storage spaces based on the plurality of images, the plurality of environmental information, and the item information using a machine learning based model. The method further includes determining, by the warehouse robot, a target storage space from the plurality of storage spaces based on the plurality of matching scores. In addition, the method further includes placing, by the warehouse robot, the item to the target storage space.

[0006] In a second aspect of embodiments of the present disclosure, an apparatus for placing an item in a warehousing environment is provided. The apparatus includes a storage space image acquisition module configured to acquire, by a warehousing robot, a plurality of images of a plurality of storage spaces in the warehousing environment via a camera. The apparatus further includes an environment information acquisition module configured to acquire, by the warehousing robot, a plurality of environment information of the plurality of storage spaces. The apparatus further includes an item information acquisition module configured to acquire, by the warehousing robot, item information associated with an item to be placed, the item information including a size of the item and an environmental requirement of the storage space. The apparatus further includes a matching degree score generation module configured to generate, by the warehousing robot, a plurality of matching degree scores corresponding to the plurality of storage spaces based on the plurality of images, the plurality of environment information, and the item information, using a machine learning based model. The apparatus further includes a target storage space determination module configured to determine, by the warehousing robot, a target storage space from the plurality of storage spaces based on the plurality of matching degree scores. In addition, the apparatus further includes an item placing module configured to place, by the warehousing robot, the item to the target storage space.

[0007] In a third aspect of embodiments of the present disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage storing one or more programs, when executed by the one or more processors, cause the one or more processors to implement a method for placing an item in a warehousing environment. The method includes acquiring, by a warehousing robot, a plurality of images of a plurality of storage spaces in the warehousing environment via a camera. The method further includes acquiring, by the warehousing robot, a plurality of environment information of the plurality of storage spaces. The method further includes acquiring, by the warehousing robot, item information associated with an item to be placed, the item information including a size of the item and an environmental requirement of the storage space. The method further includes generating, by the warehousing robot, a plurality of matching degree scores corresponding to the plurality of storage spaces based on the plurality of images, the plurality of environment information, and the item information, using a machine learning based model. The method further includes determining, by the warehousing robot, a target storage space from the plurality of storage spaces based on the plurality of matching degree scores. In addition, the method further includes placing, by the warehousing robot, the item to the target storage space.

[0008] In a fourth aspect of the embodiments of the present disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer readable medium and includes machine executable instructions that, when executed, cause a machine to implement a method for placing an item in a warehouse environment. The method includes obtaining, by a warehouse robot, a plurality of images of a plurality of storage spaces in the warehouse environment via a camera. The method further includes obtaining, by the warehouse robot, a plurality of environment information of the plurality of storage spaces. The method further includes obtaining, by the warehouse robot, item information associated with the item to be placed, the item information including a size of the item and an environmental requirement for the storage space. The method further includes generating, by the warehouse robot, a plurality of matching scores corresponding to the plurality of storage spaces based on the plurality of images, the plurality of environment information, and the item information, utilizing a machine learning based model. The method further includes determining, by the warehouse robot, a target storage space from the plurality of storage spaces based on the plurality of matching scores. In addition, the method further includes placing, by the warehouse robot, the item to the target storage space.

[0009] The summary is provided to introduce a selection of concepts, in a simplified form, that are further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF DRAWINGS

[0010] The above and other features, aspects and advantages of various embodiments of the present disclosure will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings, in which like reference numbers represent like elements throughout. In the drawings:

[0011] FIG. 1 shows a schematic diagram of an example environment in which various embodiments of the present disclosure can be implemented;

[0012] FIG. 2 shows a flowchart of a method for placing an item in a warehouse environment, according to some embodiments of the present disclosure;

[0013] FIG. 3 shows a schematic diagram of an example matching score determination model, according to some embodiments of the present disclosure;

[0014] FIG. 4 shows a schematic diagram of an example fusion module for a matching score determination model, according to some embodiments of the present disclosure;

[0015] FIG. 5 shows a schematic diagram of an example of determining a plurality of storage spaces, according to some embodiments of the present disclosure;

[0016] FIG. 6 shows a schematic diagram of an example of determining a placement location in a target storage space, according to some embodiments of the present disclosure;

[0017] FIG. 7 illustrates a block diagram of an apparatus for placing items in a warehousing environment, according to some embodiments of the present disclosure; and

[0018] FIG. 8 illustrates a block diagram of a device capable of implementing various embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] It can be understood that all user-related data involved in the technical solution should be obtained and used after the user's authorization. This means that in the technical solution, if the user's personal information needs to be used, the user's explicit consent and authorization are required before obtaining these data, otherwise the relevant data collection and use will not be carried out. It should also be understood that in the implementation of the technical solution, relevant laws and regulations should be strictly followed in the process of data collection, use and storage, and necessary technical and measures should be taken to protect the user's data security and ensure the safe use of data.

[0020] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein, but rather these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only and are not intended to limit the scope of protection of the present disclosure.

[0021] In the description of embodiments of the present disclosure, the term "comprising" and its similar terms are understood to be open-ended, i.e., "including but not limited to". The term "based on" is understood to be "based, at least in part, on". The term "one embodiment" or "the embodiment" is understood to be "at least one embodiment". The terms "first", "second", and the like can refer to different or identical objects unless otherwise explicitly stated. Other explicit and implicit definitions can also be included below.

[0022] In traditional warehousing solutions, warehousing robots usually rely on pre-set rules, warehouse management systems, and item classification strategies to determine the storage location for placing items. For example, in some warehousing solutions, a fixed storage location can be assigned to each item, and the warehousing robot will always place the item at the same location, which has the advantage of simple operation and the robot can quickly find the item. In other warehousing solutions, the warehousing robot classifies items according to their characteristics (such as size, weight, etc.) and places them in a specific category of storage area. Each area corresponds to one or more types of items, which helps to improve the efficiency of picking and placing.

[0023] However, in these warehousing solutions, when the warehousing robots place the goods according to the pre-set rules, if the designated placement position is abnormal, the robots cannot handle it flexibly, resulting in a decrease in the performance of the entire system. For example, when the designated position has been occupied, the warehousing robot cannot handle it flexibly autonomously, but needs to send a request to the control center to obtain an updated storage position. In addition, when the environment (such as temperature and humidity, etc.) of the storage space where the designated position is located is abnormal, the warehousing robot cannot autonomously select a more suitable storage position to place the goods.

[0024] To this end, embodiments of the present disclosure provide a solution for placing goods in a warehousing environment. In this solution, the warehousing robot can obtain multiple images of multiple storage spaces in the warehousing environment through a camera. In addition, the warehousing robot can also obtain multiple environment information of the multiple storage spaces. In addition, the warehousing robot can also obtain goods information associated with the goods to be placed, the goods information including the size of the goods and the environmental requirements of the storage space. Then, the warehousing robot can generate multiple matching degree scores corresponding to the multiple storage spaces by using a machine learning-based model based on the multiple images, the multiple environment information, and the goods information. Then, the warehousing robot can determine a target storage space from the multiple storage spaces based on the multiple matching degree scores. Then, the warehousing robot can place the goods to the target storage space.

[0025] In this way, the warehousing robot can autonomously select a target storage space that meets the storage requirements of the goods from the multiple storage spaces in the warehousing environment, so as to flexibly handle various abnormal situations, improve the operating efficiency of the system, and enhance the robustness of the system. In addition, compared with selecting a storage space based on artificially labeled rules, the storage space selected by using a machine learning-based model can more accurately meet the storage requirements of the goods, thereby reducing the damage of the goods due to the mismatch between the environment of the storage space and the storage requirements of the goods.

[0026] FIG. 1 shows a schematic diagram of an example environment 100 in which multiple embodiments of the present disclosure can be implemented. As shown in FIG. 1, the environment 100 includes a robot 102, which has both grasping and transporting capabilities. For example, the robot 102 can have components such as a mechanical arm, a gripper, a fork, etc., allowing it to pick up and place goods to a specific location. The transporting capability can allow the robot 102 to move between multiple locations where multiple storage spaces are located. The robot 102 is also configured with a camera 104, which allows the robot 102 to take pictures of objects such as the warehousing environment, the storage space, or the goods to obtain corresponding images.

[0027] As shown in FIG. 1, the environment 100 includes an item 106 to be placed, which includes a size 108 and an environmental requirement 110. The size 108 can be represented in various ways, such as three-dimensional size, two-dimensional size, volume, shape, etc. The environmental requirement 110 can indicate the requirement of the item 106 on the storage environment, which depends on the physical and chemical properties of the item 106. In some embodiments, the environmental requirement 110 can include a requirement on temperature and a requirement on humidity, which can directly affect the quality and shelf life of the item. For example, for refrigerated food or perishable video, the temperature of the storage environment needs to be between 0 and 5 degrees Celsius, and the humidity needs to be between 65% and 75% relative humidity, lower temperature can slow down the growth of bacteria and yeast, and moderate humidity can prevent food from drying or excessive moisture loss. For electronic products, high temperature can cause electronic devices to overheat, high humidity can cause components to corrode or short circuit, and low humidity helps prevent static electricity, so the temperature of the storage environment needs to be between 15 and 25 degrees Celsius, and the humidity needs to be between 30% and 50% relative humidity.

[0028] As shown in FIG. 1, the environment 100 can also include a plurality of storage spaces 112-1, 112-2, …, and 112-N (collectively referred to as storage spaces 112 herein). The plurality of storage spaces 112 can have different available spaces and environmental information. In embodiments of the present disclosure, the robot 102 can use its own camera 104 to obtain images 114-1, 114-2, …, and 114-N (collectively referred to as images 114 herein) of the respective storage spaces 112, which can include information associated with the available space in the storage space 112. In addition, the robot 102 can also obtain environmental information 116-1, 116-2, …, and 116-N (collectively referred to as environmental information 116 herein) of the respective storage spaces 112 in real time, which can be obtained, for example, by sensors arranged in the storage spaces 112. For example, the environmental information 116 can include parameters of the environment in which the storage space 112 is located, such as temperature, humidity, etc.

[0029] In the environment 100, the robot 102 can obtain the size 108 of the item 106, the environmental requirement 110 of the item 106, the images 114 of the storage spaces 112, and the environmental information 116 of the storage spaces 112, to autonomously determine a storage space suitable for placing the item 106 from the plurality of storage spaces 112. In the environment 100, the matching degree determination model 118 is a model based on machine learning, which is trained to determine the matching degree between the item and the storage space. The matching degree determination model 118 can be locally deployed in the robot 102, or can be deployed on a server with which the robot 102 can communicate.

[0030] When placing the item 106, the robot 102 can determine a matching degree score between the item 106 and each storage space 112 respectively by utilizing the matching degree determination model 118. As shown in FIG. 1, the robot 102 can determine a matching degree score 120-1 between the item 106 and the storage space 112-1, a matching degree score 120-2 between the item 106 and the storage space 112-1, a matching degree score 120-N between the item 106 and the storage space 112-N, and the like (herein the matching degree scores 120-1, 120-2, …, and 120-N are also collectively referred to as the matching degree scores 120) respectively. Then, the robot 102 can determine a target storage space 122 for placing the item 106 based on the matching degree scores 120, the target storage space 122 having available space for placing the item 106 and its environment being able to meet the environmental requirement 110 of the item 106.

[0031] In this way, the robot 102 is able to autonomously select a target storage space 122 that meets the storage requirement of the item 106 from a plurality of storage spaces 112, thereby being able to flexibly handle various abnormal situations, improving the operation efficiency of the system, and enhancing the robustness of the system. In addition, compared to selecting a storage space based on artificially labeled rules, the target storage space 122 selected by utilizing a model based on machine learning (e.g., the matching degree determination model 118) is able to more accurately meet the storage requirement of the item 106, thereby being able to reduce the damage of the item due to the mismatch between the environment of the storage space and the storage requirement of the item.

[0032] FIG. 2 illustrates a flowchart of a method 200 for placing an item in a warehouse environment, according to some embodiments of the present disclosure. The method 200 can be performed by a warehouse robot, for example, the method 200 can be performed by the robot 102 in FIG. 1. As shown in FIG. 2, at block 202, the warehouse robot can acquire a plurality of images of a plurality of storage spaces in a warehouse environment by a camera. For example, in the environment 100 as shown in FIG. 1, the robot 102 can acquire a plurality of images 114 of a plurality of storage spaces 112 by the camera 104 of itself. The available space in the storage space 112 can be displayed in the images 114.

[0033] At block 204, the warehouse robot can acquire a plurality of environmental information of the plurality of storage spaces. For example, in the environment 100 as shown in FIG. 1, the robot 102 can acquire a plurality of environmental information 116 of the plurality of storage spaces 112. The environmental information 116 can be acquired by a sensor disposed in the storage space 112, for example. For example, the environmental information 116 can include parameters of the environment where the storage space 112 is located, such as temperature, humidity, and the like.

[0034] At block 206, the warehouse robot can obtain item information associated with the item to be placed, the item information including a size of the item and an environmental requirement of the item for a storage space. For example, in the environment 100 as shown in FIG. 1, the robot 102 can obtain the size 108 and the environmental requirement 110 of the item 106 to be placed. The size 108 can be, for example, a three-dimensional size, a two-dimensional size, a volume, a shape, etc. The environmental requirement 110 can indicate a requirement of the item 106 for a storage environment, which in some embodiments can include a requirement of the item 106 for a temperature of the storage environment and a requirement for humidity.

[0035] At block 208, the warehouse robot can generate, based on the plurality of images, the plurality of environmental information, and the plurality of item information, a plurality of matching scores corresponding to the plurality of storage spaces using a machine learning based model. For example, in the environment 100 as shown in FIG. 1, the matching degree determination model 118 is a machine learning based model trained to determine a matching degree between an item and a storage space. The robot 102 can input the plurality of images 114 and the plurality of environmental information 116 of the plurality of storage spaces 112, and the size 108 and the environmental requirement 110 of the item 106 to the matching degree determination model 118. The matching degree determination model 118 can determine, based on these input data, a matching score 120 between the item 106 and each of the storage spaces 112, respectively, which can indicate a matching degree between the item 106 and the storage space 112, e.g., a larger matching score 120 can represent a higher matching degree between the item 106 and the corresponding storage space 112.

[0036] At block 210, the warehouse robot can determine a target storage space from the plurality of storage spaces based on the plurality of matching scores. For example, in the environment 100 as shown in FIG. 1, the robot 102 can determine, based on the plurality of matching scores 120, a target storage space 122 for placing the item 106 from the plurality of storage spaces 112. For example, the robot 102 can determine the storage space with the largest matching score as the target storage space 122, or the robot 102 can obtain a predetermined matching score threshold and determine the target storage space 122 from one or more storage spaces satisfying the matching score threshold according to a predetermined rule.

[0037] At block 212, the warehouse robot can place the item to the target storage space. For example, in the environment 100 as shown in FIG. 1, after determining the target storage space 122, the robot 102 can carry the item 106 and move to a location where the target storage space 122 is located. Then, the robot 102 can place the item 106 into the target storage space 122 using a grasping component such as a robotic arm, a gripper, or a fork, etc.

[0038] In this way, the warehouse robot can autonomously select a target storage space from multiple storage spaces that meets the storage requirement of the item, thereby flexibly handling various abnormal situations, improving the operation efficiency of the system, and enhancing the robustness of the system. In addition, compared to selecting a storage space based on manually calibrated rules, the target storage space selected by the model based on machine learning can more accurately meet the storage requirement of the item, thereby reducing damage to the item due to the mismatch between the environment of the storage space and the storage requirement of the item.

[0039] In some embodiments, when generating the plurality of matching degree scores corresponding to the plurality of storage spaces using the matching degree determination model, the matching degree determination model can obtain a first image and first environment information corresponding to a first storage space in the plurality of storage spaces. Then, the matching degree determination model can generate a first matching degree score for the first storage space based on the first image, the first environment information, and the item information.

[0040] In some embodiments, the matching degree model can include a first deep neural network. The matching degree determination model can generate a fusion vector based on the first image, the first environment information, and the item information. Then, the matching degree determination model can generate the first matching degree score using the first deep neural network based on the fusion vector.

[0041] In some embodiments, the matching degree determination model further includes a convolutional neural network, a second deep neural network, and a third deep neural network. The matching degree determination model can generate an image vector using the convolutional neural network based on the first image. The matching degree determination model can also generate an environment vector using the second deep neural network based on the first environment information. The matching degree determination model can also generate an item vector using the third deep neural network based on the item information. Then, the matching degree determination model can generate a fusion vector by fusing the image vector, the environment vector, and the item vector.

[0042] FIG. 3 shows a schematic diagram of an example matching degree determination model 300 according to some embodiments of the present disclosure. The matching degree determination model 300 may, for example, be the matching degree determination model 118 in FIG. 1. As shown in FIG. 3, the matching degree determination model 300 includes a convolutional neural network 302, a deep neural network 304, a deep neural network 306, a fusion module 308, and a deep neural network 310. The matching degree determination model 300 can receive as input an image 312 of a storage space 311, environment information 314 of the storage space 311, and item information 316, and output a matching degree score 330.

[0043] In the matching degree determination model 300, the convolutional neural network 302 can generate an image vector 322 based on the image 312. The convolutional neural network 302 is capable of extracting multi-level features from the image 312. For the image 312 of the storage space 311, the convolutional neural network 302 can identify visual features such as spatial layout, spatial size, occupancy, and the like. In addition, the convolutional neural network 302 can capture local features in the image 312 through local receptive fields and convolution kernels, and the receptive field of the convolutional neural network 302 can gradually expand as the network layer increases, thereby being able to capture global information in the image 312. This is beneficial to retaining information in the storage space image, because both local and global information can have an impact on the matching degree. In this way, the hierarchical structure of the convolutional neural network 302 can extract visual features in the image 312 layer by layer, providing rich feature representations for subsequent fusion and decision-making.

[0044] In addition, the deep neural network 304 can generate an environment vector 324 based on the environment information 314, and the deep neural network 306 can generate an item vector 326 based on the item information 316. The deep neural networks 304 and 306 are good at processing numerical and categorical data. For example, for the environment information 314, the deep neural network 304 can perform multi-layer nonlinear mapping on the input data such as temperature, humidity, and the like to extract potential environmental features, thereby generating a low-dimensional environment vector 324, which can effectively capture the complex relationships between the data in the environment information 314. For the item information 316, the deep neural network 306 can effectively extract features from the data such as the size, temperature requirement, and humidity requirement of the item, thereby generating a comprehensive representation, i.e., the item vector 326. In this way, the matching degree determination model 300 can extract more complex high-order features from a variety of simple input data.

[0045] After generating the image vector 322, the environment vector 324, and the item vector 326, the matching degree determination model 300 can utilize the fusion module 308 to fuse these vectors together to generate a fusion vector 328. For example, in the fusion module 308, the image vector 322, the environment vector 324, and the item vector 326 can be spliced together to generate the fusion vector 328. In this way, the fusion vector 328 can integrate different types of features together, which helps the matching degree determination model 300 to capture the dependency between the image 312, the environment information 314, and the item information 316. In this way, by representing the mutual relationship between various features in a unified manner, the matching degree determination model 300 can better determine whether the storage space is suitable for placing the item.

[0046] After generating the fusion vector 328, the matching degree determination model 300 can generate a matching degree score 330 between the item and the storage space 311 based on the fusion vector 328 using the deep neural network 310. The deep neural network 310 is good at processing nonlinear data relationships. In determining the matching degree score, there can be complex nonlinear relationships between the conditions of the storage space (e.g., temperature, humidity, available space size, etc.) and the requirements of the item (e.g., size, temperature requirement, humidity requirement, etc.). For example, the requirements of certain items for the storage space can not be a linear relationship (e.g., the combined effect of temperature and humidity on the item). In addition, the interaction between the image vector, the environment vector, and the item vector is not simply linearly added. The deep neural network 310 can capture these complex nonlinear relationships through a multi-layer structure and a nonlinear activation function (e.g., ReLU, sigmoid, etc.), so that the matching degree determination model 300 can better learn the matching relationship between the storage space and the item, and improve the accuracy of the predicted matching degree score 330.

[0047] In some embodiments, the fusion module can include an attention network, and the fusion module can generate a storage space vector using the attention network based on the image vector and the environment vector. Then, the fusion module can generate the fusion vector by concatenating the storage space vector and the item vector. FIG. 4 shows a schematic diagram of an example fusion module 400 for a matching degree determination model according to some embodiments of the present disclosure. The fusion module 400 may, for example, be the fusion module 308 in FIG. 3.

[0048] As shown in FIG. 4, the fusion module 400 includes an attention network 402, which can fuse an image vector 412 (e.g., the image vector 322 in FIG. 3) including image features of the storage space with an environment vector 414 (e.g., the environment vector 324 in FIG. 3) including environment features of the storage space to generate a storage space vector 404. The storage space vector 404 can integrate the features associated with the storage space together, so as to comprehensively represent the feature information of the storage space.

[0049] In some embodiments, the attention network 402 can be a multi-head self-attention network, which can concatenate the image vector 412 and the environment vector 414 into a sequence, and then use a multi-head self-attention layer to process the concatenated sequence. The multi-head self-attention network can average each position in the sequence output from the multi-head self-attention layer to generate the fused storage space vector 404. In this way, the multi-head attention mechanism can understand the relationship between the image and environment features from different perspectives. In addition, the fused storage space vector 404 contains more information, so as to improve the accuracy of matching degree prediction.

[0050] In some embodiments, the attention network 402 can be a cross-attention network, which is capable of capturing complex interactions between the image vector 412 and the environment vector 414. For example, the attention network 402 can take the image vector 412 as a query for extracting relevant information from the environment vector, and take the environment vector 414 as a key and value for providing reference information to help adjust the representation of the image vector 412. Then, a cross-attention layer can be used to compute a representation that fuses the image information and the environment information. The attention network 402 can directly use the output of the cross-attention layer as the storage space vector 404, or further fuse it with the original image vector 412 and the environment vector 414 to generate the storage space vector 404. In this way, the image vector 412 is able to focus on relevant information from the environment vector 414, so that the model is able to better understand how the environmental features of the storage space affect the placement of the item.

[0051] As shown in FIG. 4, after generating the storage space vector 404, the fusion module 400 can concatenate the storage space vector 404 with the item vector 416 (e.g., the item vector 326 in FIG. 3) to generate a fusion vector 428. The fusion vector 428 is able to effectively fuse the feature information of the storage space and the feature information of the item, so as to improve the accuracy of the predicted matching degree between the storage space and the item.

[0052] In some embodiments, in the training phase of the matching degree determination model, a second image and a second environment information corresponding to a second storage space, and a second item information are obtained. Then, a real matching degree score of the second storage space and the second item information can be obtained. Then, based on the second image, the second environment information, and the second item information, a second matching degree score can be generated by using the machine learning-based model. Then, the machine learning-based model can be trained based on the second matching degree score and the real matching degree score. In some embodiments, a mean square error loss can be calculated based on the second matching degree score and the real matching degree score. Then, the machine learning-based model can be trained by minimizing the mean square error loss

[0053] For example, in the preparation phase of the training data, images and environment information of real storage spaces in a warehouse environment, and item information can be obtained. In addition, known real matching degree scores can also be obtained as labels. Then, the image data can be scaled, cropped, and standardized to ensure that the image sizes input to the model are consistent. In addition, the environment data (e.g., temperature, humidity, etc.) and the item information (e.g., size, temperature requirement, humidity requirement, etc.) can also be standardized to ensure that the numerical values are within a unified range (e.g., standardized to a value between 0 and 1).

[0054] When training the model, the training data can be used to predict the matching degree scores, and a mean squared error loss can be calculated based on the predicted matching degree scores and the pre-collected true matching degree scores. Then, the mean squared error loss can be minimized by adjusting the parameters of the model. For image data, data augmentation (e.g., rotation, translation, scaling, etc.) can be used to improve the generalization capability of the model. In addition, regularization can also be introduced in the model to prevent overfitting. After the mean squared error loss is less than a predetermined threshold, the training can be ended, and the trained matching degree determination model can be deployed into the warehouse robot to predict the matching degree scores of the items and the storage spaces in real time.

[0055] In some embodiments, the warehouse environment can include a lot of storage spaces, and in order to filter out a plurality of storage spaces that allow placing a target item from the storage spaces, the warehouse robot can acquire an item image of the item through a camera. Then, the warehouse robot can determine a type of the item based on the item image by using a first vision model. Then, the warehouse robot can determine a plurality of storage spaces corresponding to the item based on the type of the item.

[0056] FIG. 5 illustrates a schematic diagram of determining a plurality of storage spaces according to some embodiments of the present disclosure. As shown in FIG. 5, the example 500 includes a robot 502, and the robot 502 is configured with a camera 504. The robot 502 needs to determine a plurality of candidate storage spaces that allow placing an item 506, so as to further determine a target storage space that places the item 506 from the candidate storage spaces. In the example 500, the robot 502 can acquire an image 508 of the item 506 through the camera 504. Then, the robot 502 can identify an item type 512 of the item 506 from the image 508 by using a vision model 510. The vision model 510, for example, can be a trained convolutional neural network trained to output a type of an item in an image based on the image. The item type 512, for example, can be a fruit, an electronic product, etc. Then, the robot 502 can determine a plurality of storage spaces 514-1, 514-2, …, and 514-N (collectively referred to as storage spaces 514 herein) that allow placing the item 506 based on the item type 512. For example, the robot 502 can pre-store a mapping table of item types and storage spaces, and determine a plurality of storage spaces 514 corresponding to the item type 512 by looking up the mapping table. The determined plurality of storage spaces 514, for example, can be the plurality of storage spaces 112 in FIG. 1.

[0057] Compared with the traditional way of scanning barcodes, the warehouse robot can determine the type of the item to be placed by using the camera equipped on itself and the vision model. In this way, even if the item does not have a barcode or the barcode is blocked or damaged, the vision system can still continue to work, thereby improving the robustness of the system. In addition, the vision model can automatically handle items in different positions and different postures, thereby improving the flexibility and stability of the system.

[0058] In some embodiments, after determining the target storage space to place the item, the placement position in the storage space can be further determined. In some embodiments, the warehouse robot can acquire an additional image corresponding to the target storage space by the camera. Then, the warehouse robot can determine the position in the target storage space for placing the item by using a second vision model based on the additional image and the size of the item. Then, the warehouse robot can place the item to the position in the target storage space.

[0059] FIG. 6 shows a schematic diagram of an example 600 of determining the placement position in the target storage space, according to some embodiments of the present disclosure. As shown in FIG. 6, the example 600 includes a robot 602, and the robot 602 is configured with a camera 604. The robot 602 needs to determine the placement position 616 in the target storage space 610 to place an item 606. In the example 600, the robot 602 can acquire the size 608 of the item 606. In addition, the robot 602 can also acquire an image 612 of the target storage space 610 by the camera 604. Then, the robot 602 can input the size 608 of the item 606 and the image 612 of the target storage space 610 into a vision model 614. The vision model 614, for example, can be a trained convolutional neural network, trained to output the position in the image where the item can be placed based on the image of the storage space and the size of the item. In the example 600, the vision model 614 can generate a suitable placement position 616 based on the size 608 and the image 612.

[0060] Compared with the way of arranging the placement of items by relying on rules in the traditional scheme, the vision model can analyze the size, shape and available space of each position, combined with the size of the item, to determine the optimal placement position, thereby improving the utilization rate of the storage space. In addition, using the vision model to determine the placement position of the item can adapt to the dynamic changes of the storage space, and evaluate the availability of the space in real time, thereby improving the success rate of item placement.

[0061] FIG. 7 illustrates a block diagram of an apparatus 700 for placing an item in a warehousing environment, according to some embodiments of the present disclosure. As shown in FIG. 7, the apparatus 700 includes a storage space image acquisition module 702 configured to acquire, by a warehousing robot, a plurality of images of a plurality of storage spaces in a warehousing environment via a camera. The apparatus 700 further includes an environment information acquisition module 704 configured to acquire, by the warehousing robot, a plurality of environment information of the plurality of storage spaces. The apparatus 700 further includes an item information acquisition module 706 configured to acquire, by the warehousing robot, item information associated with an item to be placed, the item information including a size of the item and an environmental requirement for the storage space. The apparatus 700 further includes a matching degree score generation module 708 configured to generate, by the warehousing robot, a plurality of matching degree scores corresponding to the plurality of storage spaces based on the plurality of images, the plurality of environment information, and the item information, utilizing a machine learning based model. The apparatus 700 further includes a target storage space determination module 710 configured to determine, by the warehousing robot, a target storage space from the plurality of storage spaces based on the plurality of matching degree scores. In addition, the apparatus 700 further includes an item placement module 712 configured to place, by the warehousing robot, the item to the target storage space.

[0062] In some embodiments, wherein the environment information in the plurality of environment information includes a humidity and a temperature of the storage space, and the environmental requirement in the item information includes a requirement for the humidity and a requirement for the temperature.

[0063] In some embodiments, wherein the matching degree score generation module 708 includes a first environment information acquisition module configured to acquire a first image and a first environment information corresponding to a first storage space in the plurality of storage spaces, and a first matching degree score generation module configured to generate, based on the first image, the first environment information, and the item information, a first matching degree score for the first storage space, utilizing the machine learning based model.

[0064] In some embodiments, wherein the machine learning based model includes a first deep neural network, and the first matching degree score generation module includes a fusion vector generation module configured to generate a fusion vector based on the first image, the first environment information, and the item information, and a fusion vector usage module configured to generate the first matching degree score based on the fusion vector, utilizing the first deep neural network.

[0065] In some embodiments, the machine learning based model further comprises a convolutional neural network, a second deep neural network, and a third deep neural network, and the fusion vector generation module comprises: an image vector generation module configured to generate an image vector based on the first image using the convolutional neural network; an environment vector generation module configured to generate an environment vector based on the first environment information using the second deep neural network; an item vector generation module configured to generate an item vector based on the item information using the third deep neural network; and a vector fusion module configured to generate the fusion vector by fusing the image vector, the environment vector, and the item vector.

[0066] In some embodiments, the machine learning based model further comprises an attention network, and the vector fusion module comprises: a storage space vector generation module configured to generate a storage space vector based on the image vector and the environment vector using the attention network; and a storage space vector usage module configured to generate the fusion vector by concatenating the storage space vector and the item vector.

[0067] In some embodiments, the item information is first item information, and the apparatus 700 further comprises: a training data acquisition module configured to acquire a second image and a second environment information corresponding to a second storage space, and second item information; a real data acquisition module configured to acquire a real matching degree score of the second storage space and the second item information; a training result generation module configured to generate a second matching degree score based on the second image, the second environment information, and the second item information using the machine learning based model; and a real data usage module configured to train the machine learning based model based on the second matching degree score and the real matching degree score.

[0068] In some embodiments, the real data usage module comprises: a loss determination module configured to calculate a mean square error loss based on the second matching degree score and the real matching degree score; and a loss usage module configured to train the machine learning based model by minimizing the mean square error loss.

[0069] In some embodiments, the apparatus 700 further comprises: an item image acquisition module configured to acquire, by the warehouse robot, an item image of the item through the camera; an item type determination module configured to determine, by the warehouse robot, a type of the item based on the item image using a first visual model; and an item type usage module configured to determine, by the warehouse robot, the plurality of storage spaces corresponding to the item based on the type of the item.

[0070] In some embodiments, the item placement module 712 comprises: an additional image acquisition module configured to acquire, by the camera, an additional image corresponding to the target storage space; an additional image utilization module configured to determine, based on the additional image and the size of the item, a position in the target storage space for placing the item by using a second visual model; and a placement position utilization module configured to place the item to the position in the target storage space.

[0071] It can be understood that, by using the device 700 of the present disclosure, at least one of the many advantages that can be achieved by the method or process as described above can be achieved. For example, the warehouse robot can autonomously select a target storage space from multiple storage spaces that meets the storage requirements of the item, thereby being able to flexibly handle various abnormal situations, improving the operating efficiency of the system, and enhancing the robustness of the system. In addition, compared to selecting a storage space based on manually calibrated rules, the target storage space selected by using a model based on machine learning can more accurately meet the storage requirements of the item, thereby being able to reduce damage to the item due to the environment of the storage space not matching the storage requirements of the item.

[0072] FIG. 8 shows a block diagram of a device 800 that can implement embodiments of the present disclosure. The device 800 may, for example, be a processing unit of the robot 102 as shown in FIG. 1. As shown in FIG. 8, the device 800 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 802 or loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for operation of the device 800 can also be stored. The CPU / GPU 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804. Although not shown in FIG. 8, the device 800 can also include a coprocessor.

[0073] Various components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunications networks.

[0074] The various methods or processes described above can be performed by the CPU / GPU 801. For example, in some embodiments, a method can be implemented as a computer software program tangibly embodied in a machine readable medium, such as the storage 808. In some embodiments, portions of the computer program or all of the computer program can be loaded onto the device 800 via the ROM 802 and / or the communications unit 809. When a computer program is loaded onto the RAM 803 and executed by the CPU / GPU 801, one or more steps or actions of the methods or processes described above can be performed.

[0075] In some embodiments, the methods and processes described above can be tied to a computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions tangibly embodied therein.

[0076] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a

[0077] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0078] Computer readable program instructions for carrying out operations of the present disclosure can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including object oriented programming languages and conventional procedural programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0079] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function / act specified in the flowchart and / or block diagram block or blocks.

[0080] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0081] The computer program product of the second aspect can include a computer readable storage medium. The computer readable storage medium can include instructions. The instructions can include one or both of: instructions for causing a computer to enable a user equipment device to receive a configuration message from a base station, the configuration message comprising an indication of a set of one or more parameters for a first type of hybrid automatic repeat request process, the first type of hybrid automatic repeat request process being associated with a first type of data; and instructions for causing a computer to enable a user equipment device to receive a configuration message from a base station, the configuration message comprising an indication of a set of one or more parameters for a first type of hybrid automatic repeat request process, the first type of hybrid automatic repeat request process being associated with a first type of data.

[0082] Embodiments of the present disclosure have been described above, with the understanding that these embodiments are exemplary only, and are not restrictive, and are not limited to the disclosed embodiments. Many modifications and changes to the described embodiments are possible, without departing from the scope and spirit of the described embodiments. The selection of terms to be used herein is intended to best explain the principles of the embodiments, practical application, or technical improvement over the technology in the market, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

A method for placing an item in a warehouse environment, comprising: acquiring, by a warehouse robot, a plurality of images of a plurality of storage spaces in the warehouse environment via a camera; acquiring, by the warehouse robot, a plurality of environment information of the plurality of storage spaces; acquiring, by the warehouse robot, item information associated with the item to be placed, the item information comprising a size of the item and an environmental requirement for a storage space; generating, by the warehouse robot, a plurality of matching score corresponding to the plurality of storage spaces based on the plurality of images, the plurality of environment information, and the item information using a machine learning based model; determining, by the warehouse robot, a target storage space from the plurality of storage spaces based on the plurality of matching score; and placing, by the warehouse robot, the item to the target storage space. The method of claim 1, wherein an environment information in the plurality of environment information comprises a humidity and a temperature of a storage space, and the environmental requirement in the item information comprises a requirement for humidity and a requirement for temperature. The method of claim 1, wherein generating, by the warehouse robot, the plurality of matching score corresponding to the plurality of storage spaces based on the plurality of images, the plurality of environment information, and the item information using the machine learning based model comprises: acquiring a first image and a first environment information corresponding to a first storage space in the plurality of storage spaces; and generating a first matching score for the first storage space based on the first image, the first environment information, and the item information using the machine learning based model. The method of claim 3, wherein the machine learning based model comprises a first deep neural network, and generating the first matching score for the first storage space based on the first image, the first environment information, and the item information using the machine learning based model comprises: generating a fusion vector based on the first image, the first environment information, and the item information; and generating the first matching score based on the fusion vector using the first deep neural network. The method of claim 4, wherein the machine learning based model further comprises a convolutional neural network, a second deep neural network, and a third deep neural network, and generating the fusion vector based on the first image, the first environment information, and the item information comprises: generating an image vector based on the first image using the convolutional neural network; generating an environment vector based on the first environment information using the second deep neural network; generating an item vector based on the item information using the third deep neural network; and generating the fusion vector by fusing the image vector, the environment vector, and the item vector. The method of claim 5, wherein the machine learning based model further comprises a fourth deep neural network, and generating the first matching score based on the fusion vector using the first deep neural network comprises: generating a first matching score based on the fusion vector using the fourth deep neural network. ​ ​ The method of claim 5, wherein the machine learning based model further comprises an attention network, and generating the fused vector by fusing the image vector, the environment vector, and the item vector comprises: generating, with the attention network, a storage space vector based on the image vector and the environment vector; and generating the fused vector by concatenating the storage space vector and the item vector. The method of claim 1, wherein the item information is first item information, and the method further comprises: obtaining second image and second environment information, and second item information corresponding to a second storage space; obtaining a real match score of the second storage space and the second item information; generating, with the machine learning based model, a second match score based on the second image, the second environment information, and the second item information; and training the machine learning based model based on the second match score and the real match score. The method of claim 7, wherein training the machine learning based model based on the second match score and the real match score comprises: calculating a mean square error loss based on the second match score and the real match score; and training the machine learning based model by minimizing the mean square error loss. The method of claim 1, further comprising: obtaining, by the warehouse robot, an item image of the item by the camera; determining, by the warehouse robot, a type of the item based on the item image with a first vision model; and determining, by the warehouse robot, the plurality of storage spaces corresponding to the item based on the type of the item. The method of claim 1, wherein placing, by the warehouse robot, the item to the target storage space comprises: obtaining an additional image corresponding to the target storage space by the camera; determining, based on the additional image and the size of the item, a position in the target storage space for placing the item with a second vision model; and placing the item to the position in the target storage space. An apparatus for placing an item in a warehouse environment, comprising: a storage space image obtaining module configured to obtain, by a warehouse robot, a plurality of images of a plurality of storage spaces in the warehouse environment by a camera; an environment information obtaining module configured to obtain, by the warehouse robot, a plurality of environment information of the plurality of storage spaces; an item information obtaining module configured to obtain, by the warehouse robot, item information associated with the item to be placed, the item information comprising a size of the item and an environment requirement of a storage space; a match score generating module configured to generate, by the warehouse robot, a plurality of match scores corresponding to the plurality of storage spaces based on the plurality of images, the plurality of environment information, and the item information with a machine learning based model; a target storage space determination module configured to determine, by the warehouse robot, a target storage space from the plurality of storage spaces based on the plurality of matching degree scores; and an item placement module configured to place, by the warehouse robot, the item to the target storage space. An electronic device comprising: a processor; and a memory coupled with the processor, the memory having stored therein instructions that, when executed by the processor, cause the electronic device to perform the method according to any one of claims 1-10. A computer program product tangibly stored on a non-transitory computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to implement the method according to any one of claims 1-10. ​

Images