An article detection method, device, electronic device, and storage medium

Abstract

The application discloses an object detection method and device, electronic equipment and storage medium. The method is applied to a delivery robot, the delivery robot comprises detachable storage layers, each storage layer is associated with a camera; the method comprises the following steps: obtaining storage layer change information of the delivery robot; according to the storage layer change information, the associated cameras of the current remaining storage layers of the delivery robot are determined again, and the detection distances of the associated cameras are determined; according to the detection distances of the associated cameras, the storage layer images shot by the associated cameras are adjusted; and based on the adjusted storage layer images, the objects in the storage layers are detected. The detachable storage layers are arranged on the robot, the problem that the robot cannot deliver objects larger than the height of the storage layers is solved by detaching the storage layers; after the storage layers are adjusted, the latest detection distances of the cameras are determined, the shot images are adjusted according to the detection distances to clearly define the object features, and the accuracy of object detection based on the adjusted images is ensured.

Description

Technical Field

[0001] This invention relates to the field of robotics, and more particularly to an article detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of intelligent technology, the service industry, especially complex service industries such as hotels and restaurants, now uses robots to deliver goods. Delivery robots can deliver food and other items to customers according to a set route.

[0003] Currently, existing delivery robots do not support the removal of the storage layer, which prevents them from delivering items taller than the storage layer, thus limiting the items they can deliver. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for detecting articles.

[0005] According to one aspect of the present invention, an item detection method is provided, applied to a delivery robot, the delivery robot including multiple storage layers including at least one detachable storage layer, each storage layer being associated with a camera for acquiring images of the storage layer; the method includes:

[0006] Obtain information on changes in the storage layer of the delivery robot;

[0007] Based on the information about changes in the storage layer, the associated cameras for each of the remaining storage layers of the delivery robot, as well as the detection distance of the associated cameras, are re-determined.

[0008] Adjust the image of the storage layer captured by the associated camera based on the detection distance of the associated camera;

[0009] Based on the adjusted image of the storage layer, the items in the storage layer are detected.

[0010] According to another aspect of the present invention, an item detection device is provided, configured in a delivery robot, the delivery robot including multiple storage layers including at least one detachable storage layer, each storage layer being associated with a camera for acquiring images of the storage layer; the device includes:

[0011] The storage layer change information acquisition module is used to acquire storage layer change information of the delivery robot;

[0012] The association determination module is used to redetermine the associated cameras of each of the remaining shelves of the delivery robot and the detection distance of the associated cameras based on the shelf change information.

[0013] An image adjustment module is used to adjust the image of the storage layer captured by the associated camera according to the detection distance of the associated camera;

[0014] The detection module is used to detect items in the shelf layer based on the adjusted shelf layer image.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the article detection method according to the embodiments of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the article detection method described in the embodiments of the present invention.

[0020] The technical solution of this invention provides a detachable storage layer on the robot. By adjusting the storage layer, the problem that the robot cannot deliver items that are taller than the storage layer is solved. Moreover, after adjusting the storage layer, the latest detection distance of the camera is determined in a timely manner, and the captured image is adjusted according to the detection distance to clarify the object features, thereby ensuring the accuracy of item detection based on the adjusted image.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1a This is a schematic diagram of the hardware structure of a delivery robot according to an embodiment of the present invention;

[0024] Figure 1b This is a schematic diagram of the interactive interface of a delivery robot provided according to an embodiment of the present invention;

[0025] Figure 1c This is a flowchart illustrating an article detection method according to Embodiment 1 of the present invention;

[0026] Figure 2 This is a schematic flowchart of an article detection method according to Embodiment 2 of the present invention;

[0027] Figure 3 This is a flowchart illustrating an article detection method according to Embodiment 3 of the present invention;

[0028] Figure 4 This is a flowchart illustrating an article detection method according to Embodiment 4 of the present invention;

[0029] Figure 5 This is a flowchart illustrating an article detection method according to Embodiment 5 of the present invention;

[0030] Figure 6 This is a schematic diagram of the structure of an article detection device according to Embodiment Six of the present invention;

[0031] Figure 7 This is a schematic diagram of the structure of an electronic device that implements the article detection method of this invention. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0033] First, the hardware structure of the delivery robot in this invention will be described. The delivery robot of this invention includes at least a mobile chassis for realizing the robot's movement / turning / stopping. Above the mobile chassis, multiple storage layers, including at least one detachable storage layer, are arranged vertically. Each storage layer includes a tray for carrying items to be delivered. The tray is just an example; other components for carrying items can also be used, and no specific limitation is made here. Each storage layer is associated with a camera (e.g., an RGB camera) for capturing images of the storage layer. Optionally, the camera for capturing images of each storage layer is deployed directly above or to the side of the tray. In specific deployment, for any storage layer, the associated camera can be located at the bottom of a target storage layer adjacent to and above that storage layer.

[0034] In one alternative implementation, taking a delivery robot used in a restaurant setting as an example, its hardware structure diagram can be found in the appendix. Figure 1a In addition to the mobile chassis, the delivery robot includes four storage layers, labeled 1, 2, 3, and 4. A camera a is located on the upper side of the tray of storage layer 1 to capture images of the storage layer 1; storage layer 1 is associated with camera a. Camera a can be positioned below the top image module of the robot, meaning it is non-removable. This ensures that the robot always has at least one layer capable of item recognition, improving the user experience. A camera b is located directly above the tray of storage layer 2 to capture images of the storage layer 2; storage layer 2 is associated with camera b. A camera c is located directly above the tray of storage layer 3 to capture images of the storage layer 3; storage layer 3 is associated with camera c. The top shelf 1 is a non-removable shelf, and stereo vision sensors m are installed on both sides of the non-removable shelf 1 to collect point cloud data of the surrounding environment of the delivery robot, so that the delivery robot can move laterally and avoid obstacles based on the point cloud data of the surrounding environment; the shelf 2 and shelf 3 are removable shelves, so that by removing shelf 2 and / or shelf 3, the delivery robot can deliver items with a height greater than the height of the shelf.

[0035] In this invention, the shelf removal process can be as follows: First, before removing the shelf, the delivery robot is powered off. Then, the user selects the shelf to be removed based on the attributes (e.g., height) of the item to be delivered and removes it. Removing the shelf actually involves dismantling the tray and related components to create more space. After removal, the delivery robot is powered on. Once powered on, the user can input information such as the number of shelves removed and the number of remaining shelves through the interactive interface provided by the delivery robot's display device (e.g., a touchscreen control screen). Alternatively, the shelf can be removed while the robot is powered on. On the robot's settings page, the user can input information such as the number of shelves removed and the number of remaining shelves through the interactive interface provided by the delivery robot's display device (e.g., a touchscreen control screen).

[0036] For example, see Appendix Figure 1bThe diagram shows the interactive interface, which displays the number of storage shelves and provides two functional areas. The first functional area provides an automatic shelf identification function. If the user triggers the automatic shelf identification function, the delivery robot automatically detects which shelf has been removed and updates the shelf count. When the automatic identification function is enabled, if the robot detects a change in the shelf count, it determines whether it is in a task state, such as performing a food delivery task. If it is in a task state, it does not update the shelf count to avoid task execution errors. After the robot finishes its task, it updates the shelf count based on the latest detected shelf number.

[0037] The second functional area provides a function to manually select and remove storage layers. Optionally, the second functional area displays a simplified diagram of the delivery robot, and each removable storage layer of the delivery robot corresponds to a control for user selection. If the user does not trigger the automatic storage layer recognition function, the user can directly select the storage layer to be removed by touching the corresponding control of the removable storage layer. The delivery robot will then respond to the user's touch operation and update the number of storage layers. For example... Figure 1b Originally there were 4 storage shelves. After removing one shelf, the number of storage shelves became 3.

[0038] Based on the aforementioned delivery robot hardware, the specific process of the item detection method of the present invention can be found in the following embodiments.

[0039] Example 1

[0040] Figure 1c The flowchart of an item detection method is provided in Embodiment 1 of the present invention. This embodiment is applicable to scenarios in which robots deliver items, typically in restaurant scenarios where robots deliver food. The method can be executed by an item detection device, which can be implemented in hardware and / or software. The item detection device can be configured in an electronic device, such as integrated into a robot device.

[0041] like Figure 1c As shown, the method for detecting this item includes:

[0042] S101. Obtain information on changes in the storage layer of the delivery robot.

[0043] In this embodiment of the invention, the information regarding changes in the delivery robot's storage layer may optionally include information about the storage layer that has been removed from the delivery robot, such as the identification information of the removed storage layer and the information of the associated camera corresponding to the removed storage layer. In one optional implementation, if the user triggers the automatic storage layer recognition function on the interactive interface, the information regarding changes in the delivery robot's storage layer is determined based on the automatic storage layer recognition result; if the user does not trigger the automatic storage layer recognition function on the interactive interface but directly selects the removed storage layer, the information about the removed storage layer is determined based on the user's triggering operation on the control corresponding to the detachable storage layer. For example, the automatic recognition function can be implemented by setting a detection sensor at the connection between the robot body and the tray. When the tray is not removed, the circuit is connected, indicating that the storage layer has a tray installed; when the tray is removed, the circuit is disconnected, indicating that the tray of the storage layer has been removed. For example, the positions of the trays on each shelf can be preset, and the size of each tray is a fixed value. The associated camera can capture images with the trays installed and with them removed, and mark the tray areas in each state. Then, the camera can determine which trays have been removed based on the size and position of the tray images in the captured images. Furthermore, since the camera is installed on the upper shelf, the automatic recognition function can determine the location based on whether there is an image signal input from each camera to the processor, combined with the detection sensors located at the connection between the camera body and the tray, thus improving accuracy.

[0044] S102. Based on the information about changes in the storage layer, redetermine the associated cameras for each of the remaining storage layers of the delivery robot, as well as the detection distance of the associated cameras.

[0045] In this embodiment of the invention, for any shelf, the associated camera of that shelf is located at the bottom of the target shelf that is adjacent to and above that shelf. For example, see... Figure 1a The associated camera b of shelf layer 2 is located at the bottom of shelf layer 1; the associated camera c of shelf layer 3 is located at the bottom of the removable shelf layer 2; where shelf layers 2 and 3 are removable shelves. If shelf layer 2 is removed, the associated camera c, located at the bottom of shelf layer 2 and used to capture images in shelf layer 3, is removed. At this time, associated camera b is used to capture images in shelf layer 3, increasing the detection distance of associated camera b; the detection distance refers to the distance from the associated camera to the bottom of the shelf layer it captures. Therefore, removing a shelf layer changes the association between some cameras and the shelf layer, and also changes the detection distance of some cameras. Thus, after determining that one or more shelf layers have been removed, it is necessary to re-determine the associated cameras of the remaining shelf layers of the delivery robot, as well as the detection distance of the associated cameras.

[0046] In one optional implementation, based on the shelf change information, the removed shelf and its associated camera are identified. Then, the associated camera and its detection distance for each of the remaining shelves can be determined through automatic shelf identification. Furthermore, since the number of removable shelves on the delivery robot is limited, statistical methods can be used to pre-calculate the associated camera and its detection distance for each of the remaining shelves after any shelf is removed, and this statistical result can be saved. Then, after determining that one or more shelves have been removed based on the shelf change information, the associated camera and its detection distance for each of the remaining shelves can be directly determined from the statistical result.

[0047] S103. Adjust the image of the storage layer captured by the associated camera according to the detection distance of the associated camera.

[0048] In this embodiment of the invention, disassembling the storage layer increases the detection distance of a certain associated camera. For example, if storage layer 2 is disassembled, the original detection distance of associated camera b is equal to the height of storage layer 2. After disassembling storage layer 2, the detection distance of associated camera b is equal to the height of storage layer 2 plus the height of storage layer 3. This increased detection distance of the associated camera leads to a smaller size of the objects in the images it captures. If the captured images are still used directly for object detection, the accuracy of object detection will decrease. Therefore, to ensure the accuracy of object detection, the images of the storage layer captured by the associated camera can be adjusted according to the detection distance of the associated camera. For example, the images of the storage layer can be enlarged to magnify the object features. Subsequently, the enlarged images can be used for object detection, improving the accuracy of object detection.

[0049] It should be noted that if the detection distance of the associated camera does not change, there is no need to adjust the image of the object layer it captures.

[0050] S104. Based on the adjusted image of the storage layer, detect the items in the storage layer.

[0051] In this embodiment of the invention, the delivery robot, having multiple storage layers, can simultaneously deliver different items to different users. Specifically, the delivery robot can plan delivery routes based on the different delivery destinations of the items, and then deliver the items sequentially according to the delivery routes; the name and destination of each delivered item can be displayed on the delivery robot's display device. Since the delivery robot may deliver items to different users simultaneously, in addition to detecting whether the user has completed picking up the item, it is also necessary to detect whether the user has picked up the item on the correct storage layer. Based on this, when the delivery robot reaches a target delivery point (i.e., the delivery destination), based on the adjusted storage layer image and combined with a pre-trained item detection model, it determines whether the item in the storage layer has been taken, that is, whether the captured image includes the item. If the image does not contain the item, it means that the item has been taken, and there is no need for manual confirmation of pickup. Moreover, before the robot continues to deliver other items, it can also determine whether there is a case of incorrect item pickup based on the current location of the delivery robot and the target delivery point of the delivered item in the storage layer; if an incorrect item is picked up, an error reminder is given, such as a voice prompt to the customer that the wrong item has been picked up.

[0052] The technical solution of this invention involves setting a detachable shelf on the robot. By removing the shelf, the problem of the robot being unable to deliver items taller than the shelf is solved, allowing the delivery of taller objects such as cakes and beer. Furthermore, after adjusting the shelf, the system promptly determines the associated camera and its latest detection distance for the remaining shelf, and adjusts the captured image based on the detection distance to magnify object features, thereby ensuring the accuracy of item detection based on the adjusted image. At this point, the plate detection model can still identify items with high precision, supporting customer self-service pickup without requiring confirmation, improving the user experience and providing merchants with diverse delivery options.

[0053] Example 2

[0054] Figure 2 This is a flowchart illustrating a method for detecting an item according to Embodiment 2 of the present invention. This embodiment is an optimization based on the above embodiment; see below. Figure 2 The method includes the following steps:

[0055] S201. Obtain information on changes in the storage layer of the delivery robot.

[0056] S202. Based on the information about changes in the storage layer and in conjunction with the preset mapping relationship, redetermine the associated cameras and detection distances of the remaining storage layers of the delivery robot.

[0057] In this embodiment, since the number of detachable storage layers on the delivery robot is limited, a mapping relationship can be pre-constructed based on manual statistics. This mapping relationship can include the detection distance between each associated camera and each storage layer; for example, for... Figure 1a The system associates camera b with storage layer 1 and stores the detection distances of camera b with storage layers 2, 3, and 4 through a mapping relationship. This mapping relationship can also include the association between the remaining storage layers and the remaining cameras after any storage layer is removed. For example, after removing storage layer 2, storage layer 1 is associated with camera a, and storage layer 3 is associated with camera b; after removing storage layer 3, storage layer 1 is associated with camera a, storage layer 2 with camera b, and storage layer 4 with camera c; and after simultaneously removing both storage layers 2 and 3, storage layer 1 is associated with camera a, and storage layer 4 with camera b. In some embodiments, considering the lower frequency of use of the bottom layer, a camera may not be installed at the bottom of the tray of storage layer 3 to save costs. That is, when the tray of storage layer 3 is not removed, storage layer 4 does not have an item detection function, but when the tray of storage layer 3 is removed, storage layer 4 can be equipped with an item detection function.

[0058] Based on the pre-determined mapping relationship, as long as the dismantled storage layer is determined according to the storage layer change information, the associated cameras of the remaining storage layers and the detection distance of the associated cameras can be determined from the mapping relationship.

[0059] S203. Adjust the image of the storage layer captured by the associated camera according to the detection distance of the associated camera.

[0060] The information about the installation and removal of the display layers is given to the app, which then feeds it to the algorithm to determine whether image processing is needed. For example, if layer 2 is removed, leaving only [a, b] for the camera and [1, 3, 4] for the display layers, the algorithm knows that the second layer has been removed. Camera a captures the image of layer 1, and camera b captures the image of layer 3. In this case, the algorithm knows to process the image by performing a resize operation. Conversely, if layer 3 is removed, but the camera remains [a, b, c], and the display layers become [1, 2, 4], camera a captures the image of layer 1, camera b captures the image of layer 2, and camera c captures the image of layer 4. In this case, the algorithm knows to process the image by performing a resize operation.

[0061] S204. Based on the adjusted image of the storage layer, detect the items in the storage layer.

[0062] In this embodiment, by pre-setting a mapping relationship, the associated cameras and detection distances of the remaining shelves can be directly determined based on the mapping relationship and the shelf change information. This eliminates the need for further analysis by the delivery robot, saving computing power and improving efficiency.

[0063] Example 3

[0064] Figure 3 This is a flowchart of an article detection method provided in Embodiment 3 of the present invention. See also... Figure 3 The method includes the following steps:

[0065] S301. Obtain information on changes in the storage layer of the delivery robot.

[0066] In this embodiment of the invention, if the mapping relationship involved in the above embodiments is not pre-built, the automatic identification function of the storage layer of the delivery robot can be used to redetermine the associated cameras of the remaining storage layers of the delivery robot and the detection distance of the associated cameras according to the storage layer change information. For the specific implementation process, please refer to steps S302-S305.

[0067] S302. Based on the information about changes in the storage layer, determine the first target storage layer that is currently being dismantled.

[0068] In this embodiment, the information regarding changes in the delivery robot's storage layers can optionally include information about the storage layers that have been removed, such as the identification information of the removed storage layers and the information of the associated cameras of the removed storage layers. This allows for the direct determination of the first target storage layer to be removed based on the identification information of the removed storage layers. It should be noted that since there are multiple removable storage layers on the delivery robot, the removed storage layers may be one or more. Therefore, it is also necessary to determine the first target storage layer based on the number of removed storage layers to ensure the accuracy of subsequent identification. Optionally, if the number of disassembled shelves is determined to be one based on the identification information of the disassembled shelves in the shelf change information, then the disassembled shelf is taken as the first target shelf; if the number of disassembled shelves is determined to be at least two based on the identification information of the disassembled shelves in the shelf change information, then the positional relationship of the disassembled shelves is determined, wherein the positional relationship can be the vertical distribution relationship of the shelves; based on the positional relationship, the shelf at the top position among the disassembled shelves is taken as the first target shelf; for example, if shelf 2 and shelf 3 are disassembled at the same time, since shelf 2 is located above shelf 3, shelf 2 is taken as the first target shelf.

[0069] S303. Determine the second target storage layer that currently exists on the delivery robot, located below the first target storage layer and adjacent to the first target storage layer.

[0070] In this embodiment, the delivery robot can automatically identify the remaining storage layers and then, based on the vertical distribution of each storage layer, quickly determine the existing second target storage layer that is located below the first target storage layer and adjacent to the first target storage layer.

[0071] S304. Re-establish the association between the first associated camera corresponding to the first target placement layer and the second target placement layer.

[0072] Optionally, the association between the first target placement layer and the first associated camera is terminated; the association between the second associated camera and the second target placement layer is terminated; wherein the second associated camera is located at the bottom of the first target placement layer and is removed from the robot as the first target placement layer is disassembled; the association between the first associated camera corresponding to the first target placement layer and the second target placement layer is re-established, so that the first associated camera is used to acquire images of the second target placement layer.

[0073] S305. Based on the height attribute of the shelf and the number of shelves that have been removed, redetermine the detection distance of the first associated camera.

[0074] The height attribute can be selected as the layer height. In this embodiment, the layer height of each shelf on the delivery robot is the same, h. Based on this, the number of shelves that are removed is n. Then, the detection distance L of the first associated camera is L = h * (n + 1). It should be noted that if the layer heights of each shelf are different, the final detection distance is obtained by adding the layer height of the shelf initially associated with the first associated camera to the layer height of the removed shelf.

[0075] S306. Adjust the image of the storage layer captured by the first associated camera according to the detection distance of the first associated camera.

[0076] Optionally, an image adjustment coefficient is determined based on the redefined detection distance of the first associated camera; wherein, the image adjustment coefficient may be an image magnification factor; different detection distances correspond to different image magnification factors; and the image of the placement layer captured by the first associated camera is adjusted according to the image adjustment coefficient.

[0077] S307. Based on the adjusted image of the storage layer, detect the items in the storage layer.

[0078] This embodiment can automatically identify the associated cameras corresponding to the remaining storage layers and the detection distance of the associated cameras, thus ensuring accurate detection of items in the future.

[0079] Example 4

[0080] Figure 4 This is a flowchart of an article detection method provided in Embodiment 4 of the present invention. See also... Figure 4 The method includes the following steps:

[0081] S401. Obtain information on changes in the storage layer of the delivery robot.

[0082] S402. Based on the information about changes in the storage layer, redetermine the associated cameras for each of the remaining storage layers of the delivery robot, as well as the detection distance of the associated cameras.

[0083] S403. Determine the image adjustment coefficient based on the detection distance of the associated camera.

[0084] Among them, the image adjustment coefficient can be selected as the image magnification factor; different detection distances correspond to different image magnification factors.

[0085] S404. Adjust the image of the storage layer captured by the associated camera based on the image adjustment coefficient.

[0086] In one alternative implementation, the image of the object layer captured by the associated camera is magnified according to the magnification factor corresponding to the detection distance, so as to magnify the object features in the image and thus ensure the accuracy of object detection based on the magnified image.

[0087] In another optional implementation, the images of the storage layers captured by the associated camera are adjusted according to image adjustment coefficients. This includes: determining the pallet vertex coordinates of the corresponding storage layer from the images captured by the associated camera, and determining the pallet area based on the pallet vertex coordinates. Since the installation position, viewing angle, and layer height of each storage layer are fixed, the pallet vertex coordinates can be determined through pre-calibration. Determining the pallet vertex coordinates through pre-calibration avoids misjudgments caused by items on the pallet obscuring the pallet edges. The pallet vertex coordinates can also be obtained by the associated camera recognizing the pallet edge contour based on image recognition. Alternatively, an object recognition algorithm can be used to determine the pallet vertex coordinates and the pallet area; based on the pallet area, a pallet area image is cropped from the storage layer images captured by the associated camera to eliminate the influence of interfering objects in the image on item detection. These interfering objects are captured due to the increased camera detection distance; and the pallet area image is adjusted according to the image adjustment coefficients. It should be noted that when performing item detection, the input image needs to be fed into a pre-trained item detection model for detection. Therefore, it is only necessary to enlarge the size of the tray area image to match the size of the sample images used for model training.

[0088] First, redefine the tray area, then adjust the image of the tray area. This improves the accuracy of item detection and allows the adjusted image to be input into a pre-built item detection model. The model doesn't need to be adjusted based on different detection distances, increasing its versatility and reducing training difficulty. For example, if the tray area image size is 800*600 during model training, the adjusted image will also be adjusted to 800*600 to maintain consistency.

[0089] S405. Based on the adjusted image of the storage layer, detect the items in the storage layer.

[0090] In this embodiment, the image of the shelf taken by the associated camera is magnified, which can magnify the features of the items. This ensures the accuracy of item recognition when the items are identified based on the magnified image.

[0091] Example 5

[0092] Figure 5 This is a flowchart illustrating an item detection method according to Embodiment 5 of the present invention. This embodiment is also applicable to situations where, after a robot that has disassembled its storage layer has delivered items, part or all of the disassembled storage layer is reinstalled onto the delivery robot. See [link to documentation]. Figure 5 The logic of this method includes the following:

[0093] S501. Obtain information on changes in the storage layer of the delivery robot.

[0094] The information on changes in storage layers may also include the identification information of newly installed storage layers; accordingly, based on the information on changes in storage layers, the cameras associated with the remaining storage layers of the delivery robot are re-determined, including steps S502-S504.

[0095] S502. If the number of newly installed shelves is determined to be one based on the identification information of the newly installed shelf, then the newly installed shelf shall be regarded as the third target shelf.

[0096] S503. Determine the fourth target storage layer that currently exists in the delivery robot, located below the third target storage layer and adjacent to the third target storage layer.

[0097] S504. Establish an association relationship between the third associated camera associated with the fourth target placement layer and the third target placement layer, so that the third associated camera can be used to acquire images of the third target placement layer.

[0098] S505. Establish an association between the fourth associated camera at the bottom of the third target placement layer and the fourth target placement layer, so that the fourth associated camera can be used to acquire images of the fourth target placement layer.

[0099] S506. Based on the images of the third and fourth storage layers respectively, identify the items in the third and fourth storage layers.

[0100] In this embodiment, when a new shelf is installed, the relationship between the shelves is adjusted to ensure that the camera can correctly capture images of the associated shelf, thereby ensuring the accuracy of item recognition for each shelf.

[0101] In this embodiment, if the number of newly installed shelves is at least two, the association between the shelves and the camera is re-established based on the existing shelves and the camera.

[0102] Example 6

[0103] Figure 6 This is a schematic diagram of an item detection device provided in Embodiment 4 of the present invention. This embodiment is applicable to scenarios where robots deliver items, typically in restaurant settings where robots deliver food. Figure 6 As shown, the device is configured on a delivery robot, which includes multiple storage layers, including at least one detachable storage layer, and each storage layer is associated with a camera for capturing images of the storage layer; the device specifically includes:

[0104] The storage layer change information acquisition module 601 is used to acquire the storage layer change information of the delivery robot;

[0105] The association determination module 602 is used to redetermine the associated cameras of each of the remaining shelves of the delivery robot and the detection distance of the associated cameras based on the shelf change information.

[0106] Image adjustment module 603 is used to adjust the image of the storage layer captured by the associated camera according to the detection distance of the associated camera;

[0107] The detection module 604 is used to detect items in the shelf layer based on the adjusted shelf layer image.

[0108] Based on the above embodiments, optionally, the association relationship determination module includes:

[0109] The first determining unit is used to redetermine the associated cameras of each of the remaining storage layers of the delivery robot and the detection distance of the associated cameras based on the information about the change in the storage layer and in conjunction with a preset mapping relationship.

[0110] Based on the above embodiments, optionally, the association determination module further includes:

[0111] The second determining unit is used to determine the first target storage layer that is currently being removed based on the storage layer change information;

[0112] The third determining unit is used to determine the second target storage layer that currently exists on the delivery robot, located below the first target storage layer and adjacent to the first target storage layer;

[0113] The association relationship determination unit is used to re-establish the association relationship between the first associated camera corresponding to the first target placement layer and the second target placement layer;

[0114] The detection distance determination unit is used to redetermine the detection distance of the first associated camera based on the height attribute of the shelf and the number of shelves that have been removed.

[0115] Optionally, based on the above embodiments, the second determining unit is further configured to:

[0116] If, based on the identification information of the disassembled shelf in the shelf change information, it is determined that the number of disassembled shelves is one, then the disassembled shelf is taken as the first target shelf.

[0117] If, based on the identification information of the disassembled shelves in the shelf change information, it is determined that the number of disassembled shelves is at least two, then the positional relationship of the disassembled shelves is determined; based on the positional relationship, the shelf at the top position among the disassembled shelves is taken as the first target shelf.

[0118] Based on the above embodiments, optionally, the image adjustment module includes:

[0119] The coefficient determination unit is used to determine the image adjustment coefficient based on the detection distance of the associated camera;

[0120] An adjustment unit is used to adjust the image of the placement layer captured by the associated camera according to the image adjustment coefficient.

[0121] Optionally, based on the above embodiments, the adjustment unit is further configured to:

[0122] From the images of the storage layer captured by the associated camera, determine the coordinates of the tray vertices of the corresponding associated storage layer, and determine the tray area based on the tray vertices.

[0123] Based on the tray area, the tray area image is cropped from the shelf layer image captured by the associated camera;

[0124] The tray area image is adjusted according to the image adjustment coefficient.

[0125] Based on the above embodiments, optionally, the top shelf of the delivery robot is a non-removable shelf; and the non-removable shelf is equipped with a stereo vision sensor to collect point cloud data of the surrounding environment of the delivery robot, so that the delivery robot can avoid obstacles based on the point cloud data of the surrounding environment.

[0126] The article detection device provided in the embodiments of the present invention can execute the article detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0127] Example 7

[0128] Figure 7 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0129] like Figure 7 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0130] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0131] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as performing an item detection method.

[0132] In some embodiments, the article detection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the article detection method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the article detection method by any other suitable means (e.g., by means of firmware).

[0133] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0134] Computer programs used to implement the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable article detection device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0135] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0136] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0137] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0138] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0139] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0140] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for detecting an article, characterized in that, The method is applied to a delivery robot, which includes multiple storage layers, including at least one detachable storage layer, with each storage layer associated with a camera for capturing images of that layer; the method includes: Obtain information on changes in the storage layer of the delivery robot; Based on the information about changes in the storage layer, the associated cameras for each of the remaining storage layers of the delivery robot, as well as the detection distance of the associated cameras, are re-determined. Adjust the image of the storage layer captured by the associated camera based on the detection distance of the associated camera; Based on the adjusted image of the storage layer, the items in the storage layer are detected.

2. The method according to claim 1, characterized in that, Based on the information regarding changes in the storage layer, the associated cameras for each of the remaining storage layers of the delivery robot, and the detection distance of the associated cameras, are re-determined, including: Based on the information about changes in the storage layer, and in conjunction with a preset mapping relationship, the associated cameras for each of the remaining storage layers of the delivery robot and the detection distance of the associated cameras are re-determined.

3. The method according to claim 1, characterized in that, Based on the information regarding changes in the storage layer, the associated cameras for each of the remaining storage layers of the delivery robot, and the detection distance of the associated cameras, are re-determined, including: Based on the information about changes in the storage layer, the first target storage layer to be removed is determined. Determine the second target shelf currently existing on the delivery robot, located below the first target shelf and adjacent to the first target shelf; Re-establish the association between the first associated camera corresponding to the first target placement layer and the second target placement layer; The detection distance of the first associated camera is re-determined based on the height attribute of the shelf and the number of shelves that have been removed.

4. The method according to claim 3, characterized in that, Based on the information regarding changes in the storage layer, the first target storage layer to be removed is determined, including: If, based on the identification information of the disassembled shelf in the shelf change information, it is determined that the number of disassembled shelves is one, then the disassembled shelf is taken as the first target shelf. If, based on the identification information of the disassembled shelves in the shelf change information, it is determined that the number of disassembled shelves is at least two, then the positional relationship of the disassembled shelves is determined; based on the positional relationship, the shelf at the top position among the disassembled shelves is taken as the first target shelf.

5. The method according to claim 1, characterized in that, Adjusting the image of the housing layer captured by the associated camera based on the detection distance of the associated camera includes: Based on the detection distance of the associated camera, determine the image adjustment coefficient; The image of the shelf layer captured by the associated camera is adjusted according to the image adjustment coefficient.

6. The method according to claim 5, characterized in that, Adjusting the image of the placement layer captured by the associated camera according to the image adjustment coefficient includes: From the images of the storage layer captured by the associated camera, determine the coordinates of the tray vertices of the corresponding associated storage layer, and determine the tray area based on the tray vertices. Based on the tray area, the tray area image is cropped from the shelf layer image captured by the associated camera; The tray area image is adjusted according to the image adjustment coefficient.

7. The method according to claim 1, characterized in that, The top shelf of the delivery robot is a non-removable shelf; and the non-removable shelf is equipped with a stereo vision sensor to collect point cloud data of the surrounding environment of the delivery robot, so that the delivery robot can avoid obstacles based on the point cloud data of the surrounding environment.

8. An item detection device, characterized in that, The device is configured for use with a delivery robot, the delivery robot including at least one detachable storage layer and multiple storage layers, each storage layer being associated with a camera for capturing images of the storage layer; the device includes: The storage layer change information acquisition module is used to acquire storage layer change information of the delivery robot; The association determination module is used to redetermine the associated cameras of each of the remaining shelves of the delivery robot and the detection distance of the associated cameras based on the shelf change information. An image adjustment module is used to adjust the image of the storage layer captured by the associated camera according to the detection distance of the associated camera; The detection module is used to detect items in the shelf layer based on the adjusted shelf layer image.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the method of any one of claims 1-7.

Images