A palletizing method, apparatus, electronic device and storage medium

Abstract

This application discloses a palletizing method, apparatus, electronic device, and storage medium, relating to the field of robotic depalletizing and palletizing. It uses a depth camera to calculate the height of goods, saving equipment costs and eliminating interruptions during the entire goods movement process, thus saving overall palletizing time. The method includes: acquiring a first depth image of the goods placement area captured by the depth camera; wherein the first depth image contains a first item; acquiring first area information based on the first depth image; controlling a moving device based on the first area information to move the first item out of the space containing the first area; after the moving device moves the first item out of the space containing the first area, acquiring a second depth image of the goods placement area captured by the depth camera; acquiring the height of the first item based on the first and second depth images; and controlling the moving device based on the height of the first item to palletize the first item.

Description

Technical Field

[0001] This application relates to the field of robotic depalletizing and palletizing, and more particularly to a palletizing method, apparatus, electronic device, and storage medium. Background Technology

[0002] In robotic mixed disassembly and packing applications, if the robot cannot determine the height of the boxes after picking them up and placing them, it will not know the required lowering height, which may lead to problems such as boxes being crushed or dropped during the packing process.

[0003] In existing technologies, one method for calculating the height of a box is to use two depth cameras: one to locate the 3D coordinates of the box being disassembled, and the other to calculate the height by photographing the side of the box. This approach requires two cameras, resulting in wasted equipment. Furthermore, during the second photograph, the camera needs to pause at a specific location to capture the side of the box, which consumes time in the overall process. Summary of the Invention

[0004] This application provides a palletizing method, apparatus, electronic device, and storage medium for calculating cargo height using a depth camera, saving equipment costs, and eliminating interruptions during the entire cargo movement process, thus saving overall palletizing time.

[0005] To achieve the above technical objectives, this application adopts the following technical solution:

[0006] In a first aspect, embodiments of this application provide a palletizing method, the method comprising: acquiring a first depth image captured by a depth camera over a goods placement area, the first depth image containing a first goods; acquiring first area information based on the first depth image, the first area being an area in the placement area where the first goods are placed; controlling a moving device based on the first area information, such that the moving device moves the first goods out of the space where the first area is located; after the moving device moves the first goods out of the space where the first area is located, acquiring a second depth image captured by a depth camera over the goods placement area, the first area in the second depth image not containing the first goods; acquiring the height of the first goods based on the first depth image and the second depth image; and controlling the moving device based on the height of the first goods, such that the moving device palletizes the first goods.

[0007] Understandably, by analyzing two depth images captured by a depth camera, the height of the goods can be quickly calculated, effectively reducing the risk of damage during the stacking process. At the same time, a single depth camera can save on equipment costs, and the entire process of moving goods is uninterrupted, saving time in the overall stacking process.

[0008] In one possible implementation, controlling the mobile device based on the first area information to move the first cargo out of the space where the first area is located includes: sending first instruction information to the mobile device based on the first area information; wherein the first instruction information is used to instruct the first cargo to be moved to a first spatial location, the first spatial location being outside the field of view of the depth camera.

[0009] It is understandable that one way to move the first cargo out of the first area is to move it outside the field of view of the depth camera. A first spatial position can be set outside the field of view of the depth camera. This method can effectively ensure that the first cargo is moved out of the first area.

[0010] In another possible implementation, the above-mentioned control of the mobile device based on the first area information to move the first goods out of the space where the first area is located includes: sending second instruction information to the mobile device based on the first area information; wherein the second instruction information is used to instruct the first goods to be moved to a second spatial position, the second spatial position being the spatial position of the first goods when being stacked; the second spatial position is mapped to the area of ​​the plane where the goods are placed and does not overlap with the first area; the method further includes: before moving the first goods to the second spatial position and after moving them out of the space where the first area is located, controlling a depth camera to acquire a second depth image.

[0011] It is understandable that after the first cargo moves out of the first area and before it reaches the second spatial position, the electronic device can control the depth camera to acquire a second depth image. This method allows the depth camera to acquire a second depth image simultaneously while the first cargo is moving, so that the first cargo does not need to stop, saving overall process time and improving overall efficiency.

[0012] In another possible implementation, the second depth image includes the second goods, and the method further includes: acquiring second region information in the second depth image based on the second depth image; wherein the second region is the area in the goods placement area where the second goods are placed; controlling a mobile device based on the second region information so that the mobile device moves the second goods out of the space where the second region is located; after the mobile device moves the second goods out of the space where the second region is located, acquiring a third depth image captured by a depth camera for the goods placement area; wherein the second region in the third depth image does not contain the second goods; acquiring the height of the second goods based on the second depth image and the third depth image; and controlling the mobile device based on the height of the second goods so that the mobile device stacks the second goods.

[0013] Understandably, if there is more than one item in the goods placement area, the next goods will be stacked after the previous stacking is completed. In the next stacking process, the second depth image captured by the previous depth camera can be used as the first depth image for the current stacking process. This method reuses depth images, reducing the number of depth camera shots, streamlining the overall stacking process, saving system power consumption, and improving stacking efficiency.

[0014] In another possible implementation, a response message sent by a mobile device is received; wherein the response message is used to indicate that the first goods have been moved out of the space where the first area is located; or, when a preset time period from the start of sending the second instruction information arrives, it is determined that the first goods have been moved out of the space where the first area is located.

[0015] Understandably, once the first item is moved out of the space containing the first area, the electronic device needs to receive a response message to inform it that the first item has been moved out of the first area. This response message allows the electronic device to promptly obtain information about the first item's departure from the first area, which facilitates subsequent operations and advances the overall process.

[0016] In another possible implementation, the depth camera is mounted directly above the cargo placement area.

[0017] Understandably, depth cameras are used to capture images of the entire area where goods are placed, and the range of the images captured by the depth camera is greater than or equal to the area where goods are placed. Therefore, the depth camera needs to be installed above the area where goods are placed, and being directly above it can effectively avoid blind spots and make the images captured by the depth camera clearer and more complete.

[0018] In another possible implementation, the first region information includes the first location information and the first depth information of the first region in the first depth image.

[0019] It is understandable that the first region information can include the position and depth information of the first region in the depth camera coordinate system, or it can include the three-dimensional coordinate values ​​of the first region in the world coordinate system. Choosing the position and depth information in the depth camera coordinate system as the values ​​for calculating the cargo height and locating the cargo position is an optimal method. It does not require transforming the coordinates to the world coordinate system for calculation, which can improve the speed of cargo positioning and height calculation, and improve system efficiency.

[0020] Secondly, this application provides a palletizing device. The palletizing device includes various modules of a method applied to the first aspect or any possible design of the first aspect.

[0021] Thirdly, this application provides an electronic device including a memory and a processor. The memory and the processor are coupled; the memory stores computer program code, which includes computer instructions. When the processor executes the computer instructions, it causes the electronic device to perform a palletizing method as described in the first aspect and any of its possible design embodiments.

[0022] Fourthly, this application provides a computer-readable storage medium including computer instructions. These computer instructions, when executed on an electronic device, cause the electronic device to perform a palletizing method as described in the first aspect and any of its possible design embodiments.

[0023] Fifthly, this application provides a computer program product comprising computer instructions. When the computer instructions are executed on an electronic device, they cause the electronic device to perform a palletizing method as described in the first aspect and any of its possible design embodiments.

[0024] For a detailed description of the second to fifth aspects and their various implementations in this application, please refer to the detailed description in the first aspect and its various implementations; and for a detailed analysis of the beneficial effects of the second to fifth aspects and their various implementations in the first aspect and its various implementations, please refer to the beneficial effect analysis in the first aspect and its various implementations, which will not be repeated here.

[0025] These or other aspects of this application will become more readily apparent in the following description. Attached Figure Description

[0026] Figure 1 A schematic diagram illustrating the implementation environment of a palletizing method provided in this application embodiment;

[0027] Figure 2 A flowchart of a palletizing method provided in an embodiment of this application;

[0028] Figure 3 A front view showing the relationship between the field of view of a depth camera and the location of a cargo placement area, provided in an embodiment of this application;

[0029] Figure 4 This application provides a locational relationship between a depth camera and the area where the cargo is located (i.e., a first area), as well as a depth relationship diagram;

[0030] Figure 5 A top view showing the positional relationship between a cargo placement area and a first cargo, provided for an embodiment of this application;

[0031] Figure 6 A top view of the cargo placement area after the first cargo has been moved out of the field of view of the depth camera, provided as an embodiment of this application;

[0032] Figure 7 This application provides a diagram showing the relationship between a second spatial location and a goods stacking area, as well as a diagram showing the positional relationship between the goods stacking area and the goods placement area.

[0033] Figure 8 A schematic diagram illustrating a mobile device moving goods to a preset height (safety height) according to an embodiment of this application;

[0034] Figure 9 This application provides a specific stacking diagram for an embodiment of the invention.

[0035] Figure 10 A top view of the second cargo position relationship in a cargo placement area provided in an embodiment of this application;

[0036] Figure 11 This is a schematic diagram of the structure of a palletizing device provided in an embodiment of this application;

[0037] Figure 12 This is a schematic diagram of the structure of another electronic device provided in an embodiment of this application. Detailed Implementation

[0038] Hereinafter, the terms "first," "second," and "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," or "third," etc., may explicitly or implicitly include one or more of that feature.

[0039] In robotic applications involving mixed unpacking and stacking, due to the inconsistent heights of the boxes being unpacked, if the robot cannot determine the box height during grasping and placement, it will be unable to ascertain the required lifting and lowering heights during handling. This can lead to box collisions during destacking and box crushing or dropping during stacking. Therefore, we need to solve the problem of calculating box height.

[0040] Based on this, this application provides a palletizing method. This method calculates the height of goods (such as boxes) to be moved using depth images captured by a depth camera, and then stacks the goods according to their height. First, the location information of the box's location is obtained by analyzing the depth image captured by the depth camera over the goods placement area. Based on this location information, a moving device is controlled to move the box. Second, after the goods are moved out of their original location, another depth image is captured over the same area. Based on the depth information of the goods contained in the two depth images, the height of the moved goods can be calculated. Finally, based on the height of the goods, the moving device is controlled to move the goods to a safe height before stacking. It is understood that by analyzing two depth images captured by a single depth camera, the height of the goods can be quickly calculated, effectively reducing the risk of damage during stacking. Simultaneously, a single depth camera saves equipment costs, and the entire goods movement process is uninterrupted, saving overall stacking time.

[0041] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0042] Please refer to Figure 1 , Figure 1 This is a schematic diagram illustrating the implementation environment of a palletizing method provided in an embodiment of this application. For example... Figure 1 As shown, the implementation environment may include: electronic device 100 and depth camera 110, mobile device 120 and cargo connected to electronic device 100.

[0043] The depth camera 110 is a device used to acquire depth images, such as a 3D depth camera. The depth camera can be installed directly above the goods placement area (e.g., pallets) to acquire depth images of the area. Alternatively, the depth camera can be installed at a location not directly above the goods placement area, where its field of view covers the entire area. This application does not limit the installation location. Generally, for ease of calculation, it is usually installed directly above the goods placement area.

[0044] The electronic device 100 is used to receive the depth image sent by the depth camera 110 and perform related processing (such as coordinate calculation / depth calculation) to implement the palletizing method provided in the embodiments of this application.

[0045] For example, the electronic device 100 can be a terminal, such as a mobile phone, tablet computer, desktop computer, laptop computer, netbook, etc., or it can be a server. The embodiments of this application do not impose special limitations on the specific form of the electronic device.

[0046] The mobile device 120 is used to grasp, move, and stack goods. The mobile device 120 receives instructions from the electronic device 100. The mobile device may include a robotic arm, a mechanical suction cup, or other device capable of moving goods; this embodiment does not impose any special limitations on the specific form of the mobile device 120.

[0047] The electronic device 100 and the depth camera 110 can be set up independently or integrated, and can be connected via wired or wireless means. This application embodiment does not impose any special restrictions on the setting or connection method of the electronic device 100 and the depth camera 110.

[0048] The electronic device 100 and the mobile device 120 can be set up independently or integrated, and can be connected by wires or wirelessly. This application embodiment does not impose any special restrictions on the setting or connection method of the electronic device 100 and the mobile device 120.

[0049] In one application scenario, a pile of express delivery boxes needs to be moved from a truck to a designated location in a warehouse. A depth camera 110 is installed directly above the truck. After acquiring the first depth image of all the boxes on the truck, the electronic device 100 sends a destacking command to the mobile device 120. The mobile device 120 grabs one of the boxes according to the destacking command and moves it to the designated location in the warehouse. At the same time, the depth camera 110 acquires a second depth image of the remaining boxes on the truck after the moved box. Based on the two depth images, the electronic device 100 calculates the height of the moved box. The mobile device 120 lowers the box to a safe height according to the height of the box and then stacks the box in the warehouse.

[0050] The palletizing method provided in this application embodiment can be applied to electronic device 110. The executing entity of the palletizing method provided in this application embodiment can also be a palletizing device. This palletizing device can be an electronic device, an application (APP) installed on the electronic device that provides palletizing functionality, a central processing unit (CPU) in the electronic device, or a control module in the electronic device for executing the palletizing method. The following description uses an electronic device as an example to illustrate the method provided in this application embodiment.

[0051] Please refer to Figure 2 This is a flowchart illustrating a palletizing method provided in an embodiment of this application. This method can be applied to the aforementioned electronic device. Figure 2 As shown, the method may include S101-S107.

[0052] S101: The electronic device acquires a first depth image of the cargo placement area captured by a depth camera. The first depth image contains the first cargo.

[0053] The cargo placement area is the area within the depth camera's field of view where the cargo is placed. For example... Figure 3 As shown, Figure 3 This is a front view showing the relationship between the depth camera's field of view and the location of the cargo placement area.

[0054] A depth image, also known as a distance image, is an image that uses the distance of each point in an image captured by a depth camera from the horizontal plane where the camera is located as pixel values. It directly reflects the geometry of the visible surface of the image. For example... Figure 4 As shown, Figure 4 This indicates the positional relationship and depth relationship between the depth camera and the area where the cargo is located (i.e., the first area). Figure 4 The depth of the first cargo is h.

[0055] The first item refers to any single item within the designated storage area. For example... Figure 5 As shown, Figure 5 This is a top view showing the positional relationship between the cargo placement area and the first cargo.

[0056] In one example, if there are multiple goods in the goods placement area, staff can set certain selection rules for the electronic device according to specific needs, and determine one of the goods as the first goods based on the rules. The selection rule can determine the first goods based on a certain pattern (e.g., the goods that are on the far left and at the top of the goods placement area), or it can be randomly selected.

[0057] S102: The electronic device acquires information about a first region based on a first depth image; wherein the first region is the region in the cargo placement area where the first cargo is placed.

[0058] The first region can be the area on the horizontal plane where the first cargo is located, mapped onto the cargo placement area. For example, if the first cargo is a cuboid, then the first region can be the area on the horizontal plane where the cuboid is located, mapped onto the cargo placement area, i.e., a rectangle.

[0059] Optionally, the first region information includes the first location information and the first depth information of the first region in the first depth image.

[0060] In one example, the first location information can be represented by the horizontal coordinates of multiple points in the first region. If the first region is rectangular, the first location information can be represented by the horizontal coordinates of the four vertices of the first region. The horizontal coordinates of a point are its coordinates within the horizontal coordinate system of the goods placement area.

[0061] The first depth information can be characterized by the depth values ​​of multiple points in the first region (i.e., the distances from multiple points to the horizontal plane where the depth camera is located). For example, if the first region is a rectangle, the first depth information can include the distances from the four vertices of the rectangle to the horizontal plane where the depth camera is located.

[0062] Optionally, the first region information includes the three-dimensional coordinate values ​​of the upper surface of the first cargo in the first depth image. The establishment of its coordinate system is not limited and can be the world coordinate system.

[0063] In one example, when the first cargo is a cuboid, the first region information contains the three-dimensional coordinates of the four vertices of the upper surface of the first cargo in the world coordinate system.

[0064] S103: The electronic device controls the mobile device based on the first area information, so that the mobile device moves the first goods out of the space where the first area is located.

[0065] The space where the first region is located is the space above the plane where the first region is located, within the field of view of the depth camera.

[0066] Moving the first cargo out of the space where the first area is located may include moving the first cargo to space location 1 and space location 2.

[0067] Spatial location 1: A spatial location outside the field of view of the depth camera.

[0068] In this case, S103 may include: the electronic device sending first instruction information to the mobile device based on the first area information; wherein the first instruction information is used to instruct the first cargo to be moved to a first spatial location, the first spatial location being outside the field of view of the depth camera.

[0069] For example, the electronic device determines first instruction information based on first area information; wherein the first instruction information includes the movement route information of the mobile device (such as movement direction and movement distance); and sends the first instruction information to the mobile device.

[0070] like Figure 6 The image shown is a top view of the cargo placement area after the first cargo has been moved out of the depth camera's field of view. Figure 6 Based on Figure 5 It is used for drawing.

[0071] Spatial location 2: A spatial location outside the space of the first region, within the field of view of the depth camera.

[0072] Optionally, the electronic device determines instruction information based on the first area information and sends the instruction information to the mobile device. This instruction information instructs the mobile device to move the first item out of the space containing the first area. For example, the instruction information may include the direction of movement of the mobile device and the distance moved in that direction.

[0073] The process of unpacking the first item is completed by moving the first item out of the space where the first area is located.

[0074] S104: After the mobile device moves the first cargo out of the space where the first area is located, the electronic device acquires a second depth image of the cargo placement area captured by the depth camera. The first area in the second depth image does not contain the first cargo.

[0075] This application does not limit the implementation method of the electronic device determining that the mobile device has moved the first goods out of the space where the first area is located. For example, it may include the following method 1 or method 2:

[0076] Method 1: When the first item is moved out of the first area, the electronic device will receive a response message sent by the mobile device, which indicates that the first item has been moved out of the space where the first area is located.

[0077] For example, when the first cargo is moved to spatial location 1, the electronic device receives a response message sent by the mobile device; wherein the response message indicates that the first cargo has been moved to the first spatial location. Alternatively, when the first cargo is moved to spatial location 2, the electronic device receives a response message sent by the mobile device. Here, the response message indicates that the first cargo has been moved out of the space containing the first area.

[0078] Method 2: The electronic device determines that the first goods have been moved out of the space where the first area is located when the preset time period from the start of sending the second instruction information arrives. That is, the electronic device predicts that the moving device has moved the first goods out of the space where the first area is located when the preset time period from the start of sending the second instruction information arrives.

[0079] Optionally, S103 may include:

[0080] The electronic device sends second instruction information to the mobile device based on the first area information; wherein, the second instruction information is used to instruct the first goods to be moved to a second spatial position, the second spatial position being the spatial position when the first goods are stacked; the second spatial position is mapped to the area of ​​the plane where the goods are placed, and does not overlap with the first area.

[0081] In this case, prior to S104, the method may further include: before the electronic device moves the first cargo to the second spatial location and after it moves out of the space where the first area is located, controlling the depth camera to acquire a second depth image.

[0082] The second spatial location can be determined manually via electronic devices. The area mapped onto the plane of the goods placement area is the goods stacking area. This goods stacking area can be determined manually via electronic devices as needed, or it can be a predefined area within the electronic devices. For example... Figure 7 The diagram shows the relationship between the second spatial location and the goods stacking area, as well as the positional relationship between the goods stacking area and the goods placement area.

[0083] Since the information about the first region is based on the three-dimensional coordinate system of the depth camera, while the movement of the first cargo by the mobile device is based on the three-dimensional coordinate system of the mobile device, the electronic device needs to perform a coordinate transformation on the information about the first region before sending the second instruction information to the mobile device. For example, a hand-eye calibration tool can be used to establish the transformation relationship between the three-dimensional coordinates of the depth camera and the three-dimensional coordinates of the mobile device.

[0084] The second instruction information may include route information for moving to the second spatial location. The mobile device grasps and moves the first cargo according to this second instruction information. This application embodiment does not limit the specific route information in the second instruction information, as long as the cargo does not collide with other cargo during the translation process. Generally, the cargo is first raised to a preset height, which is the safe height for cargo translation (i.e., the height at which it does not collide with other cargo). Figure 8 As shown, Figure 8 The mobile device moves the goods to a preset height (safe height).

[0085] Optionally, the electronic device controlling the depth camera to acquire a second depth image may include: the electronic device sending instruction information to the depth camera, the instruction information being used to instruct the depth camera to acquire a second depth image.

[0086] In this optional implementation, the electronic device acquires the second depth image before the first cargo is moved to the second spatial position and after it has left the space containing the first area. The height of the first cargo can then be calculated based on this second depth image. In other words, the height of the first cargo is calculated synchronously during its movement to the second spatial position. The first cargo does not need to stop during the entire movement process, saving overall process time and improving overall efficiency.

[0087] S105: The electronic device obtains the height of the first cargo based on the first depth image and the second depth image.

[0088] The electronic device acquires second depth information of the first region based on the second depth image.

[0089] Since the first region in the second depth image does not contain the first cargo, the second depth information of the first region obtained based on the second depth image is the depth information contained in the first region after the first cargo is removed.

[0090] The second depth information can be characterized by the depth values ​​of multiple points in the first region after the first cargo is removed (i.e., the distances from multiple points to the horizontal plane where the depth camera is located). For example, if the first region is a rectangle, the second depth information can include the distances from the four vertices of the rectangle to the horizontal plane where the depth camera is located.

[0091] The first depth image contains first depth information of the first region, and the second depth image contains second depth information of the first region. Since the first depth information is the depth value of multiple points in the first region before the first cargo is removed, and the second depth information is the depth value of the same multiple points in the first region after the first cargo is removed, the height of the first cargo can be obtained by the distance difference between these multiple points using the first depth information and the second depth information.

[0092] In one example, if the first region is a rectangle, the first depth information can include the distances from the four vertices of the rectangle to the horizontal plane where the depth camera is located, and the second depth information can include the distances from the four vertices of the rectangle to the horizontal plane where the depth camera is located. Therefore, the difference between the depth values ​​of the four vertices in the first depth information and the second depth information represents the height of the first cargo. If the difference values ​​are equal, one of them can be randomly selected as the height of the first cargo; if the difference values ​​are not equal, the smallest value can be selected as the height of the first cargo.

[0093] Optionally, based on the second depth image, information about a third region within the second depth image can be obtained for the first region. This third region information includes the three-dimensional coordinates of the lower surface of the first cargo in the second depth image. The coordinate system used is not limited and can be a world coordinate system. Based on the third region information and the first region information, the height of the first cargo can be obtained.

[0094] In one example, when the first cargo is a cuboid, the third region information contains the three-dimensional coordinates of the four vertices of the upper surface of the first cargo in the world reference system corresponding to the four points in the second region. For example, the three-dimensional coordinates of the four vertices of the lower surface of the first cargo in the world coordinate system. The height of the first cargo is obtained by the difference between the three-dimensional coordinates of the four vertices of the upper surface of the first cargo in the first region information and the three-dimensional coordinates of the four vertices of the upper surface of the first cargo in the second region corresponding to the four vertices of the lower surface in the vertical position.

[0095] S106: The electronic device controls the moving device based on the height of the first cargo, so that the moving device can stack the first cargo.

[0096] Before the mobile device moves the first item to the second spatial position, the electronic device calculates the required descent height based on the height of the first item. After the first item is lowered to a safe distance from the ground (i.e., a distance at which the first item will not be damaged when placed in the stacking area), the electronic device controls the mobile device to lower the first item, completing the stacking of the first item.

[0097] Since the second spatial position is preset, placing goods there cannot guarantee that the height is at a safe distance from the ground, potentially posing a risk of damage to the first item. Lowering the first item to the horizontal level of the stacking area would waste time and slow down the overall stacking process. Therefore, based on the height of the first item, it is necessary to lower it to a safe distance from the ground after reaching the second spatial position before placing it down. This ensures safe stacking while saving time in the overall stacking process.

[0098] This embodiment of the solution can calculate the height of goods based on the depth camera coordinate system. The reference coordinate system can also be arbitrarily set to other coordinate systems, such as the world coordinate system. When the reference coordinate system is the world coordinate system, the height of the goods can be calculated by converting the depth value obtained from the depth image into coordinate values ​​in the world coordinate system. Therefore, this application embodiment does not limit the selection of the reference coordinate system. Generally, the method of calculating the height of goods based on the depth camera coordinate system is relatively convenient.

[0099] like Figure 9 As shown in one example, Figure 9 This demonstrates a specific stacking process in steps S101-S106 when selecting spatial location 1. The first item needs to be moved from one item placement area to another. Point A is a preset point (first spatial location) outside the field of view of the depth camera and above the first item. Point B is a preset point (second spatial location) above the area where the first item needs to be stacked. Before the first item moves, the depth camera first acquires a first depth image of the area containing the first item, obtaining first position information and first depth information for the first area. The moving device moves the first item to point A based on the first position information and first depth information. Then, the electronic device controls the depth camera to acquire a second depth image of the area containing the first item, obtaining second position information and second depth information for the first area. Simultaneously, the first item continues to move towards point B. Upon reaching point B, the electronic device determines the height of the first item based on the first and second depth images and controls the moving device to lower the first item to a safe distance from the ground before placing it down.

[0100] Optionally, S107: After the first goods are destacking and stacking are completed, during the process of destacking and stacking the second goods, the second depth image of the first goods is used as the first depth image of the second goods.

[0101] Specifically: When the second depth image contains the second goods, the method further includes: acquiring second region information in the second depth image based on the second depth image; wherein the second region is the area in the goods placement area where the second goods are placed; controlling the moving device based on the second region information so that the moving device moves the second goods out of the space where the second region is located; thus, the destacking process of the second goods is completed. After the moving device moves the second goods out of the space where the second region is located, acquiring a third depth image of the goods placement area captured by the depth camera; wherein the second region in the third depth image does not contain the second goods; acquiring the height of the second goods based on the second depth image and the third depth image; controlling the moving device based on the height of the second goods so that the moving device stacks the second goods. Thus, the stacking process of the second goods is completed.

[0102] The second depth image contains the second cargo, which is a single item within the cargo placement area, and its determination method can be the same as that of the first cargo. For example... Figure 10 As shown, Figure 10 This is a top view showing the relationship between the cargo placement area and the location of the second cargo.

[0103] One palletizing process is completed in steps S101-S106. If there is only one item in the placement area of ​​S101-S106, the palletizing ends after S106. If there is more than one item in the placement area of ​​S101-S106, the next palletizing process continues after S106. In the next palletizing process, the second depth image captured by the depth camera in the previous process can be used as the first depth image for the current palletizing process. This method reuses depth images, which can reduce the number of depth camera shots, save the overall palletizing process, save system power consumption, and improve palletizing efficiency.

[0104] This application provides a palletizing method. First, the location and depth information of the goods are obtained by analyzing the depth image captured by a depth camera over the goods placement area. Based on this location and depth information, a moving device moves the goods. Second, after the goods are moved out of the placement area, a depth image is captured again to obtain the depth information of the image after the goods have been moved. Based on these two depth images, the height of the moved goods is determined. Finally, based on the height of the goods, they are moved to a safe height and then stacked. It is understood that by analyzing two depth images captured by a single depth camera, the height of the goods can be quickly calculated, effectively reducing the risk of damage during stacking. Simultaneously, a single depth camera saves equipment costs, and the entire goods movement process is uninterrupted, saving overall stacking time.

[0105] The foregoing primarily describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the aforementioned functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0106] This application also provides a palletizing device. For example... Figure 11 The diagram shown is a structural schematic of a palletizing device 200 provided in an embodiment of this application.

[0107] The palletizing device 200 includes: an acquisition module 210, configured to acquire a first depth image captured by a depth camera over a goods placement area, wherein the first depth image contains the first goods; based on the first depth image, acquire first area information, wherein the first area is the area in the placement area where the first goods are placed; a control module 220, configured to control a moving device based on the first area information, so that the moving device moves the first goods out of the space where the first area is located; the acquisition module 210 is further configured to acquire a second depth image captured by a depth camera over the goods placement area after the moving device moves the first goods out of the space where the first area is located, wherein the first area in the second depth image does not contain the first goods; based on the first depth image and the second depth image, acquire the height of the first goods; the control module 220 is further configured to control the moving device based on the height of the first goods, so that the moving device palletizes the first goods.

[0108] In one possible embodiment, the control module 220 is specifically configured to: send first instruction information to the mobile device based on first area information; wherein the first instruction information is used to instruct the first cargo to be moved to a first spatial location, the first spatial location being outside the field of view of the depth camera.

[0109] In another possible embodiment, the control module 220 is specifically used to: send second instruction information to the mobile device based on the first area information; wherein the second instruction information is used to instruct the first goods to be moved to a second spatial position, the second spatial position being the spatial position when the first goods are stacked; the second spatial position is mapped to the area of ​​the plane where the goods are placed, and does not overlap with the first area.

[0110] In another possible embodiment, the control module 220 is further configured to: control the depth camera to acquire a second depth image before moving the first cargo to the second spatial location and after moving it out of the space where the first area is located.

[0111] In another possible embodiment, the acquisition module 210 is further configured to: acquire second region information in the second depth image based on the second depth image; wherein the second region is the region in the cargo placement area where the second cargo is placed.

[0112] In another possible embodiment, the control module 220 is further configured to: control the mobile device based on the second area information, so that the mobile device moves the second goods out of the space where the second area is located.

[0113] In another possible embodiment, the acquisition module 210 is further configured to: acquire a third depth image of the cargo placement area captured by the depth camera after the mobile device moves the second cargo out of the space where the second area is located; wherein the second area in the third depth image does not contain the second cargo; and acquire the height of the second cargo based on the second depth image and the third depth image.

[0114] In another possible embodiment, the control module 220 is further configured to: control the moving device based on the height of the second cargo, so that the moving device can stack the second cargo.

[0115] In another possible embodiment, the palletizing device further includes a receiving module 230, configured to: receive a response message sent by the mobile device; wherein the response message indicates that the first goods have been moved out of the space where the first area is located;

[0116] Alternatively, the palletizing device may further include a determining module 240 for determining, upon arrival of a preset time period from the start of sending the second instruction information, that the first goods have been moved out of the space where the first area is located.

[0117] In another possible embodiment, the depth camera is mounted directly above the cargo placement area.

[0118] In another possible embodiment, the first region information includes first location information and first depth information of the first region in the first depth image.

[0119] Of course, the palletizing device 200 provided in this application embodiment includes, but is not limited to, the above-described modules.

[0120] Figure 12 This is a schematic diagram of the structure of another electronic device 300 provided in an embodiment of this application. For example... Figure 12 As shown, the electronic device 300 includes a processor 301, a memory 302, and a network interface 303.

[0121] The processor 301 includes one or more CPUs. The CPU can be a single-core CPU or a multi-core CPU.

[0122] The memory 302 includes, but is not limited to, RAM, ROM, EPROM, flash memory, or optical memory.

[0123] Optionally, the processor 301 implements the palletizing method provided in this application embodiment by reading instructions stored in the memory 302, or the processor 301 implements the palletizing method provided in this application embodiment by internally stored instructions. When the processor 301 implements the method in the above embodiments by reading instructions stored in the memory 302, the memory 302 stores instructions for implementing the palletizing method provided in this application embodiment.

[0124] Network interface 303 is a wired interface (port), such as FDDI or GE interface. Alternatively, network interface 303 is a wireless interface. It should be understood that network interface 303 includes multiple physical ports and can be used to acquire images. Optionally, the electronic device also includes bus 304, through which the processor 301, memory 302, and network interface 303 are typically interconnected, or interconnected in other ways.

[0125] In actual implementation, the electronic device 200, the acquisition module 210, the control module 220, the receiving module 230, and the determination module 240 can be implemented by the processor calling computer program code in memory. The specific execution process can be found in the description of the method section above, and will not be repeated here.

[0126] Another embodiment of this application provides an electronic device including a memory and a processor. The memory and the processor are coupled; the memory stores computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device performs the steps of the method shown in the above-described method embodiments.

[0127] Another embodiment of this application provides a computer-readable storage medium storing computer instructions that, when executed on an electronic device, cause the electronic device to perform each step of the method flow shown in the above method embodiment.

[0128] Another embodiment of this application provides a chip system applied to an electronic device. The chip system includes one or more interface circuits and one or more processors. The interface circuits and processors are interconnected via lines. The interface circuits are used to receive signals from the electronic device's memory and send signals to the processor, the signals including computer instructions stored in the memory. When the electronic device's processor executes the computer instructions, the electronic device performs each step of the method flow shown in the above method embodiments.

[0129] In another embodiment of this application, a computer program product is also provided, which includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the various steps of the method flow shown in the above method embodiments.

[0130] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks, SSDs).

[0131] The above description is merely a specific embodiment of this application. Any variations or substitutions conceived by those skilled in the art based on the specific embodiments provided in this application should be covered within the protection scope of this application.

Claims

1. A palletizing method, characterized in that, include: A first depth image is acquired by a depth camera over a cargo placement area; wherein the first depth image contains a first cargo; the depth camera is mounted directly above the cargo placement area, and the number of depth cameras is one; multiple cargoes are placed in the cargo placement area. Based on the first depth image, first region information is obtained; wherein, the first region is the region in the placement area where the first goods are placed; Based on the information of the first area, the mobile device is controlled to move the first goods out of the space where the first area is located; Before the first cargo is moved to the second spatial position, and after it is moved out of the space where the first area is located, the depth camera is controlled to acquire a second depth image of the cargo placement area, and the second depth image is obtained; wherein, the first area in the second depth image does not contain the first cargo; the second spatial position is the spatial position when the first cargo is stacked; the second spatial position is mapped to the area of ​​the plane where the cargo placement area is located, and does not overlap with the first area; The height of the first cargo is obtained based on the first depth image and the second depth image; The moving device is controlled based on the height of the first cargo so that it can stack the first cargo. The second depth image contains the second cargo, and the method further includes: After acquiring the second depth image, during the palletizing process of the second goods, the second depth image of the first goods is used as the first depth image of the second goods, and the second goods are palletized based on the second depth image.

2. The method according to claim 1, characterized in that, The step of controlling the mobile device based on the first area information, so that the mobile device moves the first goods out of the space where the first area is located, includes: Based on the first area information, a first instruction message is sent to the mobile device; wherein the first instruction message is used to instruct the first cargo to be moved to a first spatial location, the first spatial location being outside the field of view of the depth camera.

3. The method according to claim 1, characterized in that, The step of controlling the mobile device based on the first area information, so that the mobile device moves the first goods out of the space where the first area is located, includes: Based on the first area information, a second instruction message is sent to the mobile device; wherein the second instruction message is used to instruct the first goods to be moved to the second spatial location.

4. The method according to any one of claims 1 to 3, characterized in that, The second depth image contains the second goods, and the step of using the second depth image of the first goods as the first depth image of the second goods, and palletizing the second goods based on the second depth image, includes: Based on the second depth image, second region information of the second region in the second depth image is obtained; wherein, the second region is the region in the cargo placement area where the second cargo is placed; The mobile device is controlled based on the second area information so that the mobile device moves the second goods out of the space where the second area is located; After the mobile device moves the second cargo out of the space where the second area is located, a third depth image is acquired by the depth camera for the cargo placement area; wherein the second area in the third depth image does not contain the second cargo; The height of the second cargo is obtained based on the second depth image and the third depth image; The moving device is controlled based on the height of the second cargo so that it can stack the second cargo.

5. The method according to claim 3, characterized in that, The method further includes: Receive a response message sent by the mobile device; wherein the response message is used to indicate that the first goods have been moved out of the space where the first area is located; Alternatively, it can be determined that the first goods have been moved out of the space where the first area is located when a preset time period has elapsed since the second instruction information was sent.

6. The method according to any one of claims 1 to 3, characterized in that, The first region information includes the first location information and the first depth information of the first region in the first depth image.

7. A palletizing device, characterized in that, include: An acquisition module is used to acquire a first depth image captured by a depth camera over a cargo placement area; wherein the first depth image contains a first cargo; the depth camera is mounted directly above the cargo placement area, and the number of depth cameras is one; based on the first depth image, first area information is acquired; wherein the first area is the area in the placement area where the first cargo is placed; A control module is used to control a mobile device based on the first area information, so that the mobile device moves the first goods out of the space where the first area is located. The acquisition module is further configured to, before moving the first cargo to the second spatial position and after moving it out of the space where the first area is located, control the depth camera to acquire a second depth image of the cargo placement area and obtain the second depth image; wherein, the first area in the second depth image does not contain the first cargo; the second spatial position is the spatial position of the first cargo when it is being stacked; the second spatial position is mapped to the area of ​​the plane where the cargo placement area is located and does not overlap with the first area; and based on the first depth image and the second depth image, obtain the height of the first cargo; The control module is also used to control the moving device based on the height of the first cargo, so that the moving device can stack the first cargo. The control module is further configured to: after acquiring the second depth image, during the process of palletizing the second goods, use the second depth image of the first goods as the first depth image of the second goods, and palletize the second goods based on the second depth image; the second depth image contains the second goods.

8. The palletizing device according to claim 7, characterized in that, The control module is specifically used for: Based on the first area information, a first instruction message is sent to the mobile device; wherein the first instruction message is used to instruct the first cargo to be moved to a first spatial location, the first spatial location being outside the field of view of the depth camera; Based on the first area information, a second instruction message is sent to the mobile device; wherein the second instruction message is used to instruct the first goods to be moved to the second spatial location; The control module is further configured to: before moving the first cargo to the second spatial position and after moving it out of the space where the first area is located, control the depth camera to acquire the second depth image; The acquisition module is further configured to: acquire second region information of the second region in the second depth image based on the second depth image; wherein, the second region is the region in the placement area where the second goods are placed; The control module is specifically used to: control the mobile device based on the second area information, so that the mobile device moves the second goods out of the space where the second area is located; The acquisition module is further configured to: after the mobile device moves the second cargo out of the space where the second area is located, acquire a third depth image captured by the depth camera for the cargo placement area; wherein the second area in the third depth image does not contain the second cargo; and acquire the height of the second cargo based on the second depth image and the third depth image; The control module is specifically used to: control the moving device based on the height of the second cargo, so that the moving device can stack the second cargo; The palletizing device further includes a receiving module for: receiving a response message sent by the mobile device; wherein the response message indicates that the first goods have been moved out of the space where the first area is located; Alternatively, the palletizing device may further include a determining module, configured to: determine that the first goods have been moved out of the space where the first area is located when a preset time period has elapsed since the second instruction information was sent; The first region information includes the first location information and the first depth information of the first region in the first depth image.

9. An electronic device, characterized in that, The device includes a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, the computer program code including computer instructions; wherein, when the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions; wherein, when the computer instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-6.

Images