Tray detection method

Abstract

The invention provides a tray detection method, and the method relates to the field of tray recognition, and the method comprises the steps: obtaining a color image and a depth image at the same moment, the color image and the depth image respectively comprising one or more trays; determining the position information of each tray in the depth image according to the position information of each tray in the one or more trays in the color image; determining a target point on each tray and an inclination angle of each tray according to the position information of each tray in the depth image; and obtaining the actual position of each tray according to the target point on each tray and the inclination angle of each tray. According to the method provided by the invention, the defects in the prior art can be overcome, and the tray detection accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of pallet identification, and in particular to a pallet detection method. Background Art

[0002] In recent years, with the rapid growth of logistics demand, more and more factories and warehouses have chosen to use automated guided vehicles (AGVs) instead of traditional manual forklifts. AGVs have multiple functions such as automatic navigation, movement, obstacle avoidance, multi-sensor control, and loading and transportation, and can achieve short-distance transportation and loading and unloading of cargo pallets. However, how to accurately detect and locate pallets and cargo has become an important problem facing AGVs.

[0003] Currently, the existing solutions for pallet detection are relatively simple, resulting in low accuracy of pallet detection. Summary of the invention

[0004] The present application provides a pallet detection method, which can overcome the defects of the prior art and improve the accuracy of pallet detection.

[0005] In a first aspect, an embodiment of the present application provides a pallet detection method, comprising:

[0006] Acquire a color image and a depth image at the same time, wherein the color image and the depth image respectively include one or more pallets;

[0007] Determine the position information of each tray in the depth image according to the position information of each tray of the one or more trays in the color image;

[0008] Determine a target point on each pallet and an inclination angle of each pallet according to position information of each pallet in the depth image;

[0009] The actual position of each pallet is obtained according to the target point on each pallet and the inclination angle of each pallet.

[0010] It can be seen that in the embodiment of the present application, a color image and a depth image are acquired at the same time, wherein the color image provides visual features such as the color and texture of the pallet, while the depth image provides the distance information between the pallet and the camera. The combined use of these two images can provide a more comprehensive understanding of the characteristics of the pallet, which helps to accurately determine the position of the pallet. In addition, the target point determines the specific position of the pallet in space and can be used as a reference point for positioning. By accurately determining the position of the target point, the error in the positioning process can be reduced and the accuracy of calculating the actual position is improved. The tilt angle describes the degree of inclination of the pallet relative to the horizontal plane, which helps to determine the direction of rotation of the pallet, thereby making the calculated actual position more accurate. Taking into account the information of the target point and the tilt angle, the accuracy and reliability of pallet positioning can be improved.

[0011] Based on the first aspect, in a possible implementation manner, before determining the position information of each tray in the depth image according to the position information of each tray of the one or more trays in the color image, the method further includes:

[0012] Converting the depth image into point cloud data;

[0013] Accordingly, determining the position information of each pallet in the depth image according to the position information of each pallet in the color image, and determining the target point on each pallet and the inclination angle of each pallet according to the position information of each pallet in the depth image, includes:

[0014] Determine point cloud data corresponding to each pallet according to position information of each pallet of the one or more pallets in the color image;

[0015] According to the point cloud data corresponding to each pallet, a target point on each pallet and an inclination angle of each pallet are determined.

[0016] It can be seen that in the embodiment of the present application, the depth image is converted into point cloud data by converting the pixels in the depth image into points in three-dimensional space, thereby obtaining richer three-dimensional information. Point cloud data can accurately reflect the shape and position of the pallet, which helps to determine the position of the pallet more accurately. In addition, point cloud data can avoid the noise or blur that may exist in the depth image, provide clearer and more accurate information, and help improve the accuracy of pallet detection. By using point cloud data, more detailed and reliable three-dimensional information can be obtained, thereby improving the accuracy and reliability of pallet detection.

[0017] Based on the first aspect, in a possible implementation, determining the target point on each pallet and the inclination angle of each pallet according to the point cloud data corresponding to each pallet includes:

[0018] Determine the point cloud data of the target plane in each pallet according to the point cloud data corresponding to each pallet, wherein the target plane of each pallet is the cutting plane when forking each pallet;

[0019] According to the point cloud data of the target plane in each pallet, point cloud data of a target horizontal line, point cloud data of a first target vertical line, and point cloud data of a second target vertical line on the target plane are determined respectively, wherein the target horizontal line is the uppermost edge line on the target plane, the first target vertical line is the left edge line of the middle support column of the pallet on the target plane, and the second target vertical line is the right edge line of the middle support column of the pallet on the target plane;

[0020] A target point on each pallet and an inclination angle of each pallet are determined according to the target horizontal line, the first target vertical line and the second target vertical line on the target plane of each pallet, wherein the target point is located on the target plane.

[0021] It can be seen that in the embodiment of the present application, the target plane in each pallet is first determined. The target plane is the entry plane when forking each pallet, which can provide a unified reference so that the forking action can be performed on a fixed plane. This can ensure that the forking process is more stable and reliable, and avoid errors and instability caused by different planes. The target horizontal line, the first target vertical line, and the second target vertical line on the target plane provide additional reference lines, which can help determine the specific position and posture of the pallet, and then accurately calculate the position of the target point and the tilt angle of the pallet, thereby improving the accuracy of pallet detection. Directly calculating the target point and the tilt angle may be affected by the initial error, but by gradually determining the target plane, the target horizontal line, the first target vertical line, and the second target vertical line, the accumulation of errors can be gradually reduced, thereby improving the accuracy and stability of detection.

[0022] Based on the first aspect, in a possible implementation, determining the point cloud data of the target plane in each pallet according to the point cloud data corresponding to each pallet includes:

[0023] Determine the point cloud data corresponding to each plane in each pallet according to the point cloud data corresponding to each pallet;

[0024] The point cloud data of the target plane in each pallet is determined according to the number of points in the point cloud data corresponding to each plane in each pallet and the distance from the centroid of the point cloud data corresponding to each plane in each pallet to the centroid of the point cloud data corresponding to each pallet.

[0025] It can be seen that in the embodiment of the present application, the point cloud data of the target plane in each pallet is determined according to the number of points in the point cloud data corresponding to each plane in each pallet and the distance from the centroid of the point cloud data corresponding to each plane in each pallet to the centroid of the point cloud data corresponding to each pallet. The number of points in the point cloud data of each plane reflects the density and distribution of the plane in the point cloud. If the number of point clouds on a plane is small, it may mean that the plane is blocked or interfered by noise, and the accuracy of detection will be affected. Therefore, by considering the number of points in the point cloud data, it can help exclude some unstable or unreliable plane data and improve the accuracy of target plane detection. The distance from the centroid of the plane to the centroid of the pallet point cloud data can reflect the positional relationship of the plane relative to the entire pallet. If the centroid of a plane is far away, it may mean that the positional relationship between the plane and the pallet is far, and it may not be the target plane. By considering the distance between the centroids, it can help exclude some planes with inappropriate positions and improve the accuracy of target plane detection. Taking into account the two factors of the number of points and the distance between the centroids, the fitness of each plane can be more comprehensively evaluated. If a plane has a sufficient number of point clouds and a moderate centroid distance, it is more likely to be the target plane. This comprehensive consideration can help eliminate errors caused by some single factors and improve the accuracy of determining the target plane.

[0026] Based on the first aspect, in a possible implementation, determining the target point on each pallet according to the target horizontal line, the first target vertical line, and the second target vertical line on the target plane of each pallet includes:

[0027] Determine a first reference point on the target plane of each pallet according to the target horizontal line and the first target vertical line on the target plane of each pallet, wherein the first reference point on the target plane of each pallet is an intersection of the target horizontal line and the first target vertical line on the target plane of each pallet;

[0028] Determine a second reference point on the target plane of each pallet according to the target horizontal line and the second target vertical line on the target plane of each pallet, wherein the second reference point on the target plane of each pallet is an intersection of the target horizontal line and the second target vertical line on the target plane of each pallet;

[0029] The target point on each pallet is determined according to the first reference point on the target plane of each pallet and the second reference point on the target plane of each pallet, wherein the target point on each pallet is the midpoint of the line segment where the first reference point on the target plane of each pallet and the second reference point on the target plane of each pallet are located.

[0030] It can be seen that in the embodiment of the present application, the first reference point is the intersection of the target horizontal line and the first target vertical line on the target plane, and the second reference point is the intersection of the target horizontal line and the second target vertical line on the target plane. The determination of these two reference points provides two key positioning points on the target plane. By utilizing the information of these two key points on the target plane and taking into account the intermediate position between them, the position of the target point can be determined more accurately, so that the target point is closer to the geometric center of the pallet, thereby more accurately describing the position of the pallet.

[0031] Based on the first aspect, in a possible implementation manner, the inclination angle of each tray is an angle formed by the target horizontal line on the target plane of each tray and the horizontal direction.

[0032] Based on the first aspect, in a possible implementation manner, the method further includes:

[0033] Extracting point cloud data within a certain range above the target point on each pallet, and determining the most densely populated area in the point cloud data within the certain range above the target point on each pallet;

[0034] Whether there is cargo on each pallet is determined according to the distance between the centroid of the most densely populated area and the target point on each pallet.

[0035] It can be seen that in the embodiment of the present application, the point cloud data within a certain range above the target point on each pallet is extracted, and the densest area in the point cloud data within a certain range above the target point on each pallet is determined. Since dense point cloud areas often correspond to the stacking areas of goods on the pallet, determining the densest area helps to determine the location where the goods may exist. Given that goods tend to be concentrated in specific areas on the pallet, the center position of the area can be obtained by calculating the center of mass of the densest area, which reflects the approximate location of the goods on the pallet. If the distance between the center of mass of the densest area and the target point is close enough, it can be considered that there are goods on the pallet; otherwise, it may mean that the pallet is empty. By considering the distance between the center of mass of the densest area and the target point, the accuracy of judging whether goods exist can be improved.

[0036] In a second aspect, an embodiment of the present application provides a pallet detection device, comprising:

[0037] An acquisition module, used to acquire a color image and a depth image at the same time, wherein the color image and the depth image respectively include one or more pallets;

[0038] a determination module, configured to determine position information of each tray in the depth image according to position information of each tray in the one or more trays in the color image;

[0039] The determination module is further used to determine the target point on each pallet and the inclination angle of each pallet according to the position information of each pallet in the depth image;

[0040] The actual position determination module is used to obtain the actual position of each pallet according to the target point on each pallet and the inclination angle of each pallet.

[0041] Based on the second aspect, in a possible implementation, the device further includes a conversion module, and the conversion module is used to convert the depth image into point cloud data;

[0042] The determination module is used to determine the point cloud data corresponding to each pallet according to the position information of each pallet of the one or more pallets in the color image;

[0043] The determination module is further used to determine the target point on each pallet and the inclination angle of each pallet according to the point cloud data corresponding to each pallet.

[0044] Based on the second aspect, in a possible implementation,

[0045] The determination module is used to determine the point cloud data of the target plane in each pallet according to the point cloud data corresponding to each pallet, and the target plane of each pallet is the cutting plane when forking each pallet;

[0046] The determination module is also used to determine, according to the point cloud data of the target plane in each pallet, the point cloud data of the target horizontal line, the point cloud data of the first target vertical line, and the point cloud data of the second target vertical line on the target plane, respectively, wherein the target horizontal line is the uppermost edge line on the target plane, the first target vertical line is the left edge line of the middle support column of the pallet on the target plane, and the second target vertical line is the right edge line of the middle support column of the pallet on the target plane;

[0047] The determination module is also used to determine a target point on each pallet and an inclination angle of each pallet based on the target horizontal line, the first target vertical line and the second target vertical line on the target plane of each pallet, wherein the target point is located on the target plane.

[0048] Based on the second aspect, in a possible implementation method, the determination module is used to determine the point cloud data of the target plane in each pallet according to the number of points in the point cloud data corresponding to each plane in each pallet and the distance from the center of mass of the point cloud data corresponding to each plane in each pallet to the center of mass of the point cloud data corresponding to each pallet.

[0049] Based on the second aspect, in a possible implementation, the determination module is used to determine a first reference point on the target plane of each pallet according to the target horizontal line and the first target vertical line on the target plane of each pallet, wherein the first reference point on the target plane of each pallet is an intersection of the target horizontal line and the first target vertical line on the target plane of each pallet;

[0050] The determination module is further used to determine a second reference point on the target plane of each pallet according to the target horizontal line and the second target vertical line on the target plane of each pallet, wherein the second reference point on the target plane of each pallet is an intersection of the target horizontal line and the second target vertical line on the target plane of each pallet;

[0051] The determination module is also used to determine the target point on each pallet based on the first reference point on the target plane of each pallet and the second reference point on the target plane of each pallet, the target point on each pallet being the midpoint of the line segment where the first reference point on the target plane of each pallet and the second reference point on the target plane of each pallet are located.

[0052] Based on the second aspect, in a possible implementation, the inclination angle of each tray is an angle formed by the target horizontal line on the target plane of each tray and the horizontal direction.

[0053] Based on the second aspect, in a possible implementation, the device further includes an extraction module, the extraction module being used to extract point cloud data within a certain range above the target point on each pallet, and determine the most densely populated area in the point cloud data within a certain range above the target point on each pallet;

[0054] The determination module is used to determine whether there is cargo on each pallet according to the distance between the centroid of the most densely populated area and the target point on each pallet.

[0055] Each functional module in the second aspect is used to implement the method described in the first aspect and the possible implementation manner of the first aspect.

[0056] In a third aspect, an embodiment of the present application provides a computing device, comprising a memory and a processor, wherein the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the method described in the above-mentioned first aspect and any possible implementation method of the first aspect.

[0057] In a fourth aspect, an embodiment of the present application provides a computer storage medium, comprising program instructions, which, when executed by an AGV, causes the AGV to execute the method described in the first aspect and any possible implementation manner of the first aspect.

[0058] In a fifth aspect, the present application provides a computer program product, including program instructions, and when the computer program product is executed by the AGV, the controller is used to execute the method described in the first aspect and any possible implementation of the first aspect. The computer program product can be a software installation package, and when it is necessary to use the method provided by any possible design of the first aspect, the computer program product can be downloaded and executed on the AGV to implement the method described in the first aspect and any possible implementation of the first aspect.

[0059] In a sixth aspect, the present application provides a forklift, comprising a forklift device and a pallet detection device, or comprising a forklift device and a computing device, wherein the pallet detection device or the computing device is used to implement the method described in the first aspect and any possible implementation method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0061] Figure 1 It is a structural schematic diagram of a pallet provided by the present application;

[0062] Figure 2 It is a structural schematic diagram of an AGV of the present application;

[0063] Figure 3 It is a flow chart of a pallet detection method provided by the present application;

[0064] Figure 4 It is a schematic diagram of an optimized network structure provided by this application;

[0065] Figure 5 It is a schematic diagram of the structure of the PEM submodule provided in this application;

[0066] Figure 6 is a schematic diagram of pallet target detection provided by the present application;

[0067] Figure 7 is a schematic diagram of a target plane of a tray provided in the present application;

[0068] Figure 8 is a schematic diagram of point cloud data of a target plane including only boundary points provided by the present application;

[0069] Fig. 9 is a schematic diagram of point cloud data of a target horizontal line on a target plane provided by the present application;

[0070] Fig.10 is a schematic diagram of determining a first target vertical line and a second target vertical line on a target plane provided by the present application;

[0071] Fig.11 is a schematic diagram of a first reference point, a second reference point and a target point provided by the present application;

[0072] Fig.12 It is a schematic diagram of the AGV provided by the present application moving to the target location along the planned path;

[0073] Fig.13 It is a flow chart of another pallet detection method provided by the present application;

[0074] Fig.14 It is a structural schematic diagram of a pallet detection device provided by the present application;

[0075] Fig.15 It is a structural schematic diagram of a computing device provided by this application. DETAILED DESCRIPTION

[0076] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0077] It should be noted that the terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0078] It should be noted that when used in this specification and the appended claims, the terms "comprises" and any of their variations are intended to cover non-exclusive inclusions. For example, a system, product or device including a series of units / devices is not limited to the listed units / devices, but may optionally include units / devices not listed, or may also optionally include other units / devices inherent to these products or devices.

[0079] It will also be understood that the term “if” may be interpreted as “when” or “upon” or “in response to determining” or “in response to detecting” or “in the event of…” depending on the context.

[0080] Before introducing the embodiments of the present application, the technical terms involved in the embodiments of the present application are first introduced.

[0081] AGV: It is an unmanned automated vehicle, usually used for material handling and transportation tasks in factories, warehouses, logistics centers and other places. AGV is usually equipped with sensors, navigation systems and control systems, and can drive autonomously according to pre-set paths and tasks without manual control. This automated vehicle can improve logistics efficiency, reduce labor costs, and play an important role in industrial production and logistics.

[0082] Point cloud: is a set of discrete points in three-dimensional space, each point has spatial coordinate information (usually x, y, z coordinates) and possible other attribute information (such as color, normal vector, etc.). Point cloud is usually used to represent and describe the three-dimensional shape and features of objects, scenes or environments. It is a commonly used data form in computer vision, robotics, cartography and other fields.

[0083] Before introducing the embodiments of the present application, the scenarios involved in the embodiments of the present application are first introduced.

[0084] AGV is widely used in transportation tasks. For example, in an automobile manufacturing plant, AGV transports parts from the warehouse or supply area to the production line to ensure the continuous operation of the production line. For another example, in a hospital or laboratory, AGV transports medicines, reagents and medical equipment. This application does not make specific restrictions on the use scenarios of AGV. The following is a detailed introduction to the AGV workflow using the automobile manufacturing scenario as an example.

[0085] In automobile manufacturing plants, multiple production lines assemble vehicles simultaneously. Each vehicle is made up of hundreds of parts, which need to be transported from the warehouse to the corresponding production line to ensure production progress and quality. Suppose a production line for a certain model needs to install parts such as engines, seats, and tires. Among them, AGV needs to take out the engine parts from the warehouse and move them to the designated location of the production line for workers to install on the vehicle.

[0086] See also Figure 1 , Figure 1A schematic diagram of the structure of a pallet provided for the present application. As a device for storing goods, a pallet can accommodate multiple goods at the same time, including but not limited to parts, finished products or other items. The pallet consists of a flat base plate and three support columns. The support columns are located at the bottom of the base plate, and there is a gap between two support columns to facilitate forking by AGV or other transportation equipment. Among them, the base plate can be made of steel plate, plastic plate or other durable materials to bear the weight of the goods; the support columns are usually made of strong metal materials to ensure the stability and safety of the pallet during transportation. It should be noted that there are many types of pallets, and this application takes one type of pallet as an example to introduce. Regarding the type of pallet, this application does not make specific restrictions.

[0087] See also Figure 2 , Figure 2 This is a schematic diagram of the structure of an AGV of this application. The AGV is equipped with a camera and a fork-picking device. The camera is used to capture images inside the warehouse and provide the AGV with the location information of the pallet, so that the AGV can plan the path to grab the pallet. The fork-picking device is usually composed of a pair of fork arms, which can flexibly adjust the position and angle of the fork to accommodate pallets of different sizes and shapes. The AGV can accurately grab the pallet through the fork-picking device and safely move the pallet to the designated location, realizing fast and efficient transportation of goods.

[0088] After receiving the task, the AGV plans the route according to the warehouse map and production line layout, goes to the location of the engine parts in the warehouse, and uses the onboard camera to scan the location of the pallet where the engine parts are stored. Once the target is determined, the AGV's fork pick-up device will descend under the pallet and tightly grasp the pallet. The AGV lifts the forked pallet and begins to move along the predetermined path. During transportation, the AGV continuously monitors the surrounding environment, uses sensors such as lidar and cameras to detect obstacles, and adjusts the route in real time to avoid obstacles. Once the AGV arrives at the designated workstation on the production line, it will accurately align and lower the fork pick-up device to place the pallet in the designated location. After the AGV completes the task, it will send a completion signal to the warehouse management system and update the location information of the parts.

[0089] See also Figure 3 , Figure 3 A schematic flow chart of a pallet detection method provided in the present application, the method includes but is not limited to the description of the following contents.

[0090] S101 . Acquire a color image and a depth image at the same time, wherein the color image and the depth image respectively include one or more trays.

[0091] AGVs usually first sense the surrounding environment through lidar or other sensors to determine the location and path of the pallet. Once the approximate location of the target pallet is determined, the AGV will approach the pallet at a certain distance and use the onboard camera to obtain a color image and depth image at the same time, where the color image and depth image each include one or more pallets.

[0092] The camera carried by the AGV can be an RGB-D camera, which is a device that integrates a color camera and a depth sensor and can capture color images and depth images at the same time. In addition to the RGB-D camera, the AGV can also use other types of cameras to obtain color images and depth images. This application does not specifically limit the type of camera.

[0093] Each pixel in a color image contains color information from three channels: red, green, and blue, which can show the appearance and color information of an object. The value of each pixel in a depth image is the distance of each object or surface from the camera in the real scene.

[0094] It should be noted that when acquiring the color image and depth image at the same time, the color image and the depth image need to be aligned to ensure that the color image and the depth image are one-to-one corresponding in space. Alignment refers to matching two different types of images (such as color image and depth image) in space so that their pixels correspond to the same position, thereby achieving consistency of color information and depth information.

[0095] S102: Convert the depth image into point cloud data.

[0096] Although the depth image provides the distance information from each object or surface to the camera in the real scene, it is still a two-dimensional image and cannot intuitively display the three-dimensional shape and structure of the object. By converting the depth image into point cloud data, the depth value of each pixel can be converted into coordinates in three-dimensional space, thereby obtaining a three-dimensional point cloud data composed of points, which is convenient for locating the position of each pallet.

[0097] The following describes how to convert depth images into point cloud data.

[0098] When converting a depth image into point cloud data, the pixel points in the depth image are usually mapped to coordinates in three-dimensional space based on the relationship between the camera's intrinsic parameter information, the depth value in the depth image, and the pixel coordinates, thereby converting it into point cloud data. The conversion formula for converting a depth image into point cloud data is as follows:

[0099]

[0100]

[0101] z p =z i

[0102] Among them, x p ,y p and z p Represents the three-dimensional coordinates of each point in the point cloud data; x i and i Represents the two-dimensional coordinates of each pixel in the depth image; f x Indicates the focal length of the camera in the x direction, which affects the scale and clarity of the image in the x direction; f y Indicates the focal length of the camera in the y direction, which affects the scaling and clarity of the image in the y direction; c x represents the coordinate of the principal point in the x direction on the depth image plane; c y Represents the coordinate of the principal point on the depth image plane in the y direction. The principal point refers to the projection point of the optical center on the imaging plane, which is also the center point of the image and affects the distortion and correction of the image.

[0103] S103: Determine the position information of each tray in the color image.

[0104] The color image is input into the trained pallet detection model, and the location information of each pallet in the color image is output.

[0105] The following introduces the components of the pallet detection model and the training process of the pallet detection model.

[0106] Before training the pallet detection model, it is necessary to build a pallet dataset. Dozens of color images are collected for each type of pallet. Then, the target categories and positions in the images are manually labeled to generate a labeling file.

[0107] The pallet detection model used in this application detects the position information of the pallet in the color image based on the YOLOV5 model. The YOLOV5 model mainly includes the Backbone module, the Neck module, and the Head module. Among them, the Backbone module is mainly responsible for extracting feature information from the input image; the Neck module is used to further process the feature map from the Backbone module to improve the detection performance; the Head module is responsible for predicting the position and category of the pallet, usually including multiple convolutional layers, pooling layers, and the final detection head to generate the target box and category probability.

[0108] It should be noted that the C3 submodule is inserted between the convolutional layers in the Backbone module and the Neck module. The C3 submodule is used to compress the channels of the feature map and increase the receptive field. The C3 submodule improves the performance and efficiency of the model by dividing the input feature map into multiple sub-channel groups and applying different convolution kernels in each group to achieve information interaction between channels.

[0109] In the process of aligning the color image with the depth image, it may be necessary to crop, scale, or transform the image to ensure that the color image and the depth image correspond to each other in space. These operations may cause some objects in the original image to be cropped or partially blocked, thereby changing the quality and content of the color image. In addition, in the pallet detection algorithm, the requirements for detection accuracy and real-time performance are high, because the detection accuracy directly affects the quality of subsequent point cloud extraction, and the real-time performance is directly related to whether the AGV can accurately operate according to the output pallet position information without affecting its speed. Therefore, it is necessary to comprehensively consider the impact of image processing on detection accuracy and real-time performance.

[0110] See also Figure 4 , Figure 4 This is a schematic diagram of an optimized network structure provided in this application. In this application, the PEM submodule is used to replace the C3 submodule in the Neck module to reduce the complexity of the model and improve the real-time performance of the model. Figure 5 , Figure 5 A schematic diagram of the structure of the PEM submodule provided for this application. The PEM submodule contains two branches, one of which uses ordinary convolution to operate on some input channels to extract spatial features and keep other channels unchanged, thereby reducing redundant operations. The other branch constructs a feature map through cross-channel interaction without channel dimensionality reduction to increase the network's attention to different features. By multiplying the two branches on the channel, the complexity of the model is reduced.

[0111] Specifically, the input feature map is first subjected to a conventional 1x1 convolution, and the formula is as follows:

[0112] x1=Conv1×1(x)

[0113] Among them, x represents the input feature map, and Conv1x1() refers to the regular 1x1 convolution, which does not change the number of channels of the input feature map.

[0114] Then, x1 performs the operations of the two branches respectively, and the operations of one branch include Split(), Conv3x3(), and Concat(). Among them, the Split() operation is used to divide the channels of the feature map into one quarter and three quarters. The Conv3x3() operation performs a 3x3 convolution on one quarter of the channel. The Concat() operation is used to connect the feature map of the three quarters channel with the feature map of the one quarter channel on the channel to ensure that the feature map sizes of the front and rear branches are consistent. The specific formula is as follows:

[0115] x ～ =Concat(Conv3×3(Split(x1)),Split(x1))

[0116] Another branch of the PEM submodule constructs feature maps through cross-channel interactions without performing channel dimensionality reduction. For the sake of brevity of the patent, it will not be elaborated here. For details, please refer to the following formula and Figure 5 .

[0117] x2=Reshape(x1)

[0118] hw=Conv1×1(AdaptivePool(x2),AdaptivePool(x2))

[0119] x ^ =matmul(Reshape(Conv3×3(x2)),Softmax(GroupNorm(Split(hw))))

[0120] Among them, the Reshape() operation is used to reduce the number of channels of the feature map to one-eighth of the original; the AdaptivePool() operation is used to adjust the size of the feature map while keeping the number of channels unchanged; the GroupNorm() operation is used to group and normalize the feature map channels; the Softmax() operation is used to map the values ​​of the feature map to the range of (0,1); and the Matmul() operation is used to perform matrix multiplication of two three-dimensional features.

[0121] Finally, the two branches of the PEM submodule are multiplied to reshape the size of the output feature map to the original feature map size before entering the module.

[0122] The labeled data set is input into the pallet detection model for training, and the performance and accuracy of the pallet detection model are improved by adjusting the parameters of the pallet detection model and optimizing the algorithm. The trained model is converted into an open neural network exchange format (such as ONNX, TensorFlow Lite, etc.) so that it can be deployed and called on different platforms and environments, thus achieving flexible application and migration of the pallet detection model.

[0123] It should be noted that when the pallet detection model predicts the location information of each pallet in the color image, multiple target boxes are generated, each containing a pallet. In order to filter out the most accurate location of each pallet, these target boxes need to be subjected to non-maximum suppression. This process aims to retain the target box with the highest confidence on each pallet, eliminate overlapping target boxes on the same pallet, and ensure that each pallet corresponds to only one target box. Finally, the pallet detection model outputs the location information of each pallet. See Figure 6 , Figure 6 A schematic diagram of pallet target detection provided for the present application. Among them, pallet 1 is located in target frame 1, and pallet 2 is located in target frame 2. The position information of each pallet in the color image can be the position of the target frame corresponding to each pallet. For example, the coordinates of the lower left corner of the target frame, the height of the target frame, and the width of the target frame are used as the position information of each pallet. For another example, the coordinates of the upper right corner of the target frame, the height of the target frame, and the width of the target frame are used as the position information of each pallet. The present application does not specifically limit the form of expression of the position information of each pallet.

[0124] S104. Determine the point cloud data corresponding to each pallet according to the position information of each pallet in the color image.

[0125] According to the position information of each target frame obtained in S103, the point cloud data of each pallet is extracted from the point cloud data converted from the depth image in S102.

[0126] Subsequently, a series of filtering operations are performed on the point cloud data of each extracted pallet, including straight-through filtering, voxel filtering, and outlier filtering. Among them, straight-through filtering refers to filtering the point cloud data in a specific dimension of the point cloud data to retain points within a specific range in the point cloud data; voxel filtering refers to gridding the point cloud data to reduce the density of the point cloud data while retaining the shape characteristics of the pallet; outlier filtering refers to judging whether each point in the point cloud is an outlier based on the distance between each point and the surrounding points. These filtering methods can effectively reduce the amount of point cloud data and reduce the computational burden of subsequent processing of the point cloud data. After the point cloud data corresponding to each pallet is filtered, the target point on the next pallet and the inclination angle of the pallet can be calculated separately, thereby achieving accurate positioning of the pallet.

[0127] S105 . Determine a target point on each pallet and an inclination angle of each pallet according to the point cloud data corresponding to each pallet.

[0128] According to the point cloud data corresponding to each pallet, the process of determining the target point on each pallet and the tilt angle of each pallet is as follows:

[0129] (1) According to the point cloud data corresponding to each pallet, the point cloud data of the target plane in each pallet is determined.

[0130] It should be noted that the target plane of each pallet is the cutting plane when the fork picks up each pallet. Figure 7 , Figure 7 A schematic diagram of a target plane of a tray provided in this application.

[0131] Figure 6 The second pallet in the figure contains three planes, namely the left plane of the pallet, the target plane of the pallet, and the top plane of the pallet. In order to accurately obtain the target plane of each pallet, it is necessary to first segment all the planes of each pallet from the point cloud data of each pallet, and then determine the target plane of each pallet among these planes.

[0132] First, all planes of each pallet are segmented from the point cloud data of each pallet. For example, the RANSAC algorithm can be used to segment all planes of each pallet from the point cloud data of each pallet. Specifically, the RANSAC algorithm randomly selects a certain number of points from the point cloud data of each pallet as initial samples, and then fits a plane model based on these samples. Next, the RANSAC algorithm calculates the distance from all points to the fitting plane, and classifies the points with a distance less than a set threshold as internal points, and the points with a distance greater than the threshold as external points. Then, the RANSAC algorithm iteratively executes the above steps until the best fitting plane model is found, and all points are divided into internal points and external points. Finally, through the RANSAC algorithm, all planes of each pallet can be effectively segmented from the point cloud data of each pallet, and the points on each segmented plane can be saved to provide a basis for subsequent target plane determination and analysis. This application does not specifically limit the method for segmenting all planes of each pallet from the point cloud data of each pallet.

[0133] Then, the target plane of each pallet is determined among all the planes of each pallet. Specifically, firstly, the points on all the planes on each segmented pallet are merged into the point cloud to obtain the point cloud data of all the planes of each pallet, and the centroid of the point cloud data of all the planes of each pallet is calculated; then, the centroid of the point cloud data of each plane in each pallet is calculated; finally, the score of each plane in each pallet is calculated, and the plane with the largest score is used as the target plane of each pallet, thereby determining the point cloud data of the target plane in each pallet.

[0134] It should be noted that the score of each plane in each pallet is determined based on the number of points in the point cloud data of each plane in each pallet and the distance from the centroid of the point cloud data of each plane in each pallet to the centroid of the point cloud data of all planes of each pallet.

[0135] The calculation of the center of mass is as follows:

[0136] Assume that there are N points in the point cloud data, and the three-dimensional coordinates of each point are p i =(x i ,y i ,z i ), where i is the index. The coordinates of the center of mass C of the point cloud data (x c ,y c ,z c ) is calculated as follows:

[0137]

[0138]

[0139]

[0140] The score for each plane in each tray is calculated as follows:

[0141]

[0142] Among them, max_points refers to the maximum number of points in the point cloud data of each plane in each pallet; plane_points_number refers to the number of points in the point cloud data of the current plane in each pallet; max_distance refers to the maximum distance from the centroid of the point cloud data of each plane in each pallet to the centroid of the point cloud data of all planes of each pallet; lane_centroid_distance refers to the distance from the centroid of the point cloud data of the current plane in each pallet to the centroid of the point cloud data of all planes of each pallet.

[0143] (2) According to the point cloud data of the target plane in each pallet, point cloud data of the target horizontal line, point cloud data of the first target vertical line, and point cloud data of the second target vertical line on the target plane are determined respectively.

[0144] After determining the point cloud data of the target plane in each tray, the boundary points of the target plane are extracted, thereby obtaining the point cloud data of the target plane including only the boundary points. Figure 8 , Figure 8 A schematic diagram of point cloud data of a target plane including only boundary points provided for this application.

[0145] According to the point cloud data of the target plane including only the boundary points, the point cloud data of the target horizontal line on the target plane is determined, wherein the target horizontal line is the uppermost edge line on the target plane. Fig. 9 , Fig. 9 A schematic diagram of point cloud data of a target horizontal line on a target plane provided by the present application. The point cloud data of the target plane including only boundary points is segmented into horizontal line segments to obtain point cloud data of multiple horizontal line segments. Fig. 9 Six horizontal line segments are segmented, and it can be seen that the values ​​of the six horizontal line segments in the vertical direction are different, and the value of the target horizontal line in the vertical direction is significantly greater than that of the other line segments. Therefore, the point cloud data of the target horizontal line on the target plane can be determined based on the value in the vertical direction.

[0146] In another implementation, the target horizontal line on the target plane is determined according to the magnitude of the values ​​of the middle points of the plurality of line segments and the value of the centroid of the target plane in the vertical direction.

[0147] It should be noted that this application does not specifically limit the method of determining the point cloud data of the target horizontal line on the target plane.

[0148] Based on the point cloud data of the target plane that only includes boundary points, the point cloud data of the first target vertical line and the point cloud data of the second target vertical line on the target plane are determined. Specifically, the point cloud data of the target plane that only includes boundary points is segmented into vertical line segments to obtain point cloud data of multiple vertical line segments. Since we only want to extract the two vertical edge line segments of the middle support column of the pallet on the target plane, the width and height of the point cloud data of the support column of the pallet are calculated, and the two vertical edge line segments of the middle support column of the pallet on the target plane are intercepted, thereby determining the point cloud data of the first target vertical line and the point cloud data of the second target vertical line on the target plane. See Fig.10 , Fig.10A schematic diagram of determining a first target vertical line and a second target vertical line on a target plane provided by the present application, wherein the first target vertical line is the left edge line of the middle support column of the pallet on the target plane, and the second target vertical line is the right edge line of the middle support column of the pallet on the target plane.

[0149] It should be noted that the point cloud data of the first target vertical line and the point cloud data of the second target vertical line on the target plane can also be determined by other methods. This application does not make any specific limitations on the method of determining the point cloud data of the first target vertical line and the point cloud data of the second target vertical line on the target plane.

[0150] (3) Determine the target point on each pallet and the inclination angle of each pallet based on the target horizontal line, the first target vertical line, and the second target vertical line on the target plane of each pallet.

[0151] The point cloud data of the target horizontal line on the target plane is mapped to the ground, and these points are fitted to obtain the target horizontal line; the point cloud data of the first target vertical line is mapped to the ground, and these points are fitted to obtain the first target vertical line; the point cloud data of the second target vertical line is mapped to the ground, and these points are fitted to obtain the second target vertical line.

[0152] According to the target horizontal line, the first target vertical line and the second target vertical line on the target plane of each pallet, the first reference point, the second reference point and the target point on the target plane of each pallet are determined. Fig.11 , Fig.11 A schematic diagram of the first reference point, second reference point and target point provided in this application.

[0153] Determine a first reference point on the target plane of each pallet according to the target horizontal line and the first target vertical line on the target plane of each pallet, wherein the first reference point on the target plane of each pallet is an intersection of the target horizontal line and the first target vertical line on the target plane of each pallet;

[0154] Determine a second reference point on the target plane of each pallet according to the target horizontal line and the second target vertical line on the target plane of each pallet, wherein the second reference point on the target plane of each pallet is an intersection point of the target horizontal line and the second target vertical line on the target plane of each pallet;

[0155] According to the first reference point on the target plane of each pallet and the second reference point on the target plane of each pallet, the target point on each pallet is determined, and the target point on each pallet is the midpoint of the line segment where the first reference point on the target plane of each pallet and the second reference point on the target plane of each pallet are located.

[0156] In one implementation, the vertical value of the target point on each pallet can also be obtained by subtracting a fixed value from the vertical value of the first reference point or the second reference point. This application does not specifically limit the vertical value of the target point on each pallet.

[0157] The tilt angle of each pallet is the angle between the target horizontal line on the target plane of each pallet and the horizontal direction. Assuming that the linear equation of the target horizontal line is Ax+By+Cz=D, the calculation formula for the tilt angle of each pallet is as follows:

[0158]

[0159] Among them, arctan() is the inverse tangent function.

[0160] S106. Obtain the actual position of each pallet according to the target point on each pallet and the inclination angle of each pallet.

[0161] After the AGV obtains the point cloud coordinates of the target point on each pallet, path planning is required to ensure that the AGV can accurately move to the target position to perform the fork operation on the pallet.

[0162] The path planning process can be divided into the following steps:

[0163] First, the AGV needs to convert the point cloud coordinates of the target point on each pallet to the AGV coordinate system. This involves inverse coordinate transformation, which uses the camera's internal and external parameter information to convert the point cloud coordinates of the target point on each pallet from the camera coordinate system to the AGV coordinate system.

[0164] Then, based on the target point on each pallet in the AGV coordinate system and the tilt angle of each pallet, a path planning algorithm (such as A* algorithm, Dijkstra algorithm, etc.) is used to calculate the path that the AGV needs to take along the way. Path planning takes into account factors such as map information, obstacle avoidance, and the shortest path to ensure that the AGV reaches the target location safely and efficiently.

[0165] Finally, according to the path obtained by path planning, the AGV controls the movement of the AGV through motion control algorithms (such as PID control, trajectory tracking control, etc.), moves to the target position along the planned path, and picks up the pallet. Fig.12 , Fig.12 A schematic diagram of the AGV provided for this application moving to the target location along a planned path.

[0166] Optionally, after obtaining the target point on each pallet and the tilt angle of each pallet, determine whether there is cargo on the pallet. Specifically, first, extract the point cloud data within a certain range above the target point on each pallet, filter the extracted point cloud data, and remove some outliers; then, perform Euclidean clustering on the processed point cloud data, and segment the most dense point cloud area by judging the distance between each point; finally, determine whether there is cargo on each pallet based on the distance between the centroid of the most dense area and the target point on each pallet.

[0167] It can be seen that in this embodiment, by converting the pixels in the depth map into a point cloud in three-dimensional space, richer three-dimensional information can be obtained. According to the position information of each pallet in the color image of one or more pallets, the point cloud data corresponding to each pallet is determined, and then the target point on each pallet and the inclination angle of each pallet are determined. The point cloud data can accurately reflect the shape and position of the pallet, which helps to accurately determine the position of the pallet. In addition, the point cloud data can avoid noise and blur, provide clearer and more accurate information, and thus improve the accuracy and reliability of pallet detection.

[0168] See also Fig.13 , Fig.13 A schematic flow chart of another pallet detection method provided in the present application, wherein the method includes but is not limited to the description of the following contents.

[0169] S201 : Acquire a color image and a depth image at the same time, wherein the color image and the depth image respectively include one or more trays.

[0170] This step is the same as step S101 and will not be repeated here for the sake of brevity of the patent.

[0171] S202: Determine the position information of each tray in the depth image according to the position information of each tray in the color image of the one or more trays.

[0172] The color image is input to the pallet detection model in S103, and the position information of each pallet in the color image is output. According to the position information of each target frame obtained in S103, the position information of each pallet in the depth image is determined.

[0173] It should be noted that the position information of each tray in the depth image is the same as the position information of each tray in the depth image in S103, and for the sake of simplicity of the patent, it will not be repeated here.

[0174] S203 . Determine a target point on each pallet and a tilt angle of each pallet according to the position information of each pallet in the depth image.

[0175] According to the position information of each pallet in the depth image, each pallet is extracted from the corresponding depth image. Next, the target point on the pallet and the tilt angle of the pallet are calculated for each extracted pallet, so as to achieve accurate positioning of the pallet.

[0176] (1) Determine the target plane in each tray.

[0177] First, all planes of each pallet are segmented from the depth map of each pallet. The method is referred to S105 and will not be described here for the sake of brevity. Then, the target plane of each pallet is determined among all the planes of each pallet. The method can be determined in S105 according to the number of pixels in each plane of each pallet and the distance from the center of mass of each plane of each pallet to the center of mass of all the planes of each pallet, or it can be determined by the area and normal vector of the plane. Finally, the depth map corresponding to the target plane of each pallet is obtained. This application does not specifically limit the method for determining the target plane of each pallet among all the planes of each pallet.

[0178] (2) Determine the target horizontal line, the first target vertical line, and the second target vertical line on the target plane.

[0179] First, the depth map corresponding to the target plane of each pallet is preprocessed, including noise removal, smoothing, and edge enhancement operations, so as to better extract the contour of the target plane of each pallet. Then, a contour extraction algorithm, such as the findContours function in OpenCV, is used to extract the contour of each pallet and obtain the boundary point set of the target plane of each pallet. Finally, the extracted contour is optimized and processed, and smoothing, closing, fitting, etc. can be performed to obtain a more accurate and continuous contour.

[0180] The target horizontal line, the first target vertical line, and the second target vertical line on the target plane are determined according to the contour of the target plane of each pallet. The definition and determination method of the target horizontal line, the first target vertical line, and the second target vertical line on the target plane have been described in S105, and for the sake of simplicity of the patent, they will not be repeated here.

[0181] (3) Determine the target point on each pallet and the inclination angle of each pallet based on the target horizontal line, the first target vertical line, and the second target vertical line on the target plane of each pallet.

[0182] The method for determining the target point of each pallet and the tilt angle of each pallet has been described in S105 and will not be repeated here for the sake of simplicity of the patent.

[0183] S204: Obtain the actual position of each pallet according to the target point on each pallet and the tilt angle of each pallet.

[0184] The method of obtaining the actual position of each pallet according to the target point on each pallet and the tilt angle of each pallet has been described in S106 and will not be repeated here. The AGV picks up the pallet according to the actual position of each pallet.

[0185] It can be seen that in the embodiment of the present application, a color image and a depth image are acquired at the same time, wherein the color image provides visual features such as the color and texture of the pallet, while the depth image provides the distance information between the pallet and the camera. The combined use of these two images can provide a more comprehensive understanding of the characteristics of the pallet, which helps to accurately determine the position of the pallet. In addition, the target point determines the specific position of the pallet in space and can be used as a reference point for positioning. By accurately determining the position of the target point, the error in the positioning process can be reduced and the accuracy of calculating the actual position is improved. The tilt angle describes the degree of inclination of the pallet relative to the horizontal plane, which helps to determine the direction of rotation of the pallet, thereby making the calculated actual position more accurate. Taking into account the information of the target point and the tilt angle, the accuracy and reliability of pallet positioning can be improved.

[0186] This application provides a pallet detection device, see Fig.14 , Fig.14 This is a structural schematic diagram of a pallet detection device 100 provided in the present application. The pallet detection device 100 includes:

[0187] An acquisition module 110, configured to acquire a color image and a depth image at the same time, wherein the color image and the depth image respectively include one or more trays;

[0188] A determination module 120, configured to determine position information of each tray in the depth image according to position information of each tray in the color image of the one or more trays;

[0189] The determination module 120 is further used to determine the target point on each pallet and the tilt angle of each pallet according to the position information of each pallet in the depth image;

[0190] The actual position determination module 130 is used to obtain the actual position of each pallet according to the target point on each pallet and the tilt angle of each pallet.

[0191] In a possible implementation, the pallet detection device 100 further includes a conversion module 140, and the conversion module 140 is used to convert the depth image into point cloud data;

[0192] The determination module 120 is used to determine the point cloud data corresponding to each pallet according to the position information of each pallet in the color image of the one or more pallets;

[0193] The determination module 120 is further used to determine the target point on each pallet and the tilt angle of each pallet according to the point cloud data corresponding to each pallet.

[0194] In a possible implementation, the determination module 120 is used to determine the point cloud data of the target plane in each pallet according to the point cloud data corresponding to each pallet, and the target plane of each pallet is the cutting plane when forking each pallet;

[0195] The determination module 120 is further used to determine, according to the point cloud data of the target plane in each pallet, the point cloud data of the target horizontal line, the point cloud data of the first target vertical line, and the point cloud data of the second target vertical line on the target plane, respectively, wherein the target horizontal line is the uppermost edge line on the target plane, the first target vertical line is the left edge line of the middle support column of the pallet on the target plane, and the second target vertical line is the right edge line of the middle support column of the pallet on the target plane;

[0196] The determination module 120 is further used to determine a target point on each pallet and an inclination angle of each pallet according to a target horizontal line, a first target vertical line and a second target vertical line on a target plane of each pallet, the target point being located on the target plane.

[0197] In a possible implementation, the determination module 120 is used to determine the point cloud data of the target plane in each pallet based on the number of points in the point cloud data corresponding to each plane in each pallet and the distance from the center of mass of the point cloud data corresponding to each plane in each pallet to the center of mass of the point cloud data corresponding to each pallet.

[0198] In a possible implementation, the determination module 120 is used to determine a first reference point on the target plane of each pallet according to a target horizontal line and a first target vertical line on the target plane of each pallet, wherein the first reference point on the target plane of each pallet is an intersection of the target horizontal line and the first target vertical line on the target plane of each pallet;

[0199] The determination module 120 is further configured to determine a second reference point on the target plane of each pallet according to the target horizontal line and the second target vertical line on the target plane of each pallet, wherein the second reference point on the target plane of each pallet is an intersection point of the target horizontal line and the second target vertical line on the target plane of each pallet;

[0200] The determination module 120 is also used to determine the target point on each pallet based on the first reference point on the target plane of each pallet and the second reference point on the target plane of each pallet, and the target point on each pallet is the midpoint of the line segment where the first reference point on the target plane of each pallet and the second reference point on the target plane of each pallet are located.

[0201] In a possible implementation, the inclination angle of each pallet is an angle formed between a target horizontal line on a target plane of each pallet and a horizontal direction.

[0202] In a possible implementation, the pallet detection device 100 further includes an extraction module 150, which is used to extract point cloud data within a certain range above the target point on each pallet, and determine the most dense area in the point cloud data within a certain range above the target point on each pallet;

[0203] The determination module 120 is used to determine whether there is cargo on each pallet according to the distance between the centroid of the most densely populated area and the target point on each pallet.

[0204] Fig.14 The various functional modules in Figures 3 to 12 The steps of the method embodiment, or, implementing Fig.13 For details of the steps of the method embodiment, please refer to Figures 1 to 12 For a description of the relevant contents in the method embodiment, or, see Fig.13 The description of the relevant contents in the method embodiment will not be repeated here for the sake of brevity of the specification.

[0205] The present application also provides a computing device, see Fig.15 , Fig.15 A schematic diagram of a computing device 200 provided in this application, the computing device is used to implement Figures 3 to 12 The method embodiment described, or, implementing Fig.13 In the method embodiment, the computing device 200 includes: a processor 210, a communication interface 220, and a memory 230. The processor 210, the communication interface 220, and the memory 230 may be interconnected via an internal bus 240, or may communicate via other means such as wireless transmission.

[0206] Taking the connection through bus 240 as an example, bus 240 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. Bus 240 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.15 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0207] The processor 210 may be composed of at least one general-purpose processor, such as a CPU, or a combination of a CPU and a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The processor 210 executes various types of digital storage instructions, such as software or firmware programs stored in the memory 230, which enables the computing device 200 to provide a wide variety of services.

[0208] The memory 230 is used to store program codes and is controlled by the processor 210 to execute the above Figures 3 to 12 The steps described in the embodiment, or, to perform the above Fig.13 For the steps described in the embodiment, please refer to the relevant description of the embodiment shown above, which will not be elaborated here.

[0209] The memory 230 may include a volatile memory, such as a RAM; the memory 230 may also include a non-volatile memory, such as a ROM, a flash memory; the memory 230 may also include a combination of the above types.

[0210] The communication interface 220 can be a wired interface (such as an Ethernet interface), an internal interface (such as a high-speed serial computer expansion bus (peripheral component interconnect express, PCIE) bus interface), a wired interface (such as an Ethernet interface) or a wireless interface (such as a cellular network interface or a wireless local area network interface) for communicating with other devices or modules.

[0211] The processor 210, the communication interface 220, etc. in the computing device 200 can implement the functions and / or various steps and methods implemented in the above-mentioned various method embodiments, and for the sake of brevity, they are not described here. The acquisition module 110 in the pallet detection device 100 can be located in the communication interface in the computing device 200; the determination module 120, the actual position determination module 130, the conversion module 140 and the extraction module 150 can be located in the processor 210 in the computing device 200.

[0212] Need to explain, Fig.15 This is only a possible implementation of the embodiment of the present application. In practical applications, the computing device may also include more or fewer components, which is not limited here. For the contents not shown or described in the embodiment of the present application, please refer to the relevant description in the embodiment described in the aforementioned method, which will not be repeated here.

[0213] The present application also provides a computer storage medium including program instructions. When the program instructions are executed by an AGV, the AGV executes part or all of the steps described in the above-mentioned pallet detection method embodiment.

[0214] The present application also provides a computer program product, including program instructions. When the program instructions are executed by the AGV, the AGV executes part or all of the steps described in the above-mentioned pallet detection method embodiment.

[0215] The present application also provides a forklift, including a forklift device and a pallet detection device, or including a forklift device and a computing device, so that the AGV performs some or all of the steps described in the above pallet detection method embodiment. In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0216] In the above embodiments, it can be implemented in whole or in part by software, hardware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product may include code. When the computer program product is read and executed by a computer, some or all steps of the method recorded in the above method embodiment may be implemented. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium, or a semiconductor medium, etc.

[0217] The steps in the method of the embodiment of the present application can be adjusted in order, combined or deleted according to actual needs; the units in the device of the embodiment of the present application can be divided, combined or deleted according to actual needs.

[0218] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A pallet detection method, characterized in that: The method comprises: Acquire a color image and a depth image at the same time, wherein the color image and the depth image respectively include one or more pallets; Determine the position information of each tray in the depth image according to the position information of each tray of the one or more trays in the color image; Determine a target point on each pallet and an inclination angle of each pallet according to position information of each pallet in the depth image; The actual position of each pallet is obtained according to the target point on each pallet and the inclination angle of each pallet.

2. The method according to claim 1, characterized in that Before determining the position information of each tray in the depth image according to the position information of each tray of the one or more trays in the color image, the method further includes: Converting the depth image into point cloud data; Accordingly, determining the position information of each pallet in the depth image according to the position information of each pallet in the color image, and determining the target point on each pallet and the inclination angle of each pallet according to the position information of each pallet in the depth image, includes: Determine point cloud data corresponding to each pallet according to position information of each pallet of the one or more pallets in the color image; According to the point cloud data corresponding to each pallet, a target point on each pallet and an inclination angle of each pallet are determined.

3. The method according to claim 2, characterized in that Determining the target point on each pallet and the inclination angle of each pallet according to the point cloud data corresponding to each pallet includes: Determine the point cloud data of the target plane in each pallet according to the point cloud data corresponding to each pallet, wherein the target plane of each pallet is the cutting plane when forking each pallet; According to the point cloud data of the target plane in each pallet, point cloud data of a target horizontal line, point cloud data of a first target vertical line, and point cloud data of a second target vertical line on the target plane are determined respectively, wherein the target horizontal line is the uppermost edge line on the target plane, the first target vertical line is the left edge line of the middle support column of the pallet on the target plane, and the second target vertical line is the right edge line of the middle support column of the pallet on the target plane; A target point on each pallet and an inclination angle of each pallet are determined according to the target horizontal line, the first target vertical line and the second target vertical line on the target plane of each pallet, wherein the target point is located on the target plane.

4. The method according to claim 3, characterized in that The step of determining the point cloud data of the target plane in each pallet according to the point cloud data corresponding to each pallet includes: The point cloud data of the target plane in each pallet is determined according to the number of points in the point cloud data corresponding to each plane in each pallet and the distance from the centroid of the point cloud data corresponding to each plane in each pallet to the centroid of the point cloud data corresponding to each pallet.

5. The method according to claim 3 or 4, characterized in that: Determining the target point on each pallet according to the target horizontal line, the first target vertical line, and the second target vertical line on the target plane of each pallet includes: Determine a first reference point on the target plane of each pallet according to the target horizontal line and the first target vertical line on the target plane of each pallet, wherein the first reference point on the target plane of each pallet is an intersection of the target horizontal line and the first target vertical line on the target plane of each pallet; Determine a second reference point on the target plane of each pallet according to the target horizontal line and the second target vertical line on the target plane of each pallet, wherein the second reference point on the target plane of each pallet is an intersection of the target horizontal line and the second target vertical line on the target plane of each pallet; The target point on each pallet is determined according to the first reference point on the target plane of each pallet and the second reference point on the target plane of each pallet, wherein the target point on each pallet is the midpoint of the line segment where the first reference point on the target plane of each pallet and the second reference point on the target plane of each pallet are located.

6. The method according to claim 5, characterized in that The inclination angle of each tray is an angle formed by the target horizontal line on the target plane of each tray and the horizontal direction.

7. The method according to any one of claims 2 to 6, characterized in that: The method further comprises: Extracting point cloud data within a certain range above the target point on each pallet, and determining the most densely populated area in the point cloud data within the certain range above the target point on each pallet; Whether there is cargo on each pallet is determined according to the distance between the centroid of the most densely populated area and the target point on each pallet.

8. A pallet detection device, characterized in that: The device comprises: An acquisition module, used to acquire a color image and a depth image at the same time, wherein the color image and the depth image respectively include one or more pallets; a determination module, configured to determine position information of each tray in the depth image according to position information of each tray in the one or more trays in the color image; The determination module is further used to determine the target point on each pallet and the inclination angle of each pallet according to the position information of each pallet in the depth image; The actual position determination module is used to obtain the actual position of each pallet according to the target point on each pallet and the inclination angle of each pallet.

9. A computing device, characterized in that The method comprises a memory and a processor, wherein the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The method comprises program instructions, and when the program instructions are executed by a computing device, the computing device is caused to execute the method according to any one of claims 1 to 7.

11. A forklift, characterized in that: The method comprises a fork picking device and the pallet detecting device as claimed in claim 8, or comprises the fork picking device and the computing device as claimed in claim 9.

Images