Pallet posture detection method, device, equipment and storage medium

Abstract

The embodiment of the present invention discloses a method, device, equipment and storage medium for pallet posture detection. The method includes: obtaining a depth image and a color image of at least one pallet; inputting the color image into a pallet detection model to determine at least one set of target corner point coordinates of the pallet, the pallet detection model is obtained based on a preset neural network training; according to at least one set of target corner point coordinates, determining the pallet area of ​​the pallet in the depth image; according to the data in the pallet area of ​​the pallet, determining the posture information of the pallet. The embodiment of the present invention can achieve the purpose of pallet posture detection.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of positioning technology, and in particular to a pallet posture detection method, device, equipment and storage medium. Background Art

[0002] With the rapid development of the logistics industry, pallets have been widely used in production, transportation, warehousing and circulation. As an important loading, unloading, storage and transportation equipment in logistics operations, pallets play a huge role in modern logistics when used with unmanned forklifts. Therefore, posture detection of pallets loaded with goods is an indispensable part of unmanned forklift logistics operations. Therefore, how to detect the posture of pallets has become a problem that needs to be solved urgently. Summary of the invention

[0003] The embodiments of the present invention provide a method, device, equipment and storage medium for detecting the position and posture of a pallet, which can achieve the purpose of detecting the position and posture of a pallet.

[0004] In a first aspect, an embodiment of the present invention provides a method for detecting a pallet posture, comprising:

[0005] acquiring a depth image and a color image of at least one pallet;

[0006] Inputting the color image into a pallet detection model to determine at least one set of target corner point coordinates of the pallet, wherein the pallet detection model is obtained based on a preset neural network training;

[0007] determining a pallet region of the pallet in the depth image according to the at least one set of target corner point coordinates;

[0008] The position and posture information of the pallet is determined according to the data in the pallet area of ​​the pallet.

[0009] In a second aspect, an embodiment of the present invention further provides a pallet posture detection device, comprising:

[0010] An image acquisition module, used to acquire a depth image and a color image of at least one tray;

[0011] A corner point determination module, used for inputting the color image into a pallet detection model to determine at least one set of target corner point coordinates of the pallet, wherein the pallet detection model is obtained based on a preset neural network training;

[0012] an area determination module, configured to determine a pallet area of ​​the pallet in the depth image according to the at least one set of target corner point coordinates;

[0013] The posture determination module is used to determine the posture information of the pallet according to the data in the pallet area of ​​the pallet.

[0014] In a third aspect, an embodiment of the present invention further provides a pallet posture detection device, comprising:

[0015] An image acquisition device for acquiring a depth image and a color image of at least one pallet;

[0016] one or more processors;

[0017] a storage device for storing one or more programs,

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the pallet posture detection methods described in the embodiments of the present invention.

[0019] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method for detecting a pallet posture described in any one of the embodiments of the present invention is implemented.

[0020] The technical solution disclosed in the embodiment of the present invention has the following beneficial effects:

[0021] By acquiring a depth image and a color image of at least one pallet, the color image is input into a pallet detection model, at least one set of target corner coordinates of the pallet is determined, and the pallet area of ​​the pallet is determined in the depth image according to the at least one set of target corner coordinates, and then the pallet posture information is determined according to the data in the pallet area of ​​the pallet. The embodiment of the present invention detects the posture of the pallet by combining the color image and the depth image, which not only achieves the purpose of pallet posture detection, but also can fully utilize the rich semantic information of the color image and reduce the processing complexity of the depth image, thereby providing conditions for improving the speed of pallet posture detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of a pallet posture detection method provided in the first embodiment of the present invention;

[0023] Figure 2 It is a flow chart of a pallet posture detection method provided in the second embodiment of the present invention;

[0024] Figure 3 It is a flow chart of a pallet posture detection method provided in Embodiment 3 of the present invention;

[0025] Figure 4A It is a schematic diagram of establishing a collection device coordinate system based on a tray-based deployment method provided in the third embodiment of the present invention;

[0026] Figure 4Bis a schematic diagram of determining a tray area of ​​a tray provided in Embodiment 3 of the present invention;

[0027] Figure 4C is a schematic diagram of determining a tray area of ​​multiple trays provided in Embodiment 3 of the present invention;

[0028] Figure 4D This is a schematic diagram of projecting pixel points in the XY plane and performing linear fitting on the three-dimensional coordinate points corresponding to all pixel points to obtain a fitting straight line, as provided in the third embodiment of the present invention;

[0029] Figure 5 It is a schematic diagram of a process of generating a pallet detection model provided by a fourth embodiment of the present invention;

[0030] Figure 6 is another schematic diagram of a process for generating a pallet detection model provided by a fourth embodiment of the present invention;

[0031] Figure 7 is a schematic diagram of a process for generating a pallet detection model provided by the fourth embodiment of the present invention;

[0032] Fig. 8A is a schematic diagram of a data set including a plurality of color images of at least one tray provided in a fourth embodiment of the present invention;

[0033] Figure 8B is a schematic diagram of obtaining an enhanced data set after enhancing a plurality of color images of at least one tray in a data set provided by a fourth embodiment of the present invention;

[0034] Fig. 9 It is a structural schematic diagram of a pallet posture detection device provided in Embodiment 5 of the present invention;

[0035] Fig.10 It is a structural schematic diagram of a pallet posture detection device provided in Example 6 of the present invention. DETAILED DESCRIPTION

[0036] The embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the embodiments of the present invention, rather than to limit the embodiments of the present invention. It is also necessary to explain that, for ease of description, only parts related to the embodiments of the present invention are shown in the accompanying drawings, rather than all structures.

[0037] The following describes in detail the pallet posture detection method, device, equipment and storage medium provided by the embodiments of the present invention in conjunction with the accompanying drawings.

[0038] Embodiment 1

[0039] Figure 1This is a flow chart of a pallet posture detection method provided in the first embodiment of the present invention. This embodiment is applicable to the scene of detecting the posture of a pallet. The method can be executed by a pallet posture detection device, which can be composed of hardware and / or software and can be integrated into a pallet posture detection device. The pallet posture detection device can be any device that can move autonomously and perform functions such as pallet handling, such as an unmanned forklift or a handling robot. The method specifically includes the following:

[0040] S101, acquiring a depth image and a color image of at least one pallet.

[0041] In an embodiment of the present invention, an image acquisition device for acquiring depth images and color images is provided on the pallet posture detection device. The image acquisition device may be a depth (Time Of Flight, referred to as TOF) camera or a stereo camera. In other words, the processor on the pallet posture detection device can acquire a depth image and a color image of at least one pallet by controlling the above-mentioned image acquisition device. Among them, the acquired depth image and color image refer to two different types of images acquired at the same time, that is, the image acquisition device acquires image data of a preset area, and the preset area includes at least one pallet, and one frame of image data corresponds to a depth image and a color image.

[0042] Specifically, when the pallet posture detection device moves to an area where pallets are placed, the depth camera or stereo camera can be controlled to obtain a depth image and a color image of at least one pallet in the area, wherein the obtained depth image and color image contain at least one pallet.

[0043] Furthermore, in order to improve the efficiency of image acquisition, the present embodiment may also control the image acquisition device to start shooting operation only when a pallet is detected and the pallet is within the field of view of the depth image or stereo camera to obtain a depth image and a color image of at least one pallet.

[0044] S102, inputting the color image into a pallet detection model to determine at least one set of target corner point coordinates of the pallet, wherein the pallet detection model is obtained based on a preset neural network training.

[0045] The target corner point coordinates refer to the vertex coordinates used to detect the position and posture of the pallet. In this embodiment, a set of target corner point coordinates is preferably a set of vertex coordinates located at the diagonal position of the pallet. For example, a set of target corner point coordinates can be the upper left corner point coordinates and the lower right corner point coordinates of the pallet, or can also be the lower left corner point coordinates and the upper right corner point coordinates of the pallet, which are not specifically limited here.

[0046] It can be understood that when at least one set of target corner point coordinates is a group, then the set of target corner point coordinates may be the upper left corner point coordinates and the lower right corner point coordinates of the pallet, or may also be the lower left corner point coordinates and the upper right corner point coordinates of the pallet; when at least one set of target corner point coordinates is multiple sets (for example, two sets), then the first set of target corner point coordinates may be the upper left corner point coordinates and the lower right corner point coordinates of the pallet, and the second set of target corner point coordinates may be the lower left corner point coordinates and the upper right corner point coordinates of the pallet; or, the first set of target corner point coordinates may be the lower left corner point coordinates and the upper right corner point coordinates of the pallet, and the second set of target corner point coordinates may be the upper left corner point coordinates and the lower right corner point coordinates of the pallet, and so on.

[0047] Optionally, after acquiring the color image and the depth image of at least one pallet, the color image is input into the pallet detection model so as to process the color image through the pallet detection model to determine at least one set of target corner point coordinates of each pallet.

[0048] It should be noted that the generation process of the tray detection model based on the preset neural network training in this embodiment will be described in detail in the following examples, and will not be elaborated on here.

[0049] S103: Determine a tray area of ​​the tray in the depth image according to the at least one set of target corner point coordinates.

[0050] Optionally, after determining at least one set of target corner point coordinates for each pallet, the rectangular area corresponding to each pallet can be determined in the corresponding acquired depth image according to the at least one set of target corner point coordinates corresponding to each pallet, and the rectangular area corresponding to each pallet can be determined as the pallet area of ​​the pallet.

[0051] It should be noted that when the number of at least one set of target corner point coordinates for each pallet is one group, a rectangular area can be determined in the depth image based on the set of target corner point coordinates; when the number of at least one set of target corner point coordinates for each pallet is two groups, two areas can be determined in the depth image based on the two sets of target corner point coordinates. Since the four corner points of a pallet are unique, the areas determined based on the two sets of target corner point coordinates should belong to the same area. That is, the pallet area of ​​the pallet can be fixedly and uniquely determined by at least one set of target corner point coordinates.

[0052] For example, if at least one set of target corner point coordinates of pallet Q is the upper left corner point coordinates Z1 (x1, y1) and the lower right corner point coordinates Z2 (x2, y2), then first, in the depth image, the target points Z1' (x1, y1) and Z2' (x2, y2) corresponding to Z1 (x1, y1) and Z2 (x2, y2) are determined respectively, so as to calculate Δx = |x1-x2| and Δy = |y1-y2| based on Z1' (x1, y1) and Z2' (x2, y2), and then use Z1' (x1, y1), Z2' (x2, y2), Δx = |x1-x2| and Δy = |y1-y2| to calculate the upper right corner point coordinates Z3' (x3, y3) and the lower left corner point coordinates Z4' (x4, y4) of pallet Q in the depth image. Thus, the tray area of ​​tray Q is determined according to Z1'(x1, y1), Z2'(x2, y2), Z3'(x3, y3) and Z4'(x4, y4), wherein x3=x1+Δx=x2, y3=y2+Δy=y1, x4=x2-Δx=x1 and y4=y1-Δy=y2.

[0053] S104: Determine the position information of the pallet according to the data in the pallet area of ​​the pallet.

[0054] The posture information includes position information and posture information.

[0055] It is worth noting that due to the working characteristics of the pallet posture detection device, the posture of the pallet has only one degree of freedom, namely the yaw angle. In other words, determining the posture information of the pallet in the embodiment of the present invention is to determine the yaw angle of the pallet.

[0056] Specifically, this embodiment can determine the position information and posture information of each pallet based on the data in the pallet area of ​​each pallet. Further, based on the position information and posture information of each pallet, the position and posture information of each pallet can be determined.

[0057] In specific implementation, the position information and posture information of the corresponding pallet can be determined based on the data in the pallet area of ​​each pallet. When determining the position information of the pallet, the coordinates of two vertices diagonally opposite to each other can be obtained from the data in the pallet area of ​​the pallet, and then the coordinates of the pallet center point of each pallet can be calculated based on the three-dimensional coordinates corresponding to the above two vertex coordinates, and the coordinates of the pallet center point can be determined as the position information of the pallet.

[0058] When determining the position and posture information of the pallet, multiple pixel points can be selected from the data in the pallet area of ​​each pallet, and the selected multiple pixel points can be projected onto the posture detection surface of the pallet, and the three-dimensional coordinates corresponding to all the pixel points projected onto the posture detection surface can be linearly fitted to obtain fitting results, and then the posture information of the pallet can be determined based on the fitting results. Among them, different fitting algorithms can be used to implement the linear fitting of the three-dimensional coordinates corresponding to the pixel points, for example, the optional fitting algorithm is the RANSAC algorithm, etc., which is not specifically limited here.

[0059] The technical solution provided by the embodiment of the present invention obtains a depth image and a color image of at least one pallet, inputs the color image into the pallet detection model, determines at least one set of target corner point coordinates of the pallet, and determines the pallet area of ​​the pallet in the depth image based on the at least one set of target corner point coordinates, and then determines the pallet posture information based on the data in the pallet area of ​​the pallet. The embodiment of the present invention detects the posture of the pallet by combining the color image and the depth image, which not only achieves the purpose of pallet posture detection, but also can fully utilize the rich semantic information of the color image and reduce the processing complexity of the depth image, thereby providing conditions for improving the speed of pallet posture detection.

[0060] Embodiment 2

[0061] It can be seen from the above introduction that the embodiment of the present invention determines the position information of the pallet based on the acquired pallet depth image and pallet color image. Figure 2 Based on the above embodiment, the pallet posture detection method provided in the embodiment of the present invention is further optimized. Figure 2 As shown, the method specifically includes:

[0062] S201, acquiring a depth image and a color image of at least one pallet, and aligning the depth image and the color image.

[0063] Generally, after obtaining a depth image and a color image of at least one pallet through a depth camera or a stereo camera, the depth image and the color image may not be registered. If the unregistered depth image and color image are used to determine the position information of the pallet, it is easy to cause the determined position to be inaccurate and have errors, thereby affecting the accuracy of subsequent pallet handling and other effects.

[0064] To this end, after acquiring the depth image and color image of at least one pallet, the present embodiment may register the depth image and the color image. In specific implementation, the depth image and the color image may be registered by calling the image registration program interface (Application Programming Interface, referred to as API) based on the image registration program corresponding to the program interface, that is, the depth image and the color image are aligned so that there is a one-to-one correspondence between the pixels in the processed depth image and the color image.

[0065] It should be noted that the depth image of at least one tray obtained in this embodiment may contain holes and edge burrs and other noises. Therefore, before aligning the depth image and the color image, this embodiment can optionally perform denoising on the depth image. Exemplarily, the depth image can be smoothed based on time and / or space filtering to reduce the adverse effects of noises such as holes and edge burrs.

[0066] S202, inputting the processed color image into the pallet detection model to determine at least one set of target corner point coordinates of the pallet, wherein the pallet detection model is obtained based on a preset neural network training.

[0067] In this embodiment, a set of target corner point coordinates may optionally include a first corner point coordinate and a second corner point coordinate. The first corner point coordinate and the second corner point coordinate are preferably a set of corner point coordinates of the pallet that are diagonally opposite to each other. For example, the upper left corner point coordinate and the lower right corner point coordinate of the pallet, or the lower left corner point coordinate and the upper right corner point coordinate of the pallet, which are not specifically limited here.

[0068] Optionally, the color image after the alignment process is input into a pallet detection model so that the pallet detection model can process the input color image to determine at least one set of target corner point coordinates of each pallet.

[0069] Exemplarily, in this embodiment, at least one set of target corner point coordinates of the pallet is determined by the pallet detection model, which can be expressed as Q i [(T0, B0, C0); (T1, B1, C1); …; (Tj, Bj, Cj)] where i represents the i-th pallet, i is an integer greater than or equal to 1; Tj represents the coordinates of the first corner point in the j-th group on the i-th pallet; Bj represents the coordinates of the second corner point in the j-th group on the i-th pallet; Cj represents the confidence of the coordinates of the first corner point and the second corner point in the j-th group on the i-th pallet.

[0070] S203: Filter at least one set of target corner point coordinates of the tray.

[0071] Specifically, the present embodiment may adopt different methods to filter at least one set of target corner coordinates of the pallet. Exemplarily, at least one set of target corner coordinates of the pallet may be filtered according to a confidence threshold and / or a pallet width-to-height ratio threshold to filter out erroneous or redundant corner coordinates. The confidence threshold and the pallet width-to-height ratio threshold may be flexibly set based on experience.

[0072] In specific implementation, a pre-configured confidence threshold and / or pallet width-to-height ratio threshold may be obtained from a configuration file, and at least one set of target corner coordinates of a pallet may be filtered based on the confidence threshold and / or pallet width-to-height ratio threshold to filter out erroneous or redundant corner coordinates. Filtering at least one set of target corner coordinates of a pallet based on the confidence threshold and / or pallet width-to-height ratio threshold may include the following situations:

[0073] In case 1, at least one set of target corner point coordinates of the pallet is filtered according to a confidence threshold to filter out erroneous or redundant corner point coordinates.

[0074] Specifically, the confidence in each group of target corner point coordinates of each tray can be compared with the confidence threshold to determine which groups of target corner point coordinates in each tray have confidence levels lower than the confidence threshold. Then, all groups of target corner point coordinates lower than the confidence threshold are filtered out.

[0075] For example, assuming that the confidence threshold is C, pallet Q has four groups of target corner coordinates, namely M0 (T0, B0, C0), M1 (T1, B1, C1), M2 (T2, B2, C2) and M3 (T3, B3, C3), where T0, T1, T2 and T3 are the first corner coordinates in each group of target corner coordinates, B0, B1, B2 and B3 are the second corner coordinates in each group of target corner coordinates, and C0, C1, C2 and C3 are the confidences of each group of target corner coordinates. Then, by comparing the confidences of the four groups of target corner coordinates: C0, C1, C2 and C3 with C respectively, if C1 and C2 are both lower than C, the second group of target corner coordinates: M1 (T1, B1, C1) and the third group of target corner coordinates: M2 (T2, B2, C2) are filtered out.

[0076] In the second case, at least one set of target corner point coordinates of the pallet is filtered according to the pallet width-to-height ratio threshold to filter out erroneous or redundant corner point coordinates.

[0077] Specifically, the width-to-height ratio corresponding to each set of target corner point coordinates of each pallet can be calculated based on the first corner point coordinates and the second corner point coordinates in each set of target corner point coordinates. Then, the width-to-height ratio corresponding to each set of target corner point coordinates is compared with the pallet width-to-height ratio threshold, so as to determine which sets of target corner point coordinates in each pallet have width-to-height ratios that are not within the pallet width-to-height ratio threshold. Then, all sets of target corner point coordinates that are not within the pallet width-to-height ratio threshold are filtered out.

[0078] In case three, at least one set of target corner point coordinates of the pallet is filtered according to the confidence threshold and the pallet width-to-height ratio threshold to filter out erroneous or redundant corner point coordinates.

[0079] Specifically, after the at least one set of target corner point coordinates of each pallet is initially filtered based on the confidence threshold, there may still be erroneous or redundant corner point coordinates. Then, this embodiment may perform secondary filtering on the at least one set of target corner point coordinates after the initial filtering based on the pallet width-to-height ratio threshold to screen out the valid set of target corner point coordinates for each pallet. That is, this embodiment may first perform primary filtering on the at least one set of target corner point coordinates of each pallet based on the confidence threshold, and then perform secondary filtering on the at least one set of target corner point coordinates of each pallet after the initial filtering based on the pallet width-to-height ratio threshold to screen out the valid set of target corner point coordinates for each pallet.

[0080] In specific implementation, a width-to-height ratio can be calculated based on the first corner point coordinates and the second corner point coordinates of each group of target corner point coordinates in at least one group of target corner point coordinates after preliminary filtering of each pallet, and then the width-to-height ratio is compared with the pallet width-to-height ratio threshold, to determine which groups of target corner point coordinates in at least one group of target corner point coordinates after preliminary filtering of each pallet have width-to-height ratios that are not within the numerical range corresponding to the pallet width-to-height ratio threshold, and filter all groups of target corner point coordinates that are not within the numerical range corresponding to the pallet width-to-height ratio threshold, thereby obtaining the valid group of target corner point coordinates of each pallet.

[0081] Continuing with the example in the above case 1, assuming that the pallet width-to-height ratio threshold is P, and the coordinates of at least one set of target corner points after preliminary filtering of pallet Q are M0(T0, B0, C0) and M3(T3, B3, C3), then based on T0(X T0 , Y T0 ) and B0(X B0 ,Y B0 ), and M3 in T3 (X T3 , Y T3 ) and B3(X B3 ,Y B3 ), and calculate the width-to-height ratio corresponding to M0 as follows: The width-to-height ratio corresponding to M1 is Then, m TB0 and m TB3 Compare with P respectively, if m TB0 If it is not within the numerical range corresponding to the pallet width-to-height ratio threshold P, the target corner point coordinates of the group M0 (T0, B0, C0) are filtered out, and the valid group of target corner point coordinates is obtained as M3 (T3, B3, C3).

[0082] For another example, assuming that the pallet width-to-height ratio threshold is P, and the coordinates of at least one set of target corner points after preliminary filtering of pallet Q are M0(T0, B0, C0) and M3(T3, B3, C3), then based on T0(X T0 , Y T0 ) and B0(X B0 ,Y B0 ), and M3 in T3 (X T3 , Y T3 ) and B3(X B3 ,Y B3 ), and calculate the width-to-height ratio corresponding to M0 as follows: The width-to-height ratio corresponding to M1 is Then, m TB0 and m TB3 Compare with P respectively, if m TB0 and m TB1 If the values ​​of the two groups of target corner coordinates M0 (T0, B0, C0) and M3 (T3, B3, C3) are not within the range corresponding to the pallet width-to-height ratio threshold value P, the two groups of target corner coordinates M0 (T0, B0, C0) and M3 (T3, B3, C3) are filtered out, and the valid group of target corner coordinates is empty. That is, the position information of the pallet Q cannot be determined. Based on this example, when it is determined that the valid group of target corner coordinates of any pallet is empty, this embodiment can control the image acquisition device to acquire a new depth image and a new color image of the pallet, and detect the position information of the pallet based on the new depth image and the new color image.

[0083] In some embodiments, at least one set of target corner point coordinates of the pallet may be filtered according to the pallet width and height reference values. Specifically, when the width difference of a set of target corner point coordinates is within the preset range of the width reference value in the pallet width and height reference values, and the height difference of a set of target corner point coordinates is within the preset range of the height reference value in the pallet width and height reference values, then the set of target corner point coordinates is valid, otherwise it is invalid.

[0084] It should be noted that the confidence in each group of target corner point coordinates in this embodiment is to facilitate the first filtering out of erroneous or redundant corner point coordinates. Therefore, after the first filtering operation, the confidence in at least one group of target corner point coordinates that pass the first filtering can be ignored, that is, each group of corner point coordinates that pass the first filtering can be expressed as Mk(Tk, Bk). Wherein, k represents the kth group of target corner point coordinates, and k is an integer greater than 1.

[0085] In an embodiment of the present invention, it is preferred to filter at least one set of target corner point coordinates of the pallet based on a confidence filtering threshold and a pallet width-to-height ratio threshold to eliminate as many erroneous or redundant corner point coordinates as possible and reduce interference factors in the subsequent determination of the pallet's posture information.

[0086] S204 , determining a tray area of ​​the tray in the processed depth image according to the at least one set of target corner point coordinates after filtering.

[0087] Specifically, after filtering at least one set of target corner point coordinates of each pallet to obtain the effective group of target corner point coordinates of each pallet, this embodiment can determine the rectangular area corresponding to each pallet in the aligned depth image based on the effective group of target corner point coordinates of each pallet, and determine the rectangular area corresponding to each pallet as the pallet area of ​​each pallet.

[0088] S205: Determine the position information of the pallet according to the data in the pallet area of ​​the pallet.

[0089] The technical solution provided by the embodiment of the present invention obtains a depth image and a color image of at least one pallet, inputs the color image into the pallet detection model, determines at least one set of target corner point coordinates of the pallet, and determines the pallet area of ​​the pallet in the depth image based on the at least one set of target corner point coordinates, and then determines the pallet's posture information based on the data in the pallet area of ​​the pallet. The embodiment of the present invention detects the posture of the pallet by combining the color image and the depth image, which not only achieves the purpose of pallet posture detection, but also can fully utilize the rich semantic information of the color image and reduce the processing complexity of the depth image, thereby providing conditions for improving the speed of pallet posture detection. In addition, by aligning the acquired depth image and color image, and filtering at least one set of target corner point coordinates of the pallet, the accuracy of the pallet posture detection can be further improved, thereby providing favorable conditions for subsequent pallet handling.

[0090] Embodiment 3

[0091] Figure 3 FIG. 1 is a flow chart of a pallet posture detection method provided in Embodiment 3 of the present invention. Based on the above embodiments, the pallet posture detection method provided in this embodiment is further explained. Figure 3 As shown, the method specifically includes:

[0092] S301, acquiring a depth image and a color image of at least one pallet, and aligning the depth image and the color image.

[0093] S302, inputting the processed color image into the pallet detection model to determine at least one set of target corner point coordinates of the pallet, wherein the pallet detection model is obtained based on a preset neural network training.

[0094] The processed color image is input into the pallet detection model to determine at least one set of target corner point coordinates of the pallet.

[0095] S303: Determine first target point coordinates corresponding to the first corner point coordinates and second target point coordinates corresponding to the second corner point coordinates in the processed depth image.

[0096] In this embodiment, at least one set of target corner point coordinates refers to a filtered valid set of target corner point coordinates. The valid set of target corner point coordinates includes a first corner point coordinate and a second corner point coordinate. The first corner point coordinate and the second corner point coordinate are preferably a set of mutually diagonal corner point coordinates of a pallet. For example, the upper left corner point coordinate and the lower right corner point coordinate of a pallet, or the lower left corner point coordinate and the upper right corner point coordinate of a pallet, which are not specifically limited here.

[0097] The first target point coordinates corresponding to the first corner point coordinates and the second target point coordinates corresponding to the second corner point coordinates refer to the pixel coordinates of the first pixel corresponding to the first corner point coordinates and the pixel coordinates of the second pixel corresponding to the second corner point coordinates in the depth image. In this embodiment, the pixel coordinates are image coordinates, that is, plane coordinates.

[0098] Specifically, there is a one-to-one correspondence between the pixels in the aligned depth image and the color image. Therefore, based on the first corner point coordinates and the second corner point coordinates in the valid group of target corner point coordinates, the first target point coordinates corresponding to the first corner point coordinates and the second target point coordinates corresponding to the second corner point coordinates can be determined in the aligned depth image, respectively, to lay the foundation for the subsequent determination of the pallet area of ​​the pallet.

[0099] For example, if the effective target corner point coordinates of pallet Q are M3(T3, B3), where T3 is the first corner point coordinate, it can be expressed as T3(X T3 , Y T3 ), B3 is the coordinate of the second corner point, which can be expressed as B3(X B3 ,Y B3 ). Then according to T3(X T3 , YT3 ) and B3(X B3 ,Y B3 ), and determine the first target point coordinates T3' corresponding to T3 and the second target point coordinates B3' corresponding to B3 in the depth image after alignment processing.

[0100] S304: Determine a tray area of ​​the tray according to the first target point coordinates and the second target point coordinates.

[0101] Continuing with the above example, if the coordinates of the first target point T3'(X T3' , Y T3' ) and the second target point coordinate B3'(X B3' ,Y B3' ), then according to T3'(X T3' , Y T3' ) and B3'(X B3' ,Y B3' ) to calculate Δx T3',B3' =|X T3' -X B3' | and Δy T3',B3' =|Y T3' -Y B3' |, and then use T3'(X T3' , Y T3' )、B3'(X B3' ,Y B3' ), Δx T3',B3' =|X T3' -X B3' | and Δy T3',B3' =|Y T3' -Y B3' |, calculate the coordinates T3" (X) of the third target point of the pallet Q in the depth image T3” , Y T3” ) and the fourth target point coordinates B3" (X B3” ,Y B3” ), so according to T3'(X T3' , Y T3' )、B3'(X B3' ,Y B3' )、T3)(X T3” , Y T3” ) and B3" (X B3” ,Y B3” ) determines the tray area of ​​tray Q. Among them, X T3” =X T3' +Δx T3',B3' , Y T3” =Y B3' +Δy T3',B3' , X B3” =XB3' -Δx T3',B3' and Y B3” =Y T3' -Δy T3',B3' .

[0102] For another example, if there are three pallets, namely pallet Q1, pallet Q2 and pallet Q3, pallet area 1 of pallet Q1, pallet area 2 of pallet Q2 and pallet area 3 of pallet Q3 can be determined respectively. The pallet area determination process is similar to the above-mentioned determination of the pallet area of ​​pallet Q, and the specific details are referred to the process of determining the pallet area of ​​pallet Q, which will not be described in detail here.

[0103] S305 , determining the three-dimensional coordinates of each pixel point in the tray area of ​​the tray according to the pixel coordinates of each pixel point.

[0104] In this embodiment, the three-dimensional coordinates refer to the coordinates corresponding to each pixel point in the image acquisition device coordinate system (camera coordinate system) calculated according to the pixel coordinates of each pixel point in the tray area.

[0105] Specifically, in this embodiment, the three-dimensional coordinates of each pixel point can be determined by the following formula (1):

[0106]

[0107] Where u and v represent the pixel coordinates (i.e., two-dimensional coordinates) of any pixel point in the tray area of ​​the tray, and f x and f y represents the focal length parameter of the image acquisition device, u0 and v0 represent the imaging point coordinates (principal point coordinates) of the image acquisition device, and x, y and z represent the three-dimensional coordinates of any pixel point in the tray area of ​​the tray in the image acquisition device coordinate system. It should be noted that z can also be represented as the depth value of the above pixel point in the image acquisition device coordinate system. In this embodiment, the image acquisition device coordinate system can be as follows: Figure 4A As shown. Among them, Figure 4A The coordinate system of the image acquisition device is set based on the deployment method of pallet Q.

[0108] The following description will be continued by taking Example 1 in S304 as an example. Assume that according to T3'(X T3' , Y T3' )、B3'(X B3' ,Y B3' )、T3)(X T3” , Y T3” ) and B3" (X B3” ,Y B3”) After determining the pallet area of ​​pallet Q, the pallet posture detection device calculates the three-dimensional coordinates of each pixel point based on the pixel coordinates of each pixel point in the pallet area using the above formula (1). T3' , Y T3' )、B3'(X B3' ,Y B3' )、T3)(X T3” , Y T3” ) and B3" (X B3” ,Y B3” ) respectively correspond to the three-dimensional coordinates: and Then, according to the three-dimensional coordinates of the above four pixel points, the three-dimensional tray area of ​​the tray Q on the depth image is determined, as follows: Figure 4B shown.

[0109] Taking Example 2 in S304 as an example, when the number of pallets is 3, namely pallet Q1, pallet Q2 and pallet Q3, the above formula (1) can be used to calculate the 3D pallet area of ​​each pallet on the depth image, as follows: Figure 4C As shown. In this example, the 3D tray area of ​​tray Q1 on the depth image is tray area 1, the 3D tray area of ​​tray Q2 on the depth image is tray area 2, and the 3D tray area of ​​tray Q3 on the depth image is tray area 3. The process of determining each 3D tray area is similar to the process of determining the 3D tray area of ​​tray Q as mentioned above. For details, refer to the process of determining the 3D tray area of ​​tray Q, which will not be described in detail here.

[0110] S306: Determine the position and posture information of the pallet according to the three-dimensional coordinates of the pixel points in the pallet area of ​​the pallet.

[0111] The posture information includes position information and attitude information. It is worth noting that due to the working characteristics of the pallet posture detection device, the posture of the pallet has only one degree of freedom, namely the yaw angle. In other words, determining the posture information of the pallet in the embodiment of the present invention is to determine the yaw angle of the pallet.

[0112] Specifically, the position information and posture information of each pallet may be determined based on the pallet area of ​​each pallet, and then the position and posture information of each pallet may be determined based on the position information and posture information of each pallet.

[0113] In a specific implementation, when determining the position information of the pallet corresponding to the pallet area of ​​each pallet, the center point coordinates of each pallet can be determined according to the three-dimensional coordinates corresponding to the first target point coordinates and the three-dimensional coordinates corresponding to the second target point coordinates, and the center point coordinates are determined as the position information of each pallet. The center point coordinates can be expressed as (x, y, z).

[0114] Exemplarily, in this embodiment, the coordinates of the center point of each pallet can be determined by the aforementioned formula (1). It should be noted that when the coordinates of the center point of the pallet are determined based on formula (1), the meanings of the parameters in formula (1) are as follows: u represents the three-dimensional coordinates corresponding to the coordinates of the first target point, v represents the three-dimensional coordinates corresponding to the coordinates of the second target point, and f x and f y represents the focal length parameter of the acquisition device, u0 and v0 represent the imaging point coordinates (principal point coordinates) of the image acquisition device, and x, y and z represent the coordinates of the center point of the pallet in the coordinate system of the image acquisition device.

[0115] Furthermore, when determining the posture information of the pallet corresponding to each pallet area, it can be achieved through the following steps:

[0116] S11, selecting a plurality of pixel points from a tray area of ​​the tray.

[0117] S12, projecting a plurality of pixel points selected from the pallet area of ​​the pallet to the posture detection surface of the pallet, performing linear fitting on the three-dimensional coordinates corresponding to all pixel points in the posture detection surface of the pallet, and determining the yaw angle of the pallet based on the fitting result.

[0118] The multiple pixel points may be selected randomly, which is not specifically limited here.

[0119] Since the posture detection surface of the pallet is a plane and in the image acquisition device coordinate system (such as Figure 4A The yaw angle of the pallet in the figure is only related to the XZ plane formed by the X-axis and the Z-axis. Therefore, in this embodiment, the posture detection surface of the pallet is the XZ detection surface, and then a slope value can be calculated by the three-dimensional coordinates (x1,0,z1) and (x2,0,z2) corresponding to any two pixel points on the XZ detection surface. Then, the inverse tangent function is calculated based on the K value to obtain the yaw angle of the pallet.

[0120] In the specific implementation process, because the three-dimensional coordinate data corresponding to a single pixel point fluctuates, the yaw angle of the pallet calculated based on the three-dimensional coordinates corresponding to any two pixel points on the XZ detection plane will have a large error and be unstable. To this end, this embodiment selects multiple pixel points from the pallet area of ​​each pallet, and projects the selected multiple pixel points onto the XZ detection plane to perform linear fitting on the three-dimensional coordinates corresponding to the projected pixel points on the XZ detection plane to obtain a fitting line. Then, the inverse tangent value is calculated based on the slope of the fitting line to obtain the yaw angle of the pallet. In other words, the yaw angle of the pallet in this embodiment is the inverse tangent value of the slope of the fitting line.

[0121] Among them, any fitting algorithm can be used to fit the three-dimensional coordinates corresponding to the projected pixel points on the XZ detection plane. In this embodiment, the fitting algorithm is preferably the RANSAC algorithm. By using the RANSAC algorithm, not only the fitting effect is good, the fitting accuracy is high, and the fitting accuracy is controllable.

[0122] For example, Figure 4D As shown, Figure 4D is a schematic diagram of projecting multiple selected pixel points onto the XZ detection plane. Figure 4D The discrete points shown are all the pixel points projected onto the XZ detection plane. Then, the three-dimensional coordinates corresponding to all the pixel points are linearly fitted by the fitting algorithm to obtain the fitting line W. The fitting line can be expressed as: z = ax + b, where a is the slope of the fitting line. Then, the inverse tangent value is calculated based on the slope a, and the inverse tangent value is determined as the yaw angle Ry = arctan (a) of the pallet.

[0123] After determining the position information and posture information of the pallet, the position information of the pallet can be obtained according to the position information and posture information. X-Y ,Ry,0 Y-Z ], where (x, y, z) represents the location information of the pallet, (0 X-Y ,Ry,0 Y-Z ) indicates the posture information of the tray.

[0124] The technical solution provided by the embodiment of the present invention obtains a depth image and a color image of at least one pallet, inputs the color image into the pallet detection model, determines at least one set of target corner point coordinates of the pallet, and determines the pallet area of ​​the pallet in the depth image based on the at least one set of target corner point coordinates, and then determines the pallet posture information based on the data in the pallet area of ​​the pallet. The embodiment of the present invention detects the posture of the pallet by combining the color image and the depth image, which not only achieves the purpose of pallet posture detection, but also can fully utilize the rich semantic information of the color image and reduce the processing complexity of the depth image, thereby providing conditions for improving the speed of pallet posture detection.

[0125] Embodiment 4

[0126] Combine the following Figure 5-Figure 7 , the generation process of the pallet detection model in the pallet posture detection method of the embodiment of the present invention is specifically described. In specific implementation, the pallet detection model can be generated in different ways. Figure 5 , the first generation process of the pallet detection model provided by the embodiment of the present invention is described. Figure 5 As shown, the method specifically includes:

[0127] S401, acquiring a data set, wherein the data set includes a plurality of color images of at least one tray.

[0128] In this embodiment, a depth camera or a stereo camera may be controlled to capture multiple color images of at least one pallet in various lighting environments and various orientations to obtain a data set; or, multiple color images of at least one pallet in various lighting environments and various orientations may be obtained from a massive database on the Internet through tools such as a web crawler to obtain a data set, etc., which are not specifically limited here.

[0129] Furthermore, after obtaining the data set, the color images with poor image quality in the data set can be eliminated, and then based on the selected high-quality color images, the generation speed and accuracy of the pallet detection model can be improved. The color images with poor image quality can be, but are not limited to: too dark brightness, too bright brightness, low definition, and blurred images.

[0130] S402: Based on a plurality of color images of at least one pallet, a preset neural network model is trained to generate a pallet detection model.

[0131] Among them, the preset neural network model can be selected as a convolutional neural network model, and there is no restriction on it here.

[0132] Specifically, several color images of at least one pallet in the acquired data set can be used as input data and input into a preset neural network model, so as to use the preset neural network model to initialize the input data and continuously adjust the parameters until the accuracy of the trained neural network model reaches the accuracy threshold, and the trained neural network model is determined as the pallet detection model. The accuracy threshold can be flexibly set according to the actual application needs and is not limited here.

[0133] Secondly, combined Figure 6 The second generation process of the pallet detection model provided by the embodiment of the present invention is described. Figure 6 As shown, the method specifically includes:

[0134] S501, acquiring a data set, and dividing the data set to obtain a training data set and a test data set, wherein the data set includes a plurality of color images of at least one tray.

[0135] S502: Training the pallet detection model based on the training data set and the preset neural network.

[0136] S503, based on the test data set, testing whether the pallet detection model currently trained meets preset requirements, and if so, stopping training the pallet detection model.

[0137] In this embodiment, the data set can be divided into a training data set and a test data set in an equal or unequal manner. The preset conditions can be flexibly set according to actual needs, for example, the accuracy of the pallet detection model can reach an accuracy threshold, etc., which is not limited here.

[0138] The preset requirements may include that the accuracy of the pallet detection model reaches an accuracy threshold, etc.

[0139] Specifically, several color images of at least one pallet in the training data set are used as input data and input into a preset neural network model, so as to initialize the input data and continuously adjust the parameters using the preset neural network model to generate a pallet detection model. Then, several color images of at least one pallet in the test data set are used to test the generated pallet detection model to verify whether the accuracy of the pallet detection model reaches the accuracy threshold.

[0140] If the accuracy threshold is reached, the pallet detection model is determined to be the final pallet detection model. If the accuracy threshold is not reached, the pallet detection model is trained again using the training data set until the accuracy of the newly generated pallet detection model reaches the accuracy threshold, and the newly generated pallet detection model is determined to be the final pallet detection model.

[0141] Again, combined Figure 7 The third generation process of the pallet detection model provided by the embodiment of the present invention is described. Figure 7 As shown, the method specifically includes:

[0142] S601, acquiring a data set, and performing enhancement processing on a plurality of color images of at least one tray in the data set to obtain an enhanced data set.

[0143] S602: Training a preset neural network model based on the enhanced data set to generate a pallet detection model.

[0144] The enhancement processing of the color image may include at least one of the following: brightness adjustment, image cropping, image translation and noise addition.

[0145] That is, after obtaining a data set including several color images of at least one pallet, each color image is enhanced in different ways to enhance the original data set, and more color images of pallets are obtained to achieve the purpose of expanding the data set.

[0146] For example, Fig. 8AThe obtained data set includes 24 color images of pallets. Then, by performing enhancement processing on the color images of the 24 pallets respectively, an enhanced data set with 52 color images of pallets can be obtained. Figure 8B .

[0147] Furthermore, the color images in the expanded data set are used as input data and input into the preset neural network model, so as to use the preset neural network model to initialize the input data and continuously adjust the parameters until the accuracy of the trained neural network model reaches the accuracy threshold, and the trained neural network model is determined as the pallet detection model.

[0148] It can be understood that, by adopting the above-mentioned multiple methods to generate the pallet detection model, this embodiment can realize the customization of the pallet detection model to meet the personalized needs of different users and improve the user experience.

[0149] Embodiment 5

[0150] In order to achieve the above objective, an embodiment of the present invention further proposes a pallet posture detection device. Fig. 9 Schematic diagram of the structure of a pallet posture detection device provided in Embodiment 5 of the present invention. Fig. 9 As shown, the pallet posture detection device 700 provided in the embodiment of the present invention includes: an image acquisition module 710 , a corner point determination module 720 , a region determination module 730 and a posture determination module 740 .

[0151] The image acquisition module 710 is used to acquire a depth image and a color image of at least one tray;

[0152] A corner point determination module 720, configured to input the color image into a pallet detection model to determine at least one set of target corner point coordinates of the pallet, wherein the pallet detection model is obtained based on a preset neural network training;

[0153] An area determination module 730, configured to determine a tray area of ​​the tray in the depth image according to the at least one set of target corner point coordinates;

[0154] The posture determination module 740 is used to determine the posture information of the pallet according to the data in the pallet area of ​​the pallet.

[0155] As an optional implementation of the embodiment of the present invention, the apparatus 700 further includes: a processing module;

[0156] Wherein, the processing module is used to align the depth image and the color image;

[0157] Accordingly, the corner point determination module 720 is specifically configured to:

[0158] The processed color image is input into the pallet detection model to determine at least one set of target corner point coordinates of the pallet.

[0159] As an optional implementation of the embodiment of the present invention, the set of target corner point coordinates includes: first corner point coordinates and second corner point coordinates;

[0160] Accordingly, the region determination module 730 is specifically configured to:

[0161] Determine the coordinates of a first target point corresponding to the coordinates of the first corner point and the coordinates of a second target point corresponding to the coordinates of the second corner point in the processed depth image;

[0162] A tray area of ​​the tray is determined according to the first target point coordinates and the second target point coordinates.

[0163] As an optional implementation of the embodiment of the present invention, the posture determination module 740 includes: a three-dimensional coordinate determination unit, a position determination unit, a posture determination unit and a posture determination unit;

[0164] Wherein, the three-dimensional coordinate determining unit is used to determine the three-dimensional coordinates of each pixel point according to the pixel coordinates of each pixel point in the tray area of ​​the tray;

[0165] a position determination unit, configured to determine the coordinates of the center point of the pallet according to the three-dimensional coordinates corresponding to the first target point coordinates and the three-dimensional coordinates corresponding to the second target point coordinates;

[0166] a posture determination unit, configured to project a plurality of pixel points selected from a tray area of ​​the tray onto a posture detection surface of the tray, perform linear fitting on three-dimensional coordinates corresponding to all pixel points on the posture detection surface of the tray, and determine a yaw angle of the tray based on the fitting result;

[0167] The position and posture determining unit is used to determine the position and posture information of the pallet according to the center point coordinates and the yaw angle of the pallet.

[0168] As an optional implementation of the embodiment of the present invention, the device 700 further includes: a filtering module;

[0169] The filtering module is used to filter at least one set of target corner point coordinates of the tray.

[0170] As an optional implementation of the embodiment of the present invention, the device 700 further includes: a data acquisition module, a data partitioning module, a model training module and a model testing module;

[0171] Wherein, the data acquisition module is used to acquire a data set, wherein the data set includes a plurality of color images of at least one tray;

[0172] A data partitioning module is used to partition the data set into a training data set and a test data set;

[0173] A model training module, used for training the pallet detection model based on the training data set and the preset neural network;

[0174] The model testing module is used to test whether the pallet detection model currently trained meets the preset requirements based on the test data set, and if so, stop training the pallet detection model.

[0175] As an optional implementation of the embodiment of the present invention, the apparatus 700 further includes: an enhancement processing module;

[0176] The enhanced data set is obtained by performing enhancement processing on a plurality of color images of at least one tray in the data set, wherein the enhancement processing includes at least one of the following: brightness adjustment, image cropping, image translation and noise addition.

[0177] It should be noted that the aforementioned explanation of the embodiment of the pallet posture detection method is also applicable to the pallet posture detection device of this embodiment, and the implementation principle is similar, which will not be repeated here.

[0178] The technical solution provided by the embodiment of the present invention detects the position and posture of the pallet by combining the color image and the depth image, which not only achieves the purpose of detecting the position and posture of the pallet, but also can fully utilize the rich semantic information of the color image and reduce the processing complexity of the depth image, thereby providing conditions for improving the speed of pallet position and posture detection.

[0179] Embodiment 6

[0180] Fig.10 Schematic diagram of the structure of a pallet posture detection device provided in the sixth embodiment of the present invention. In this embodiment, the pallet posture detection device can be an unmanned forklift or a handling robot. Fig.10 As shown, the pallet posture detection device 800 includes: an image acquisition device 810, a processor 820, a storage device 830, an input device 840 and an output device 850; wherein the image acquisition device 810 can be a depth camera or a stereo camera, etc.; the number of the processor 820 can be one or more, Fig.10 A processor 820 is taken as an example; the image acquisition device 810, the processor 820, the storage device 830, the input device 840 and the output device 850 of the pallet posture detection device 800 can be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0181] Among them, the pallet posture detection device 800 can move autonomously and perform functions such as pallet handling;

[0182] An image acquisition device 810, for acquiring a depth image and a color image of at least one tray;

[0183] One or more processors 820; fixed on the body of the pallet posture detection device 800, and

[0184] The storage device 830 is used to store one or more programs and is fixed on the body of the pallet posture detection device 800.

[0185] When the one or more programs are executed by the one or more processors 820, the one or more processors 820 implement the pallet posture detection method described in any embodiment of the present invention.

[0186] The storage device 830, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the pallet posture detection method in the embodiment of the present invention (for example, the image acquisition module 710, the corner point determination module 720, the region determination module 730 and the posture determination module 740 in the pallet posture detection device 700). The processor 820 executes various functional applications and data processing of the subject by running the software programs, instructions and modules stored in the storage device 830, that is, realizes the above-mentioned pallet posture detection method.

[0187] The storage device 830 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the storage device 830 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the storage device 830 may further include a memory remotely arranged relative to the processor 820, and these remote memories may be connected to the pallet posture detection device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0188] The input device 840 may be used to receive input digital or character information and generate key signal input related to user settings and function control. The output device 850 may include display devices such as a display screen.

[0189] It should be noted that the aforementioned explanation of the embodiment of the pallet posture detection method is also applicable to the pallet posture detection device of this embodiment, and the implementation principle is similar, which will not be repeated here.

[0190] The pallet posture detection device provided in the embodiment of the present invention detects the pallet posture by combining color images and depth images, which not only achieves the purpose of pallet posture detection, but also can fully utilize the rich semantic information of color images and reduce the processing complexity of depth images, thereby providing conditions for improving the speed of pallet posture detection.

[0191] Embodiment 7

[0192] The computer-readable storage medium provided in the embodiment of the present invention stores a computer program, and when the computer program is executed by a processor, the method for detecting the position and posture of a tray according to the embodiment of the present invention is implemented. The method includes:

[0193] acquiring a depth image and a color image of at least one pallet;

[0194] Inputting the color image into a pallet detection model to determine at least one set of target corner point coordinates of the pallet, wherein the pallet detection model is obtained based on a preset neural network training;

[0195] determining a pallet region of the pallet in the depth image according to the at least one set of target corner point coordinates;

[0196] The position and posture information of the pallet is determined according to the data in the pallet area of ​​the pallet.

[0197] Of course, the computer-readable storage medium provided in an embodiment of the present invention, whose computer-executable instructions are not limited to the method operations described above, can also execute related operations in the pallet posture detection method provided in any embodiment of the present invention.

[0198] Through the above description of the implementation method, the technicians in the relevant field can clearly understand that the embodiments of the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a mobile robot to execute the methods described in each embodiment of the embodiments of the present invention.

[0199] It is worth noting that in the embodiment of the above-mentioned device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present invention.

[0200] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the embodiments of the present invention are not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the embodiments of the present invention. Therefore, although the embodiments of the present invention are described in more detail through the above embodiments, the embodiments of the present invention are not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the embodiments of the present invention, and the scope of the embodiments of the present invention is determined by the scope of the appended claims.

Claims

1. A pallet posture detection method, characterized in that: include: acquiring a depth image and a color image of at least one pallet; Performing alignment processing on the depth image and the color image; Input the processed color image into a pallet detection model to determine at least one set of target corner point coordinates of the pallet, wherein the pallet detection model is obtained based on a preset neural network training; wherein the set of target corner point coordinates includes: a first corner point coordinate and a second corner point coordinate; the first corner point coordinate and the second corner point coordinate are a set of corner point coordinates of the pallet that are diagonally opposite to each other; Determine the coordinates of a first target point corresponding to the coordinates of the first corner point and the coordinates of a second target point corresponding to the coordinates of the second corner point in the processed depth image; determining a pallet area of ​​the pallet according to the first target point coordinates and the second target point coordinates; Determining position information of the pallet according to data in the pallet area of ​​the pallet; Wherein, determining the position information of the pallet according to the data in the pallet area of ​​the pallet includes: Determine the three-dimensional coordinates of each pixel point according to the pixel coordinates of each pixel point in the tray area of ​​the tray; Determine the coordinates of the center point of the pallet according to the three-dimensional coordinates corresponding to the first target point coordinates and the three-dimensional coordinates corresponding to the second target point coordinates; Projecting a plurality of pixel points selected from a tray area of ​​the tray onto a posture detection surface of the tray, performing linear fitting on three-dimensional coordinates corresponding to all pixel points in the posture detection surface of the tray, and determining a yaw angle of the tray based on the fitting result; The position information of the pallet is determined according to the coordinates of the center point and the yaw angle of the pallet.

2. The method according to claim 1, characterized in that After inputting the processed color image into a pallet detection model to determine at least one set of target corner point coordinates of the pallet, the method further includes: A filtering process is performed on at least one set of target corner point coordinates of the tray.

3. The method according to claim 1, characterized in that Before acquiring the depth image and the color image of at least one pallet, the method further includes: acquiring a data set comprising a plurality of color images of at least one tray; Dividing the data set into a training data set and a test data set; Based on the training data set and the preset neural network, training the pallet detection model; Based on the test data set, test whether the pallet detection model currently trained meets the preset requirements. If so, stop training the pallet detection model.

4. The method according to claim 3, characterized in that After obtaining the data set, the method further includes: Performing enhancement processing on a plurality of color images of at least one tray in the data set to obtain an enhanced data set, wherein the enhancement processing includes at least one of the following: brightness adjustment, image cropping, image translation, and noise addition.

5. A pallet posture detection device, characterized in that: include: An image acquisition module, used to acquire a depth image and a color image of at least one tray; A processing module, used for aligning the depth image and the color image; A corner point determination module, used for inputting the processed color image into a pallet detection model to determine at least one set of target corner point coordinates of the pallet, wherein the pallet detection model is obtained based on a preset neural network training; wherein the set of target corner point coordinates includes: a first corner point coordinate and a second corner point coordinate; the first corner point coordinate and the second corner point coordinate are a set of corner point coordinates of the pallet that are diagonally opposite to each other; an area determination module, configured to determine a pallet area of ​​the pallet in the depth image according to the at least one set of target corner point coordinates; A posture determination module, used to determine the posture information of the pallet according to the data in the pallet area of ​​the pallet; The region determination module is specifically used for: Determine the coordinates of a first target point corresponding to the coordinates of the first corner point and the coordinates of a second target point corresponding to the coordinates of the second corner point in the processed depth image; determining a pallet area of ​​the pallet according to the first target point coordinates and the second target point coordinates; Among them, the posture determination module includes: A three-dimensional coordinate determining unit, configured to determine the three-dimensional coordinates of each pixel point according to the pixel coordinates of each pixel point in the tray area of ​​the tray; a position determination unit, configured to determine the coordinates of the center point of the pallet according to the three-dimensional coordinates corresponding to the first target point coordinates and the three-dimensional coordinates corresponding to the second target point coordinates; a posture determination unit, configured to project a plurality of pixel points selected from a tray area of ​​the tray onto a posture detection surface of the tray, perform linear fitting on three-dimensional coordinates corresponding to all pixel points on the posture detection surface of the tray, and determine a yaw angle of the tray based on the fitting result; The position and posture determining unit is used to determine the position and posture information of the pallet according to the center point coordinates and the yaw angle of the pallet.

6. A pallet posture detection device, characterized in that: include: An image acquisition device for acquiring a depth image and a color image of at least one pallet; one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the pallet posture detection method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the pallet posture detection method as described in any one of claims 1 to 4.

Images