A pallet inspection method, apparatus and electronic equipment

Abstract

This invention provides a pallet detection method, apparatus, and electronic device, relating to the field of image recognition technology. The method includes: acquiring an image to be detected captured by an image acquisition device; wherein the image acquisition device is located at the tip of one fork of a forklift, and a camera faces directly in front of the tip of the fork; determining the target pallet detection result corresponding to the image to be detected based on a preset correspondence between image features and pallet detection results; wherein the correspondence is determined based on a first type of sample image including complete pallet samples and a second type of sample image including incomplete pallet samples. Compared with existing technologies, the solution provided by this invention can improve the working efficiency of forklifts.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a tray detection method, apparatus, and electronic device. Background Technology

[0002] With the rapid development of the logistics industry and the continuous advancement of robotics technology, safe and efficient operating processes have become essential for logistics companies to enhance their market competitiveness. Consequently, forklifts have become indispensable equipment in logistics warehouses for storing and retrieving goods. A forklift, in particular, is a mobile robot with forks that can move relatively heavy objects.

[0003] Typically, goods are placed on pallets. Therefore, the process of a forklift moving goods involves the forks inserting under the pallet and lifting it upwards. Then, the forklift moves the pallet to another location. This process of the forks inserting under the pallet and lifting it upwards can be called pallet insertion / removal. Thus, before using the forklift's fork structure to move goods, it is necessary to first check if a pallet is present in front of the forks.

[0004] In related technologies, two different sensors are usually installed at the end of the fork teeth, and the images in front of the fork teeth obtained by the sensors are used to detect the presence of a tray. Summary of the Invention

[0005] The purpose of this invention is to provide a pallet detection method, apparatus, and electronic device to detect pallets close to the fork teeth while ensuring the accuracy of the acquired image in front of the fork teeth. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of the present invention provide a pallet detection method, the method comprising:

[0007] Acquire an image to be detected by an image acquisition device; wherein the image acquisition device is located at the tip of one fork of a forklift, and the camera faces directly in front of the tip of the fork;

[0008] Based on a preset correspondence between image features and tray detection results, the target tray detection result corresponding to the image to be detected is determined; wherein, the correspondence is determined based on a first type of sample image including complete tray samples and a second type of sample image including incomplete tray samples.

[0009] Optionally, in one specific implementation, the step of determining the target tray detection result corresponding to the image to be detected based on a preset correspondence between image features and tray detection results includes:

[0010] The image to be detected is input into a preset tray detection model, and the output result of the tray detection model is obtained as the target tray detection result corresponding to the image to be detected.

[0011] The tray detection model is trained based on the first type of sample images and the second type of sample images.

[0012] Optionally, in one specific implementation, the training method of the tray detection model includes:

[0013] Acquire first-class sample images and second-class sample images with added sample labels, and train a preset initial model using the acquired first-class sample images and second-class sample images; wherein, the sample labels are tray detection results representing the presence of trays;

[0014] When the initial model meets the preset conditions, training stops, and the tray detection model is obtained.

[0015] Optionally, in one specific implementation, if the target pallet detection result indicates the presence of a pallet, the method further includes:

[0016] Based on the image to be detected, determine the positional relationship between the pallet captured in the image and the fork tines of the forklift.

[0017] According to the aforementioned positional relationship, adjust the position of the fork tines and control the adjusted fork tines to insert into the pallet.

[0018] Optionally, in one specific implementation, the step of determining the positional relationship between the pallet captured in the image to be detected and the fork tines of the forklift, based on the image to be detected, includes:

[0019] Obtain the depth map of the image to be detected;

[0020] Based on the depth map, a plane fit is performed on the tray to obtain the spatial position of the tray;

[0021] Based on the spatial location of the forklift, determine the spatial position of the fork tips;

[0022] The positional relationship between the pallet and the fork tines is determined based on their spatial positions.

[0023] Secondly, embodiments of the present invention provide a pallet detection device, the device comprising:

[0024] An image acquisition module is used to acquire the image to be detected acquired by an image acquisition device; wherein, the image acquisition device is located at the tip of one fork of the forklift, and the camera faces directly in front of the tip of the fork;

[0025] The image detection module is used to determine the target tray detection result corresponding to the image to be detected based on a preset correspondence between image features and tray detection results; wherein the correspondence is determined based on a first type of sample image including complete tray samples and a second type of sample image including incomplete tray samples.

[0026] Optionally, in one specific implementation, the image detection module is specifically used for:

[0027] The image to be detected is input into a preset tray detection model, and the output result of the tray detection model is obtained as the target tray detection result corresponding to the image to be detected.

[0028] The tray detection model is trained based on the first type of sample images and the second type of sample images.

[0029] Optionally, in one specific implementation, the apparatus further includes:

[0030] The model training module is used to acquire the first type of sample images and the second type of sample images with sample labels added, and to train the preset initial model using the acquired first type of sample images and the second type of sample images.

[0031] The sample labels mentioned above represent the pallet detection results where a pallet is present; when the initial model meets the preset conditions, training stops, and the pallet detection model is obtained.

[0032] Optionally, in one specific implementation, if the target pallet detection result indicates the presence of a pallet, the device further includes:

[0033] The relationship determination module is used to determine the positional relationship between the pallet captured in the image to be detected and the fork tines of the forklift, based on the image to be detected.

[0034] The adjustment control module is used to adjust the position of the fork tines according to the positional relationship, and control the adjusted fork tines to insert into the pallet.

[0035] Optionally, in one specific implementation, the relationship determination module is specifically used for:

[0036] Obtain a depth map of the image to be detected; perform planar fitting on the pallet based on the depth map to obtain the spatial position of the pallet; determine the spatial position of the fork tips based on the spatial position of the forklift; determine the positional relationship between the pallet and the fork tines based on the spatial position of the pallet and the spatial position of the fork tines.

[0037] Thirdly, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0038] Memory, used to store computer programs;

[0039] When a processor executes a program stored in memory, it implements the steps of any of the tray detection methods provided in the first aspect above.

[0040] Fourthly, embodiments of the present invention provide a computer-readable medium storing a computer program, which, when executed by a processor, implements the steps of any of the tray detection methods provided in the first aspect.

[0041] Fifthly, embodiments of the present invention provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps of any of the tray detection methods described in the first aspect.

[0042] Beneficial effects of the embodiments of the present invention:

[0043] As can be seen from the above, the solution provided in the embodiments of the present invention first acquires the image to be inspected by the image acquisition device during pallet inspection. Since the image acquisition device is located at the tip of one fork of the forklift, and the camera of the image acquisition device faces directly in front of the tip of the fork, the distance between the pallet and the fork teeth is shortened when acquiring the image, improving image accuracy. Furthermore, only one image acquisition device is needed, reducing the cost of pallet inspection.

[0044] After acquiring the image to be detected, the target tray detection result corresponding to the image can be determined based on a preset correspondence between image features and tray detection results. Since this correspondence is based on a first-class sample image including complete tray samples and a second-class sample image including incomplete tray samples, a correspondence can be established between image features of images with incomplete tray imaging and tray detection results. Therefore, even when the image to be detected has incomplete tray imaging (i.e., does not contain a complete tray), the tray detection result can still be determined using the image features of the image to be detected through the aforementioned correspondence.

[0045] Based on this, the solution provided in the embodiments of the present invention can detect pallets that are close to the fork teeth while ensuring the accuracy of the image in front of the fork teeth, thereby improving the working efficiency of the forklift. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A schematic flowchart of the first pallet detection method provided in an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of a forklift.

[0049] Figure 3 This is a schematic diagram of a forklift fork.

[0050] Figure 4 This is a schematic diagram of a tray;

[0051] Figure 5 A first type of sample image provided in an embodiment of the present invention;

[0052] Figure 6 A second type of sample image provided in an embodiment of the present invention;

[0053] Figure 7 This is a schematic flowchart of a pallet detection method provided in an embodiment of the present invention;

[0054] Figure 8 This is a schematic flowchart of a pallet detection method provided in an embodiment of the present invention;

[0055] Figure 9 A flowchart illustrating a training method for a pallet detection model provided in an embodiment of the present invention;

[0056] Figure 10 This is a schematic diagram of the structure of a pallet detection device provided in an embodiment of the present invention;

[0057] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] This invention provides a pallet detection method. This method is applicable to any application scenario requiring pallet detection, such as forklifts used to move goods in logistics parks, forklifts used to move heavy objects on construction sites, and forklifts used to move crops in agriculture. Furthermore, this method can be applied to various electronic devices such as image acquisition devices and servers. For example, it can be applied to an image acquisition device used to acquire images to be detected, or it can be applied to a forklift control server that can communicate with the image acquisition device used to acquire images to be detected. Therefore, this invention does not limit the application scenario or the executing entity of the method; the executing entity of the method will be referred to as an electronic device.

[0060] An embodiment of the present invention provides a pallet detection method, which may include the following steps:

[0061] Acquire an image to be detected by an image acquisition device; wherein the image acquisition device is located at the tip of one fork of a forklift, and the camera faces directly in front of the tip of the fork;

[0062] Based on a preset correspondence between image features and tray detection results, the target tray detection result corresponding to the image to be detected is determined; wherein, the correspondence is determined based on a first type of sample image including complete tray samples and a second type of sample image including incomplete tray samples.

[0063] As can be seen from the above, the solution provided in the embodiments of the present invention first acquires the image to be inspected by the image acquisition device during pallet inspection. Since the image acquisition device is located at the tip of one fork of the forklift, and the camera of the image acquisition device faces directly in front of the tip of the fork, the distance between the pallet and the fork teeth is shortened when acquiring the image, improving image accuracy. Furthermore, only one image acquisition device is needed, reducing the cost of pallet inspection.

[0064] After acquiring the image to be detected, the target tray detection result corresponding to the image can be determined based on a preset correspondence between image features and tray detection results. Since this correspondence is based on a first-class sample image including complete tray samples and a second-class sample image including incomplete tray samples, a correspondence can be established between image features of images with incomplete tray imaging and tray detection results. Therefore, even when the image to be detected has incomplete tray imaging (i.e., does not contain a complete tray), the tray detection result can still be determined using the image features of the image to be detected through the aforementioned correspondence.

[0065] Based on this, the solution provided in the embodiments of the present invention can detect pallets that are close to the fork teeth while ensuring the accuracy of the image in front of the fork teeth, thereby improving the working efficiency of the forklift.

[0066] The pallet detection method provided by the present invention will be described in detail below with reference to the accompanying drawings.

[0067] Figure 1 A pallet detection method provided in this embodiment of the invention, such as... Figure 1 As shown, the method includes the following steps S101-S102.

[0068] S101: Acquire the image to be detected acquired by the image acquisition device;

[0069] The image acquisition device is located at the tip of one of the forks of the forklift, with the camera facing directly in front of the tip of the fork.

[0070] During pallet insertion and removal, the forklift is first moved to a designated location near the pallet. Then, the image to be inspected is acquired by the image acquisition device installed on the forklift when it is in the designated position. Furthermore, the image to be inspected can include a complete pallet or an incomplete pallet.

[0071] Forklifts typically have at least one fork. For example, such as Figure 2 As shown, the forklift has two forks 210, wherein each fork 210 has a fork tip 220.

[0072] In this way, an image acquisition device can be installed at the tip of one of the forks of a forklift, with the camera of the image acquisition device facing directly in front of the tip. Thus, the image acquisition device can capture an image of an object located directly in front of the fork. Here, "directly in front of the fork" refers to the direction in which the tip of the fork is facing during the insertion or removal process.

[0073] For example, such as Figure 3As shown, a camera 330 is installed on the tip 320 of one of the forks 310 of the forklift, with the camera head facing directly in front of the tip 320. This camera 330 serves as an image sensor mounted on the tip 320 of the fork 310, acquiring an image in front of the tip 320 to detect whether a pallet is present in front of the tip 320.

[0074] In this way, the image to be inspected can be acquired using the image acquisition device installed above. This shortens the distance between the pallet and the fork teeth, improves the image accuracy of the acquired image, and reduces the cost of pallet inspection by requiring only one image acquisition device.

[0075] The image acquisition device can be any type of device capable of image acquisition, such as an image sensor or camera.

[0076] When the image acquisition device is the executing entity of the pallet detection method provided in this embodiment of the invention, after acquiring the image to be detected, the image acquisition device can further determine the target pallet detection result of the image to be detected. Specifically, after determining the target pallet detection result, the image acquisition device can send the target pallet detection result to the forklift's control server, thereby enabling the control server to control the forklift's next action; alternatively, the image acquisition device can also directly control the forklift's next action, in which case the image acquisition device can be considered part of the forklift's control server.

[0077] When the image acquisition device is not the executing entity of the tray detection method provided in this embodiment of the invention, after acquiring the image to be detected, the image acquisition device can send the image to be detected to the executing entity of the tray detection method provided in this embodiment of the invention to perform subsequent related operations.

[0078] S102: Based on the preset correspondence between image features and tray detection results, determine the target tray detection result corresponding to the image to be detected.

[0079] The correspondence is determined based on a first type of sample image that includes complete pallet samples and a second type of sample image that includes incomplete pallet samples.

[0080] After acquiring the image to be detected, the image features of the image to be detected can be obtained first. Then, according to the preset correspondence between the image features and the tray detection results, the tray detection result corresponding to the image features of the image to be detected is determined. The determined tray detection result is the target tray detection result corresponding to the image to be detected. The target tray detection result can indicate whether there is a tray in the image to be detected. That is, the target tray detection result can indicate whether the image acquisition device has acquired the image of the tray when acquiring the image to be detected.

[0081] For example, if the preset correspondence between image features and tray detection results is the correspondence between each feature vector and the tray detection result, then after obtaining the image features of the image to be detected, it is possible to determine the specified feature vector among the feature vectors included in the above correspondence that has a similarity to the image features of the image to be detected that is greater than a specified threshold. The tray detection result corresponding to the determined specified feature vector is the tray detection result corresponding to the image features of the image to be detected. In other words, the tray detection result corresponding to the determined specified feature vector is the target tray detection result corresponding to the image to be detected.

[0082] Specifically, when the target pallet detection result indicates that a pallet exists in the image to be detected, it means that there is a pallet that needs to be picked up in front of the forklift, and thus, the forklift can be controlled to pick up the pallet.

[0083] When the target pallet detection result indicates that there is no pallet in the image to be detected, it means that there is no pallet to be picked up in front of the forklift. Therefore, the forklift can move to another place to perform pallet detection again.

[0084] Since the above correspondence is determined based on the first type of sample image including complete tray samples and the second type of sample image including incomplete tray samples, the correspondence between the image features of the incomplete tray image and the tray detection result can be established. Therefore, when the tray image in the image to be detected is incomplete, that is, it does not contain a complete tray, the image features of the image to be detected can still be used to determine the tray detection result of the image to be detected through the above correspondence.

[0085] The above correspondence between image features and tray detection results is determined based on a first type of sample image including complete tray samples and a second type of sample image including incomplete tray samples.

[0086] A pallet is a container used to hold goods. For example, such as Figure 4 The image shows a tray consisting of three legs and two holes in between, and this type of tray can be made of various materials and colors.

[0087] In other words, for the pallet samples used in the application scenario of the pallet detection method provided in the embodiments of the present invention, a first type of sample image including complete pallet samples and a second type of sample image including incomplete pallet samples can be obtained in advance.

[0088] For example, when the tray sample is as follows Figure 4 The tray shown can be used to obtain, for example... Figure 5 The first type of sample image shown, and as shown Figure 6 The second type of sample image is shown.

[0089] In this way, for each sample image in the first and second class of sample images, the image features of the sample image can be extracted, and then the initial correspondence between the image features and the tray detection result used to characterize the sample image including the tray can be obtained; after obtaining all the initial correspondences, the above-mentioned correspondence between image features and tray detection results can be established by further utilizing all the initial correspondences.

[0090] For example, if the image features of each of the above sample images are feature vectors, then each initial correspondence is an initial correspondence between a feature vector and the tray detection result used to characterize the sample image including the tray. Thus, the average value of the feature vectors in each of the above initial correspondences can be calculated, and a correspondence between the average value and the tray detection result used to characterize the sample image including the tray can be established as a correspondence between image features and tray detection results.

[0091] For example, if the image features of each sample image are feature vectors, then each initial correspondence is an initial correspondence between a feature vector and the tray detection result used to characterize the sample image including the tray. Thus, all initial correspondences can be determined as correspondences between image features and tray detection results.

[0092] In this embodiment of the invention, no specific limitation is made on the method of establishing the correspondence between image features and tray detection results.

[0093] Based on this, the solution provided in the embodiments of the present invention can detect pallets that are close to the fork teeth while ensuring the accuracy of the image in front of the fork teeth, thereby improving the working efficiency of the forklift.

[0094] Typically, the purpose of pallet detection is to insert or remove pallets. Therefore, in one possible implementation, such as... Figure 7 As shown, if the target pallet detection result indicates the presence of a pallet, the pallet detection method provided in this embodiment of the invention may further include the following steps S103-S104:

[0095] S103: Based on the image to be detected, determine the positional relationship between the pallet captured in the image and the fork tines of the forklift.

[0096] S104: Adjust the position of the fork tines according to the positional relationship, and control the fork tines after the position adjustment to pick up the pallet.

[0097] In this specific implementation, if the target pallet detection result indicates the presence of a pallet, it means that there is a pallet in front of the forklift that needs to be picked up, and that pallet is the pallet included in the image to be detected.

[0098] In this way, the spatial position of the pallet included in the image to be detected can be determined based on the image data of the pallet portion included in the image to be detected. Then, based on the spatial position of the forklift's forks and the spatial position of the pallet, the positional relationship between the pallet and the forklift's forks can be determined. Subsequently, the position of the forklift's forks can be adjusted according to the determined positional relationship so that the forklift's forks can pick up the pallet. For example, the forklift's forks can be adjusted to be below the pallet, and thus, the position-adjusted forks can be controlled to pick up the pallet.

[0099] Optionally, the spatial position of the fork tines of the forklift can be determined based on the spatial position of the forklift, the installation position of the fork tines on the forklift, and the length of the fork tines.

[0100] Alternatively, when the tray included in the image to be detected is complete, the image coordinates of the tray in the image to be detected can be directly converted into the spatial coordinates of the tray in space using the relevant parameters of the image acquisition device, thereby determining the spatial position of the tray using the spatial coordinates.

[0101] For example, when the image acquisition device is a camera, the image coordinates of the tray in the image to be detected can be directly converted into the spatial coordinates of the tray in space using the camera's intrinsic and extrinsic parameters.

[0102] Correspondingly, when the tray included in the image to be detected is incomplete, since the image coordinates of the tray in the image cannot be directly obtained, the spatial position of the tray cannot be directly obtained through coordinate transformation. Therefore, a depth map of the image to be detected can be obtained, and the depth point cloud of the incomplete tray surface can be extracted from the depth map. Pose recognition can then be performed on the incomplete tray to determine its spatial position.

[0103] Optionally, the image acquisition device described above may simultaneously include a first acquisition module for acquiring the image to be detected and a second acquisition module for acquiring a depth map. The positional relationship between the image acquired by the first acquisition module and the depth map acquired by the second acquisition module is fixed. Therefore, when the first acquisition module acquires the image to be detected, the second acquisition module can simultaneously acquire the depth map, and the acquired depth map is the depth map of the image to be detected. In this way, the depth map of the image to be detected can be acquired simultaneously with the acquisition of the image to be detected.

[0104] Optionally, the image acquisition device installed at the tip of one fork of the forklift for acquiring the image to be detected can be a visible light image acquisition device, thus acquiring the image to be detected as a visible light image. Alternatively, a depth map acquisition device can be installed on the forklift for acquiring a depth image. Furthermore, the positional relationship between the depth map acquisition device and the visible light image acquisition device is fixed, thus the positional relationship between the depth map acquired by the depth map acquisition device and the visible light image acquired by the visible light image acquisition device is fixed. Therefore, while the visible light image acquisition device acquires the image to be detected, the depth map acquisition device simultaneously acquires the depth map of the image to be detected. In this way, the depth map of the image to be detected can be acquired simultaneously with the acquisition of the image to be detected.

[0105] Optionally, the image acquisition device installed at the tip of one fork of the forklift for acquiring the image to be detected can be a visible light image acquisition device, thus acquiring the image to be detected as a visible light image. Alternatively, a depth map acquisition device can be installed on the forklift for acquiring a depth image. Furthermore, the positional relationship between the depth map acquisition device and the visible light image acquisition device is fixed, thus the positional relationship between the depth map acquired by the depth map acquisition device and the visible light image acquired by the visible light image acquisition device is fixed. Moreover, when the pallet included in the image to be detected is incomplete, the depth map acquisition device can be controlled to acquire the depth map of the image to be detected. In this way, the depth map of the image to be detected can be obtained even when the pallet included in the image to be detected is incomplete.

[0106] Based on this, in one optional implementation, such as Figure 8 As shown, step S103 above, which determines the positional relationship between the target pallet captured in the image to be detected and the fork tines of the forklift, may include the following steps S801-S804:

[0107] S801: Obtain the depth map of the image to be detected;

[0108] S802: Based on the depth map, perform planar fitting on the pallet to obtain the spatial position of the pallet;

[0109] S803: Determine the spatial position of the fork tips based on the spatial location of the forklift;

[0110] S804: Determine the positional relationship between the pallet and the fork tines based on the spatial position of the pallet and the spatial position of the fork tines.

[0111] In this specific implementation, the spatial position of the fork tips of the forklift can be determined based on the spatial location of the forklift.

[0112] Optionally, the spatial position of the fork tips can be determined based on the spatial position of the forklift, the installation position of the fork tines on the forklift, and the length of the fork tines.

[0113] Furthermore, if the target tray detection result indicates the presence of a tray, a depth map of the image to be detected can be obtained, and a plane fitting can be performed on the tray captured in the image to be detected based on the depth map.

[0114] If the pallet included in the image to be detected is incomplete, the depth point cloud of the surface of the incomplete pallet included in the image to be detected can be obtained after the above-mentioned planar fitting. Then, the pose of the pallet can be determined based on the depth point cloud to obtain the spatial position of the pallet. Furthermore, the positional relationship between the pallet and the fork tines can be determined based on the spatial position of the pallet and the spatial position of the fork tines.

[0115] Of course, if the tray included in the image to be detected is complete, the depth map of the image to be detected can also be obtained, and the tray in the image to be detected can be fitted to a plane based on the depth map.

[0116] Optionally, in one specific implementation, step S102 above, which determines the target tray detection result corresponding to the image to be detected based on a preset correspondence between image features and tray detection results, may include the following step 11:

[0117] Step 11: Input the image to be detected into the preset tray detection model and obtain the output result of the tray detection model as the target tray detection result corresponding to the image to be detected;

[0118] The tray detection model is trained based on first-class sample images and second-class sample images.

[0119] In this specific implementation, a tray detection model can be trained in advance using the first type of sample images and the second type of sample images, and the tray detection model establishes a correspondence between image features and tray detection results.

[0120] In this way, after obtaining the image to be detected, it can be directly input into the tray detection model. The tray detection model can then learn from the image to be detected, obtain the image features of the image to be detected, and then use the established correspondence between the image features and the tray detection results to obtain the target tray detection result corresponding to the image to be detected, and output the target tray detection result.

[0121] In other words, after the image to be detected is input into the above tray detection model, the output of the tray detection model is the target tray detection result of the image to be detected.

[0122] Alternatively, in one specific implementation, such as Figure 9 As shown, the training method for a pallet detection model provided in this embodiment of the invention may include the following steps S901-S902:

[0123] S901: Obtain the first type of sample image and the second type of sample image with sample labels added, and use the obtained first type of sample image and the second type of sample image to train the preset initial model;

[0124] S902: When the initial model meets the preset conditions, stop training and obtain the tray detection model.

[0125] In this specific implementation, a first type of sample image including complete tray samples and a second type of sample image including incomplete tray samples can be obtained, and each first type of sample image and each second type of sample image is labeled with a sample label to indicate the presence of a tray.

[0126] In the process of training the tray detection model, the electronic device used for model training can pre-build an initial model, and then input the first type of sample images and the second type of sample images into the initial model for training, thereby obtaining the tray detection model.

[0127] During training, the initial model can learn the image features of each first-class sample image and each second-class sample image, and output the labels of each sample. After learning from a large number of first-class and second-class sample images, the initial model gradually establishes the correspondence between image features and tray detection results, thus obtaining the tray detection model.

[0128] Training can be stopped once the initial model meets the preset conditions, resulting in a tray detection model.

[0129] Optionally, the above preset condition may be that the number of iterations for each first-class sample image and each second-class sample image reaches a preset number.

[0130] Optionally, the above preset condition can be that the error between the true value and the predicted value of the sample label of each first-class sample image and each second-class sample image is less than a preset error.

[0131] In this way, after obtaining the image to be detected, it can be directly input into the tray detection model mentioned above. Thus, the tray detection model can learn from the image to be detected, obtain the image features of the image to be detected, and then use the established correspondence between the image features and the tray detection results to obtain the target tray detection result corresponding to the image to be detected, that is, the target tray detection result of the image to be detected.

[0132] The electronic device used for model training and the electronic device that executes the tray detection method provided in this embodiment of the invention may be the same electronic device or different electronic devices.

[0133] Corresponding to the pallet detection method provided in the above embodiments of the present invention, the present invention also provides a pallet detection device.

[0134] Figure 10 This is a schematic diagram of the structure of a pallet detection device provided in an embodiment of the present invention, as shown below. Figure 10 As shown, the device may include the following modules:

[0135] The image acquisition module 1010 is used to acquire the image to be detected acquired by the image acquisition device; wherein the image acquisition device is located at the tip of any one of the forks of the forklift, and the camera faces directly in front of the tip of the fork.

[0136] The image detection module 1020 is used to determine the target tray detection result corresponding to the image to be detected based on a preset correspondence between image features and tray detection results; wherein the correspondence is determined based on a first type of sample image including complete tray samples and a second type of sample image including incomplete tray samples.

[0137] As can be seen from the above, the solution provided in the embodiments of the present invention first acquires the image to be inspected by the image acquisition device during pallet inspection. Since the image acquisition device is located at the tip of one fork of the forklift, and the camera of the image acquisition device faces directly in front of the tip of the fork, the distance between the pallet and the fork teeth is shortened when acquiring the image, improving image accuracy. Furthermore, only one image acquisition device is needed, reducing the cost of pallet inspection.

[0138] After acquiring the image to be detected, the target tray detection result corresponding to the image can be determined based on a preset correspondence between image features and tray detection results. Since this correspondence is based on a first-class sample image including complete tray samples and a second-class sample image including incomplete tray samples, a correspondence can be established between image features of images with incomplete tray imaging and tray detection results. Therefore, even when the image to be detected has incomplete tray imaging (i.e., does not contain a complete tray), the tray detection result can still be determined using the image features of the image to be detected through the aforementioned correspondence.

[0139] Based on this, the solution provided in the embodiments of the present invention can detect pallets that are close to the fork teeth while ensuring the accuracy of the image in front of the fork teeth, thereby improving the working efficiency of the forklift.

[0140] Optionally, in one specific implementation, the image detection module 1020 is specifically used for:

[0141] The image to be detected is input into a preset tray detection model, and the output result of the tray detection model is obtained as the target tray detection result corresponding to the image to be detected.

[0142] The tray detection model is trained based on the first type of sample images and the second type of sample images.

[0143] Optionally, in one specific embodiment, the apparatus further includes:

[0144] The model training module is used to acquire first-class sample images and second-class sample images with added sample labels, and to train a preset initial model using the acquired first-class sample images and second-class sample images; wherein, the sample labels are tray detection results representing the presence of a tray; when the initial model meets preset conditions, training stops and the tray detection model is obtained.

[0145] Optionally, in one specific embodiment, if the target pallet detection result indicates the presence of a pallet, the device further includes:

[0146] The relationship determination module is used to determine the positional relationship between the pallet captured in the image to be detected and the fork tines of the forklift, based on the image to be detected.

[0147] The adjustment control module is used to adjust the position of the fork tines according to the positional relationship, and control the adjusted fork tines to insert into the pallet.

[0148] Optionally, in one specific embodiment, the relationship determination module is specifically used for:

[0149] Obtain a depth map of the image to be detected; perform planar fitting on the pallet based on the depth map to obtain the spatial position of the pallet; determine the spatial position of the fork tips based on the spatial position of the forklift; determine the positional relationship between the pallet and the fork tines based on the spatial position of the pallet and the spatial position of the fork tines.

[0150] Corresponding to the above embodiments of the present invention, an electronic device is also provided, such as... Figure 11 As shown, it includes a processor 1110, a communication interface 1120, a memory 1130, and a communication bus 1140, wherein the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other through the communication bus 1140.

[0151] Memory 1130 is used to store computer programs;

[0152] When the processor 1110 executes the program stored in the memory 1130, it implements the steps of any of the tray detection methods provided in the above embodiments of the present invention.

[0153] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0154] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0155] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0156] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0157] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described tray detection methods.

[0158] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the tray detection methods described above.

[0159] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0160] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0161] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments, electronic device embodiments, computer-readable storage medium embodiments, and computer program product embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0162] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A pallet detection method, characterized in that, The method includes: Acquire an image to be detected by an image acquisition device; wherein the image acquisition device is located at the tip of one fork of a forklift, and the camera faces directly in front of the tip of the fork; Based on a pre-defined correspondence between image features and tray detection results, the target tray detection result corresponding to the image to be detected is determined; wherein, the correspondence is determined based on a first type of sample image including complete tray samples and a second type of sample image including incomplete tray samples; If the target tray detection result indicates the presence of a tray, obtain the depth map of the image to be detected; Based on the depth map, the tray is fitted with a plane to obtain the spatial position of the tray, including: if the tray included in the image to be detected is incomplete, then after plane fitting, the depth point cloud of the surface of the incomplete tray included in the image to be detected is obtained, and the pose of the tray is determined based on the depth point cloud of the surface of the incomplete tray to obtain the spatial position of the tray. Based on the spatial location of the forklift, determine the spatial position of the fork tips; The positional relationship between the pallet and the fork tines is determined based on the spatial position of the pallet and the spatial position of the fork tines. According to the aforementioned positional relationship, adjust the position of the fork tines and control the adjusted fork tines to insert into the pallet.

2. The method according to claim 1, characterized in that, The determination of the target tray detection result corresponding to the image to be detected based on a preset correspondence between image features and tray detection results includes: The image to be detected is input into a preset tray detection model, and the output result of the tray detection model is obtained as the target tray detection result corresponding to the image to be detected. The tray detection model is trained based on the first type of sample images and the second type of sample images.

3. The method according to claim 2, characterized in that, The training methods for the tray detection model include: Acquire first-class sample images and second-class sample images with added sample labels, and train a preset initial model using the acquired first-class sample images and second-class sample images; wherein, the sample labels are tray detection results representing the presence of trays; When the initial model meets the preset conditions, training stops, and the tray detection model is obtained.

4. A pallet detection device, characterized in that, The device includes: An image acquisition module is used to acquire the image to be detected acquired by an image acquisition device; wherein, the image acquisition device is located at the tip of one fork of the forklift, and the camera faces directly in front of the tip of the fork; An image detection module is used to determine the target tray detection result corresponding to the image to be detected based on a preset correspondence between image features and tray detection results; wherein, the correspondence is determined based on a first type of sample image including complete tray samples and a second type of sample image including incomplete tray samples; The relationship determination module is used to: if the target pallet detection result indicates the presence of a pallet, acquire a depth map of the image to be detected; and perform planar fitting on the pallet based on the depth map to obtain the spatial position of the pallet, including: if the pallet included in the image to be detected is incomplete, then after planar fitting, obtain the depth point cloud of the surface of the incomplete pallet included in the image to be detected; determine the pose of the pallet based on the depth point cloud of the surface of the incomplete pallet to obtain the spatial position of the pallet; determine the spatial position of the fork tips based on the spatial position of the forklift; and determine the positional relationship between the pallet and the fork tines based on the spatial position of the pallet and the spatial position of the fork tines. The adjustment control module is used to adjust the position of the fork tines according to the positional relationship, and control the adjusted fork tines to insert into the pallet.

5. The apparatus according to claim 4, characterized in that, The image detection module is specifically used for: The image to be detected is input into a preset tray detection model, and the output result of the tray detection model is obtained as the target tray detection result corresponding to the image to be detected. The tray detection model is trained based on the first type of sample images and the second type of sample images.

6. The apparatus according to claim 5, characterized in that, The device further includes: The model training module is used to acquire the first type of sample images and the second type of sample images with sample labels added, and to train the preset initial model using the acquired first type of sample images and the second type of sample images. The sample labels mentioned above represent the pallet detection results where a pallet is present; when the initial model meets the preset conditions, training stops, and the pallet detection model is obtained.

7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-3.

Images