Forklift control method and device, vehicle machine, forklift and storage medium

Abstract

The invention discloses a forklift control method, a forklift control device, a vehicle machine, a forklift and a computer readable storage medium. Relates to the technical field of automatic control of forklifts. Inputting the image into a preset target detection model for processing so as to determine whether a column exists in the image; under the condition that the stand columns exist in the image, whether a tray exists in the image or not is determined according to the number and the distribution condition of the stand columns; under the condition that the tray exists in the image, obtaining the distance between the tray and the binocular camera; and controlling the forklift to move according to the distance between the tray and the binocular camera. According to the forklift, the binocular camera with lower cost is used for replacing a depth camera, and the manufacturing cost of the forklift can be reduced.

Description

Technical Field

[0001] The present application relates to the technical field of forklift control, and more specifically, to a forklift control method, a forklift control device, a vehicle computer, a forklift, and a computer-readable storage medium. Background Art

[0002] In the field of warehousing and logistics, unmanned forklifts have higher loading and unloading efficiency than traditional forklifts that rely on manual operation. At present, unmanned forklifts obtain image information and depth information of the photographed object through depth cameras, and then obtain the position and size information of the photographed object in three-dimensional space, and process the position and size information through detection and segmentation models to determine the distance between the pallet and the unmanned forklift. The unmanned forklift determines the distance it needs to move based on the distance between the pallet and itself, so as to accurately lift the pallet with a forklift. However, the high cost of depth cameras is not conducive to the widespread application of unmanned forklifts. Summary of the invention

[0003] Embodiments of the present application provide a forklift control method, a forklift control device, a vehicle computer, a forklift, and a computer-readable storage medium.

[0004] The control method of the forklift of the embodiment of the present application includes: using the binocular camera to capture an image; inputting the image into a preset target detection model for processing to determine whether there are pillars in the image; if the pillars exist in the image, determining whether there are pallets in the image based on the number and distribution of the pillars; if the pallet exists in the image, obtaining the distance between the pallet and the binocular camera; and controlling the movement of the forklift according to the distance between the pallet and the binocular camera.

[0005] The control device of a forklift in an embodiment of the present application includes a binocular camera and a processing module, wherein the binocular camera is used to capture images; the processing module is used to: input the image into a preset target detection model for processing to determine whether there are pillars in the image; if the pillars exist in the image, determine whether there are pallets in the image based on the number and distribution of the pillars; if the pallet exists in the image, obtain the distance between the pallet and the binocular camera; and control the movement of the forklift based on the distance between the pallet and the binocular camera.

[0006] An embodiment of the present application also provides a vehicle computer, which includes a memory and a processor, wherein the memory is configured to store a computer program, and when the processor executes the computer program, it implements the following control method: using the binocular camera to capture an image; inputting the image into a preset target detection model for processing to determine whether there are pillars in the image; if the pillars exist in the image, determining whether there are pallets in the image based on the number and distribution of the pillars; if the pallet exists in the image, obtaining the distance between the pallet and the binocular camera; and controlling the movement of the forklift based on the distance between the pallet and the binocular camera.

[0007] The embodiment of the present application also provides a forklift, the forklift includes a forklift control device and a vehicle computer, the forklift control device includes a binocular camera and a processing module, the binocular camera is used to shoot an image; the processing module is used to: input the image into a preset target detection model for processing to determine whether there is a column in the image; if the column exists in the image, determine whether there is a pallet in the image according to the number and distribution of the columns; if the pallet exists in the image, obtain the distance between the pallet and the binocular camera; and control the movement of the forklift according to the distance between the pallet and the binocular camera. The vehicle computer includes a memory and a processor, the memory is configured to store a computer program, and the processor implements the following control method when executing the computer program: use the binocular camera to shoot an image; input the image into a preset target detection model for processing to determine whether there is a column in the image; if the column exists in the image, determine whether there is a pallet in the image according to the number and distribution of the columns; if the pallet exists in the image, obtain the distance between the pallet and the binocular camera; and control the movement of the forklift according to the distance between the pallet and the binocular camera.

[0008] The present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the following control method when executed by a processor: using the binocular camera to capture an image; inputting the image into a preset target detection model for processing to determine whether there are pillars in the image; if the pillars exist in the image, determining whether there are pallets in the image based on the number and distribution of the pillars; if the pallet exists in the image, obtaining the distance between the pallet and the binocular camera; and controlling the movement of the forklift based on the distance between the pallet and the binocular camera.

[0009] In the forklift control method, forklift control device, vehicle computer, forklift and computer-readable storage medium provided by the present application, the image captured by the binocular camera of the forklift is processed by the acquired target detection model to determine whether there are pillars in the image. When there are pillars in the image, it is determined whether there are pallets in the image based on the number and distribution of the pillars. When there are pallets in the image, the distance between the pallet and the binocular camera is determined based on the binocular ranging algorithm, so that the movement of the forklift is controlled according to the distance between the pallet and the binocular camera to ensure that the forklift can accurately lift the pallet. The present application replaces the depth camera with a lower-cost binocular camera, and also achieves the purpose of the forklift automatically and accurately lifting the pallet, greatly reducing the manufacturing cost of the forklift.

[0010] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0012] Figure 1 is a flow chart of a control method for a forklift in some embodiments of the present application;

[0013] Figure 2 is a schematic structural diagram of a forklift control device in some embodiments of the present application;

[0014] Figure 3 is a schematic diagram of the structure of a forklift and a pallet in some embodiments of the present application;

[0015] Figure 4 It is a flow chart of inputting an image into a preset target detection model for processing in a forklift control method of some embodiments of the present application to determine whether there is a pillar in the image;

[0016] Figure 5 It is a schematic diagram of the structure of a preset target detection model in some implementation modes of the present application;

[0017] Figure 6 It is a schematic diagram of the structure of a downsampling module in a preset target detection model in some implementation modes of the present application;

[0018] Figure 7 is a schematic diagram of the structure of a Bottleneck module in a preset target detection model in some implementation modes of the present application;

[0019] Figure 8 It is a schematic diagram of the structure of the preset improvement module of some implementation methods of the present application;

[0020] Fig. 9 It is a flow chart of inputting a first down-sampled feature map, a second down-sampled feature map, and a third down-sampled feature map into a feature fusion network to obtain a first fused feature map, a second fused feature map, and a third fused feature map in a control method for a forklift in some embodiments of the present application;

[0021] Fig.10 It is a flow chart of inputting the third down-sampled feature map, the second transition map and the first transition map into a preset improvement module in a forklift control method of some embodiments of the present application to obtain the first fusion feature map, the third transition map and the fourth transition map;

[0022] Fig.11 A schematic flow chart of a forklift control method according to some embodiments of the present application, wherein a preset improvement module is used to perform feature extraction processing on a third down-sampled feature map, a first transition map, and a second transition map to obtain an eighth transition map corresponding to the third down-sampled feature map, the first transition map, and the second transition map;

[0023] Fig.12 It is a schematic diagram of a process of inputting a first fused feature map, a second fused feature map, and a third fused feature map into a result conversion network to determine whether there is a pillar in an image in a forklift control method of some embodiments of the present application;

[0024] Fig.13 It is a flowchart of determining whether there is a pallet in an image according to the number and distribution of columns in a forklift control method of some embodiments of the present application when there are columns in the image;

[0025] Fig.14 It is a schematic diagram of a process of determining whether there is a pallet in an image according to the distribution of a central column and an edge column in a control method of a forklift in some embodiments of the present application;

[0026] Fig.15 It is a schematic diagram of the structure of a tray in some embodiments of the present application;

[0027] Fig.16 is a flowchart of a forklift control method according to other embodiments of the present application;

[0028] Fig.17 It is a flowchart of adjusting the position and posture of a forklift according to the relative position of a binocular camera and a pallet in a forklift control method according to other embodiments of the present application when a pallet exists in an image;

[0029] Fig.18 is a schematic diagram of the relative position between a forklift and a pallet in some embodiments of the present application;

[0030] Fig.19 It is a schematic diagram of a process of obtaining the distance between a pallet and a binocular camera when a pallet exists in an image in a forklift control method according to other embodiments of the present application;

[0031] Fig. 20 is a schematic diagram of the distance between the tray and the binocular camera in some embodiments of the present application;

[0032] Fig.21 is a schematic diagram of the relative position between a forklift and a pallet in some embodiments of the present application;

[0033] Fig. 22 is a schematic diagram of the relative position between a forklift and a pallet in some embodiments of the present application;

[0034] Fig.23 is a schematic diagram of the relative position between a forklift and a pallet in some embodiments of the present application;

[0035] Fig.24 It is a schematic diagram of a process of controlling the movement of a forklift according to the distance between a pallet and a binocular camera in a forklift control method in some embodiments of the present application;

[0036] Fig.25 is a schematic structural diagram of a forklift according to certain embodiments of the present application;

[0037] Fig.26 It is a schematic diagram of the connection status of a computer-readable storage medium and a processor in certain embodiments of the present application.

[0038] Description of main component symbols:

[0039] Forklift 100;

[0040] A control device 10 for a forklift;

[0041] Binocular camera 11; processing module 12;

[0042] Processor 20;

[0043] Computer readable storage medium 200; computer program 202;

[0044] Car machine 30;

[0045] Pallet 40; pallet body 41; fork 42; column 43;

[0046] Center column 431; geometric center of center column 4311; edge column 432; geometric center of edge column 4321. DETAILED DESCRIPTION

[0047] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the embodiments of the present application.

[0048] In the field of warehousing and logistics, unmanned forklifts have higher loading and unloading efficiency than traditional forklifts that rely on manual operation. At present, unmanned forklifts obtain image information and depth information of the photographed object through depth cameras, and then obtain the position and size information of the photographed object in three-dimensional space, and process the position and size information from the depth camera through a detection and segmentation model to determine the distance between the pallet and the unmanned forklift. The unmanned forklift determines the distance it needs to move based on the distance between the pallet and itself, thereby ensuring that the unmanned forklift can accurately lift the pallet. However, the cost of depth cameras is relatively high, which is not conducive to the widespread use of unmanned forklifts. How to solve the problem of difficulty in promoting unmanned forklifts due to the high cost of depth cameras has become a difficult problem that technicians in this field urgently need to solve. In order to solve this problem, the present application provides a forklift control method (such as Figure 1 and Fig.16 As shown), forklift control device (such as Figure 2 and Fig.25 As shown), vehicle machine 30 (as shown Fig.25 As shown), forklift 100 (as Figure 3 and Fig.25 ) and computer readable storage medium 200 (as shown Fig.26 shown).

[0049] See also Figures 1 to 3 , the control method of the forklift in the embodiment of the present application includes:

[0050] 03: Use the binocular camera 11 to capture images;

[0051] 04: Input the image into a preset target detection model for processing to determine whether there is a pillar 43 in the image;

[0052] 05: When there are pillars 43 in the image, determine whether there is a tray 40 in the image according to the number and distribution of the pillars 43;

[0053] 07: When the tray 40 exists in the image, obtain the distance between the tray 40 and the binocular camera 11; and

[0054] 08: Control the movement of the forklift 100 according to the distance between the pallet 40 and the binocular camera 11.

[0055] The control method of the forklift described above can be applied to the control device 10 of the forklift 100. The control device 10 of the forklift of the embodiment of the present application includes a binocular camera 11 and a processing module 12, wherein the binocular camera 11 is used to capture images. The processing module 12 is used to: input the image into a preset target detection model for processing to determine whether there is a column 43 in the image; if there is a column 43 in the image, determine whether there is a pallet 40 in the image according to the number and distribution of the columns 43; if there is a pallet 40 in the image, obtain the distance between the pallet 40 and the binocular camera 11; and control the movement of the forklift 100 according to the distance between the pallet 40 and the binocular camera 11.

[0056] The control device 10 of the forklift 100 is one of the core control components installed inside the forklift 100, and is responsible for managing and coordinating the operation of various parts of the forklift 100. The control device 10 of the forklift 100 controls the working state of various components of the forklift 100, such as the engine, gearbox, brake system, steering system, etc., by collecting, processing and analyzing data from various sensors, such as data from sensors such as cameras, ranging radars, and infrared ranging devices, so as to ensure the safe and efficient operation of the forklift 100. The control device 10 of the forklift 100 controls the overall performance of the forklift 100 by adjusting the parameters inside the forklift 100. In the present application, the control device 10 of the forklift 100 is used to perform target detection on the image captured by the binocular camera 11 through a preset target detection model, and further process the image through the processing module 12 according to the target detection result of the image to determine the overall movement of the forklift 100.

[0057] More specifically, the control device 10 of the forklift 100 includes a binocular camera 11 and a processing module 12, the binocular camera 11 is used to execute 03, and the processing module is used to execute 04, 05, 07 and 08. The binocular camera 11 is a camera composed of two independent cameras and has a structure similar to human eyes. The binocular camera 11 can simultaneously capture images of the same scene from two perspectives. The binocular camera 11 is generally used in the fields of robot vision, intelligent driving, drones, measurement, security monitoring, etc. In the present application, the binocular camera 11 is arranged on the control device 10 of the forklift 100 and is configured to capture images around the control device 10 of the forklift 100. The processing module 12 is the core module of the control device 10 of the forklift 100. The processing module 12 is used to process the image captured by the binocular camera 11 through a preset target detection model to determine whether there is a column 43 in the image captured by the binocular camera 11, and determine whether there is a pallet 40 in the image captured by the binocular camera 11 when there is a column 43. When the pallet 40 exists in the image captured by the binocular camera 11, the processing module 12 can obtain the distance between the pallet 40 and the binocular camera 11, thereby controlling the movement of the forklift 100 according to the distance between the pallet 40 and the binocular camera 11 to ensure that the forklift 100 can accurately lift the pallet 40.

[0058] More specifically, in method 03 and method 04, the preset target detection model has no special requirements for the input image, so the binocular camera 11 can use the single camera function to capture the image, that is, only one of the two independent cameras is activated to capture the image, so as to reduce the overall energy consumption of the control device 10 of the forklift 100. Figure 3 Taking a three-column pallet as an example, since the pallet 40 includes a pallet body 41 and columns 43, the pallet body 41 is a flat structure in the pallet 40 for supporting goods, and the columns 43 extend from the side of the pallet body 41 away from the side carrying the goods. The number of columns 43 is usually multiple, and the multiple columns 43 are spaced apart from each other. During the target detection process, the pallet body 41 is easily confused with rectangular goods. If the processing module 12 uses the pallet body 41 as the detection target to identify the pallet 40, target recognition errors are likely to occur, affecting the accuracy of controlling the movement of the forklift 100. Therefore, in the present application, when the processing module 12 performs target detection on the image through a preset target detection model, it can detect the columns 43 ( Figure 3The recognition of the pallet 40 is realized by identifying the portion within the dashed box in the image). In method 05, when the processing module 12 determines that there are columns 43 in the image, it is also necessary to determine that there is a pallet 40 in the image based on the number and distribution of the columns 43. The three-column pallet is a commonly used pallet in the warehousing and logistics process. Therefore, taking the three-column pallet as an example, the processing module 12 determines that there is a pallet 40 in the image when the number of columns 43 in the image is three and the distribution of these three columns 43 conforms to the distribution of the columns 43 of the three-column pallet. In methods 07 and 08, when the processing module 12 determines that there is a pallet 40 in the image, in one example, the binocular camera 11 can use the binocular camera mode to obtain images from two perspectives, thereby helping the processing module 12 obtain the distance between the pallet 40 and the binocular camera 11. In addition, in the present application, the distance between the pallet 40 and the binocular camera 11 can also be obtained by other means, such as by using a radar or a distance measuring unit to obtain the distance between the pallet 40 and the binocular camera 11, that is, first sending and receiving radar signals through the radar or distance measuring unit, and then obtaining the distance between the pallet 40 and the binocular camera 11 according to the distance, time and speed formula. The method by which the processing module 12 obtains the distance between the pallet 40 and the binocular camera 11 will be described in detail below. When the distance between the pallet 40 and the binocular camera 11 is obtained, the processing module 12 controls the movement of the forklift 100 according to the distance between the pallet 40 and the binocular camera 11 to ensure that the forklift 100 can accurately lift the pallet 40, that is, to achieve control of the forklift 100.

[0059] In certain embodiments, see Figure 2 , Figure 4 , Figure 5 , Figure 6 and Figure 7 ,The preset target detection model includes a backbone network, a feature fusion network and a result conversion network. The backbone network includes multiple downsampling modules. 04: Input the image into the preset target detection model for processing to determine whether there is a pillar 43 in the image, including:

[0060] 041: Input the image into the backbone network, and perform downsampling processing through multiple downsampling modules to obtain multiple downsampling feature maps, where the multiple downsampling feature maps include a first downsampling feature map P3, a second downsampling feature map P4, and a third downsampling feature map P5;

[0061] 043: Input the first down-sampled feature map P3, the second down-sampled feature map P4, and the third down-sampled feature map P5 into the feature fusion network to obtain the first fused feature map Q3, the second fused feature map Q4, and the third fused feature map Q5; and

[0062] 045: Input the first fused feature map Q3, the second fused feature map Q4 and the third fused feature map Q5 into the result transformation network to determine whether there is a pillar 43 in the image.

[0063] The above-mentioned forklift control method can be applied to the control device 10 of the forklift 100, and the processing module 12 is used to: input the image into the backbone network, and obtain multiple down-sampling feature maps after down-sampling processing by multiple down-sampling modules, and the multiple down-sampling feature maps include a first down-sampling feature map P3, a second down-sampling feature map P4 and a third down-sampling feature map P5; input the first down-sampling feature map P3, the second down-sampling feature map P4 and the third down-sampling feature map P5 into the feature fusion network to obtain a first fused feature map Q3, a second fused feature map Q4 and a third fused feature map Q5; and input the first fused feature map Q3, the second fused feature map Q4 and the third fused feature map Q5 into the result conversion network to determine whether there is a pillar 43 in the image.

[0064] Specifically, the preset target detection model includes a backbone network, a feature fusion network and a result conversion network. In method 041, the input image is downsampled by five downsampling modules in the backbone network in turn, and downsampled feature maps of 2 times, 4 times, 8 times, 16 times and 32 times are obtained respectively. That is, the backbone network in the implementation scheme of the present application includes a downsampling module. The processing module 12 downsamples the image through the backbone network of the preset target detection model. The downsampling module in the present application uses the C2f module (Cross Channel Feature Fusion Module). The C2f module is a module used for feature extraction and fusion in a deep learning network. The internal structure of the C2f module is as follows Figure 6 As shown in Figure 1, the C2f module includes the CBS module, the Split module, and the Bottleneck module. The C2f module is designed to enhance feature representation in the convolutional neural network (CNN) and improve the performance of the convolutional neural network by effectively combining feature information from different channels. The core idea of ​​the C2f module is to divide the input feature map into two parts: a direct path and a processing path, and then process the features differently through these two paths, and finally fuse the processed features. The internal structure of the Bottleneck module in the C2f module is shown in Figure 1. Figure 7As shown, the Bottleneck module is a commonly used structure in deep neural networks and is often used in the ResNet (residual network) architecture. The design purpose of the Bottleneck module is to reduce the number of parameters and computational complexity of the model while maintaining network performance. The Bottleneck module includes the CBS module. Whether it is the CBS module in the C2f module or the CBS module in the Bottleneck module, it includes three layers of structures: convolution kernel (Convolution), batch normalization layer (Batch Normalization) and activation function layer (Activation). S represents the step size during the convolution operation. S=1 means that during the convolution process, the convolution kernel moves one pixel on the image each time. K represents the size of the convolution kernel. K=1 and K=3 represent 1*1 convolution kernel and 3*3 convolution kernel respectively. The Split module represents the segmentation operation of the input image. For example, an image with 4 channels is converted into two images with 2 channels. For example, please combine Figure 6 , when image 50 passes through the C2f module, the first CBS module will process image 50 first to reduce the number of channels of image 50 and improve the processing speed of subsequent modules. The Split module is a commonly used image processing module in the fields of computer vision and deep learning. The main function of the Split module is to segment the input image or feature map to adapt to different application scenarios and task requirements. After reducing the number of channels, image 50 is segmented by the Split module. A part of image 50 (hereinafter referred to as: image 501) will be processed by multiple Bottleneck modules. The Bottleneck module includes a series of convolution operations, such as Figure 7 As shown, it includes 1x1 and 3x3 convolution kernels. When image 501 passes through the Bottleneck module, the number of channels of the feature map will be reduced by the 1x1 convolution kernel, and then the number of channels will be increased in the 3x3 convolution kernel, so that the features are extracted by the Bottleneck module. The C2f module increases the nonlinearity of the network by setting multiple Bottleneck modules, that is, the downsampling process can reduce the size and resolution of the image, that is, by first reducing the number of channels of image 501 and then increasing the number of channels of image 501, the features of image 501 can be better extracted, and the number of pixels in the processing process can be reduced to improve the image processing efficiency. Finally, the C2f module merges and splices the processed image 501 with another part of the image 50 that retains the original information (another part of the image 50 that has not been processed by the Bottleneck module) to obtain image 502. Image 502 retains both the original features of image 50 and the features processed by multiple Bottleneck modules, thereby improving the accuracy of subsequent image feature processing.

[0065] It is understandable that, in addition to being able to perform feature fusion on the input image, the C2f module can also reduce the image pixels that need to be processed during the image processing process through the internal CBS module, thereby reducing the storage space required for the image and improving the image processing speed. In addition, in the process of reducing image pixels, the image resolution will also decrease. More specifically, in the process of reducing image pixels, the pixel value of a certain point in the image will be averaged with the pixel value of its neighboring points, and the high-frequency noise in the image will gradually become smoother, while the low-frequency information in the image will be retained. Since high-frequency noise is usually redundant information in the image, and low-frequency information is usually objects or large areas of background in the image, each downsampling process can filter out redundant information in the image and improve the accuracy of target recognition.

[0066] In method 043 and method 045, the processing module 12 inputs the first down-sampled feature map P3 (8 times down-sampled feature map of the input image), the second down-sampled feature map P4 (16 times down-sampled feature map of the input image) and the third down-sampled feature map P5 (32 times down-sampled feature map of the input image) obtained through the backbone network processing into the feature fusion network for further fusion processing to improve the accuracy of the feature maps, and inputs the obtained first fused feature map Q3, second fused feature map Q4 and third fused feature map Q5 into the result transformation network. The result transformation network can realize the recognition of the target (column 43) based on the input first fused feature map, second fused feature map and third fused feature map. This process will be described in detail below.

[0067] In certain embodiments, see Figure 2 , Figure 5 and Fig. 9 , the feature fusion network includes a preset improvement module ( Figure 5 CSA Block in the CSA Block), 043: input the first down-sampled feature map P3, the second down-sampled feature map P4 and the third down-sampled feature map P5 into the feature fusion network to obtain the first fused feature map Q3, the second fused feature map Q4 and the third fused feature map Q5, including:

[0068] 0431: after upsampling the third down-sampled feature map P5, the third down-sampled feature map P5 is fused and added with the second down-sampled feature map P4 to obtain a first transition map P6;

[0069] 0432: After upsampling the first transition map P6, the first downsampling feature map P3 is fused and added to obtain a second transition map P7;

[0070] 0433: Input the second transition map P7, the first transition map P6 and the third down-sampled feature map P5 into the preset improvement module to obtain the first fusion feature map Q1, the third transition map Q2 and the fourth transition map Q3;

[0071] 0434: After downsampling the first fused feature map Q1, it is fused and added with the third transition map Q2 to obtain the second fused feature map Q4;

[0072] 0435: After downsampling the second fused feature map Q4, it is fused and added with the fourth transition map Q3 to obtain the third fused feature map Q5.

[0073] The above-mentioned forklift control method can be applied to the control device 10 of the forklift 100, and the processing module 12 is used for: after upsampling the third down-sampled feature map P5, it is fused and added with the second down-sampled feature map P4 to obtain a first transition map P6; after upsampling the first transition map P6, it is fused and added with the first down-sampled feature map P3 to obtain a second transition map P7; the second transition map P7, the first transition map P6 and the third down-sampled feature map P5 are input into the preset improvement module (CSA Block) to obtain a first fused feature map Q1, a third transition map Q2 and a fourth transition map Q3; after downsampling the first fused feature map Q1, it is fused and added with the third transition map Q2 to obtain a second fused feature map Q4; after downsampling the second fused feature map Q4, it is fused and added with the fourth transition map Q3 to obtain a third fused feature map Q5.

[0074] Specifically, upsampling can improve the resolution of the image, thereby restoring or generating more detailed image details. Fusion addition refers to adding the pixel values ​​of corresponding positions of two pictures of the same size and resolution to obtain a new picture. The fusion addition process can increase the original features of the picture. In method 0431 and method 0432, the processing module 12 performs upsampling on the third down-sampled feature map P5, and then fuses and adds it with the second down-sampled feature map P4 to obtain a first transition map P6. Compared with the third down-sampled feature map P5, the first transition map P6 restores the detailed information of the image that may be lost during the downsampling process, and the small objects or detailed features in the image are clearer. The processing module 12 also performs upsampling on the first transition map P6, and then fuses and adds it with the first down-sampled feature map P3 to obtain a second transition map P7. In method 0433, the third down-sampled feature map P5, the second transition map P7, and the first transition map P6 are images of different sizes and resolutions. The third down-sampled feature map P5, the second transition map P7, and the first transition map P6 are processed by a preset improvement module, so that the feature map output by the preset improvement module excludes high-frequency noise interference while not missing small-sized objects with a small number of pixels in the image, thereby ensuring the accuracy of target detection of the pillar 43. In methods 0434 and 0435, the processing module 12 performs down-sampling processing on the first fused feature map Q1 and the second fused feature map Q4 in the same manner and principle as above, which will not be repeated here.

[0075] More specifically, the NanoDet model is a high-performance lightweight target detection model. In the present application, the preset target detection model is an improved Nanodet model. The improvement is to use a preset improvement module (CSA Block module) to improve the PAN network of the original Nanodet model (the feature fusion network of the original Nanodet model) to obtain a new CSA-PAN network (the feature fusion network of the preset target detection model). The PAN network of the original Nanodet model will complete the upsampling and downsampling processing through linear interpolation. Compared with the improved feature fusion network, the PAN network of the original Nanodet model can also reduce the amount of calculation, but some feature information will be lost during the processing. Therefore, this application is Figure 5 The CSA Block module is added at the position shown to form a CSA-PAN network. The purpose of this is to filter the features lost in the linear interpolation process, that is, the upsampling process, and extract the area that the feature fusion network pays most attention to from the existing features, thereby reducing the loss of features in the upsampling process.

[0076] In certain embodiments, see Figure 2 , Figure 5 , Figure 8 and Fig.10 , 0433: Input the second transition map P7, the first transition map P6 and the third down-sampled feature map P5 into the preset improvement module to obtain the first fusion feature map Q1, the third transition map Q2 and the fourth transition map Q3, including:

[0077] 04331: using a preset improvement module to perform feature extraction processing on the second transition map P7, the first transition map P6 and the third down-sampled feature map P5 respectively to obtain eighth transition maps V1, V2, V3 corresponding to the second transition map P7, the first transition map P6 and the third down-sampled feature map P5 respectively;

[0078] 04333: The eighth transition map V1 corresponding to the third down-sampling feature map P5 is fused with the third down-sampling feature map P5 to obtain the fourth transition map Q3, the eighth transition map V2 corresponding to the first transition map P6 is fused with the first transition map P6 to obtain the third transition map Q2, and the eighth transition map V3 corresponding to the second transition map P7 is fused with the second transition map P7 to obtain the first fused feature map Q1.

[0079] The above-mentioned forklift control method can be applied to the control device 10 of the forklift 100, and the processing module 12 is used to: use a preset improvement module to perform feature extraction processing on the second transition map P7, the first transition map P6 and the third down-sampled feature map P5 respectively to obtain the eighth transition maps V1, V2, V3 corresponding to the second transition map P7, the first transition map P6 and the third down-sampled feature map P5 respectively; fuse the eighth transition map V1 corresponding to the third down-sampled feature map P5 with the third down-sampled feature map P5 to obtain the fourth transition map Q3, fuse the eighth transition map V2 corresponding to the first transition map P6 with the first transition map P6 to obtain the third transition map Q2, and fuse the eighth transition map V3 corresponding to the second transition map P7 with the second transition map P7 to obtain the first fused feature map Q1.

[0080] Specifically, the processing module 12 uses the preset improvement module to perform feature extraction processing on the third down-sampled feature map P5, the first transition map P6 and the second transition map P7 to obtain the eighth transition map ( Figure 8 V1, V2, V3 in the figure). The processing module 12 fuses the eighth transition map V1 corresponding to the third down-sampled feature map P5 with the third down-sampled feature map P5 to obtain a fourth transition map Q3, fuses the eighth transition map V2 corresponding to the first transition map P6 with the first transition map P6 to obtain a third transition map Q2, and fuses the eighth transition map V3 corresponding to the second transition map P7 with the second transition map P7 to obtain a first fused feature map Q1, thereby ensuring that the third down-sampled feature map P5 can retain the original information of the third down-sampled feature map P5 after the feature extraction of the preset improvement module, ensuring that the first transition map P6 can retain the original information of the first transition map P6 after the feature extraction of the preset improvement module, and ensuring that the second transition map P7 can retain the original information of the second transition map P7 after the feature extraction of the preset improvement module.

[0081] See also Figure 2 , Figure 5 , Figure 8 and Fig.11 In some embodiments, the preset improvement module includes a plurality of first-class convolutional layers ( Figure 8 1*1conv, c' module, i.e. 1*1 convolution kernel), multiple second-class convolution layers ( Figure 8 3*3conv, c' / 2 module, i.e. 3*3 convolution kernel), spatial information collection layer ( Figure 8 Global Average Pooling module in ), normalization layer ( Figure 8 BN in ), activation layer ( Figure 8 RELU in the layer) and weighted layer ( Figure 804331: using a preset improvement module to perform feature extraction processing on the second transition map P7, the first transition map P6 and the third down-sampled feature map P5 to obtain eighth transition maps V1, V2, V3 corresponding to the second transition map P7, the first transition map P6 and the third down-sampled feature map P5, respectively, including:

[0082] 043311: input the third down-sampled feature map P5, the first transition map P6 and the second transition map P7 as input maps to the preset improvement module respectively;

[0083] 043312: After the input image is segmented, it is processed by the first type of convolution layer and the second type of convolution layer to obtain the fifth transition image F1 and the sixth transition image F0;

[0084] 043313: After the fifth transition map F1 and the sixth transition map F0 are fused and added, and then processed by the second type of convolutional layer, the seventh transition map F2 is obtained;

[0085] 043314: after fusing and adding the fifth transition map F1 and the seventh transition map F2, the first transition map a1 and the second transition map a2 are obtained by sequentially processing through a spatial information collection layer, two first-class convolutional layers, a normalization layer, an activation layer, two first-class convolutional layers, and an assignment weight layer; and

[0086] 043315: After weighted fusion of the fifth transition map F1 and the first assignment weight map a1, and weighted fusion of the seventh transition map F2 and the second assignment weight map a2, and then processed by the first type of convolutional layer, the eighth transition maps V1, V2, and V3 are obtained.

[0087] The forklift control method can be applied to the control device 10 of the forklift 100, and the processing module 12 is used to: input the third down-sampled feature map P5, the first transition map P6 and the second transition map P7 as input maps to the preset improvement module respectively; after the input map is segmented and processed, it is processed by the first type of convolution layer and the second type of convolution layer to obtain the fifth transition map F1 and the sixth transition map F0; after the fifth transition map F1 and the sixth transition map F0 are fused and added, they are processed by the second type of convolution layer to obtain the seventh transition map F2; After the fifth transition map F1 and the seventh transition map F2 are fused, added and segmented, they are processed in sequence by a spatial information collection layer, two first-class convolutional layers, a normalization layer, an activation layer, two first-class convolutional layers and an assignment weight layer to obtain a first assignment weight map a1 and a second assignment weight map a2; and after weighted fusion of the fifth transition map F1 and the first assignment weight map a1, and weighted fusion of the seventh transition map F2 and the second assignment weight map a2, respectively, and then processed by the first-class convolutional layer, the eighth transition maps V1, V2, and V3 are obtained.

[0088] Specifically, the preset improvement module first divides the input image into two groups of images to reduce the number of parameters. Both groups are processed by the first type of convolution layer and the second type of convolution layer to obtain the fifth transition image F1 and the sixth transition image F0. After that, the processing module 12 merges and adds the fifth transition image F1 and the sixth transition image F0, and then processes them by the second type of convolution layer to obtain the seventh transition image F2, so that the seventh transition image F2 can retain the image semantic information from the two split groups and expand the receptive field of the target detection model. After that, the processing module 12 converts the fifth transition image ( Figure 8 F1 in) and the seventh transition diagram ( Figure 8 F2) in fusion and addition:

[0089]

[0090] Among them, X i Represents the input image information, F i The processing module 12 segments the result of the fusion addition of the fifth transition map and the seventh transition map, and sequentially processes the spatial information collection layer, two first-class convolutional layers, the normalization layer, the activation layer, two first-class convolutional layers, and the weight assignment layer to obtain the first weight assignment map ( Figure 8 a1 in) and the second assignment weight map ( Figure 8 The spatial information collection layer is used to collect global spatial information (i.e. the positions of all pixels in the image) by generating channel statistics image data:

[0091]

[0092] α and β represent the point-by-point spatial information of the height and width of the feature map, respectively. Finally, the processing module 12 weightedly fuses the fifth transition map F1 with the first assignment weight map a1, and weighted fuses the seventh transition map F2 with the second assignment weight map a2, and then processes the first type of convolutional layer to obtain the eighth transition map. The channel-wise soft attention formula is used to aggregate the weighted fusion represented by the basis array, that is, the weighted fusion of the fifth transition map F1 and the first assignment weight map a1, and the seventh transition map F2 and the second assignment weight map a2. The channel-wise soft attention formula can capture the key information in the feature map. The calculation formula for the feature map of the cth channel is:

[0093]

[0094] Among them, a i is the assignment weight, and the specific calculation method is as follows:

[0095]

[0096] It represents that the global spatial information S performs two 1x1 convolutions on the weight values ​​of the cth channel of the image. At the same time, between these two 1x1 convolutions, the normalization layer and the activation layer serve to accelerate the training process and enhance the nonlinear expression ability of the model.

[0097] See also Figure 2 , Figure 5 and Fig.12 In some embodiments, 045: inputting the first fused feature map Q3, the second fused feature map Q4, and the third fused feature map Q5 into the result transformation network to determine whether there is a pillar 43 in the image, including:

[0098] 0451: Obtain the pixel value of each pixel in the first fused feature map Q3, the second fused feature map Q4 and the third fused feature map Q5;

[0099] 0453: Calculate the probability value of each pixel point in the first fused feature map Q3, the second fused feature map Q4 and the third fused feature map Q5 being the column 43 based on the pixel value of each pixel point in the first fused feature map Q3, the second fused feature map Q4 and the third fused feature map Q5, and use it as the weight value; and

[0100] 0455: Determine whether there is a pillar 43 in the image based on the weight value of each pixel point in the first fusion feature map Q3, the second fusion feature map Q4 and the third fusion feature map Q5.

[0101] The above-mentioned forklift control method can be applied to the control device 10 of the forklift 100, and the processing module 12 is used to: obtain the pixel value of each pixel point in the first fusion feature map Q3, the second fusion feature map Q4 and the third fusion feature map Q5; calculate the probability value of each pixel point in the first fusion feature map Q3, the second fusion feature map Q4 and the third fusion feature map Q5 being a pillar 43 based on the pixel value of each pixel point in the first fusion feature map Q3, the second fusion feature map Q4 and the third fusion feature map Q5, and use it as a weight value; and determine whether there is a pillar 43 in the image according to the weight value of each pixel point in the first fusion feature map Q3, the second fusion feature map Q4 and the third fusion feature map Q5.

[0102] Specifically, the processing module 12 calculates the probability value of each pixel point in the first fused feature map Q3, the second fused feature map Q4 and the third fused feature map Q5 being a pillar 43 according to the pixel value of each pixel point in the first fused feature map Q3, the second fused feature map Q4 and the third fused feature map Q5, and uses the probability value as a weight value. For example, the processing module 12 stores a preset probability value calculation formula and a preset probability value range to calculate the probability value according to the pixel value. The processing module 12 determines the image area whose probability value in the first fused feature map Q3, the second fused feature map Q4 and the third fused feature map Q5 is within the preset probability value range according to the preset probability value range, and generates a target frame in the image area whose probability value is within the preset probability value range. If any feature map in the first fused feature map Q3, the second fused feature map Q4 and the third fused feature map Q5 generates a target frame, the processing module 12 determines that there is a pillar 43 in the image. If the first fused feature map Q3 , the second fused feature map Q4 , and the third fused feature map Q5 do not generate a target frame, the processing module 12 determines that there is no pillar 43 in the image.

[0103] It is understandable that in the field of warehousing and logistics, when the forklift 100 is performing target recognition of the column 43, the distance between the forklift 100 and the pallet 40 must be at least 3 meters. When the distance between the column 43 and the binocular camera 11 is more than 3 meters, the column 43 is a small target in the image, that is, the target detection of the column 43 is classified as the target detection of a small target. However, the existing Nanodet model often has problems such as missed detection and false detection of small targets. The reason is that the Nanodet model does not pay enough attention to the features of small target objects, especially during the downsampling operation of the Nanodet model, which will compress the feature information of the image, thereby causing the features of small target objects to be ignored, and then causing missed detection and false detection. This application selects the Nanodet model as the main framework of the target detection model of this application, and changes the backbone network of Nanodet to the backbone network of the YOLOV8 model, so as to ensure that the backbone network can extract rich image texture features. The feature fusion network adds a preset improvement module, which enables the feature fusion network to better capture the complex semantic information in the image and solve the problem of premature loss of image detail information during downsampling. The preset improvement module can also improve the target detection performance and maintain high precision and real-time performance of target detection when the parameter amount is extremely small. In this application, the preset target detection model can also use the Dice Loss+Focal Loss loss function combination to increase the convergence speed of the model. In general, in this application, the target detection model obtained after the improvement of NanoDet can achieve more accurate target detection and positioning, and has the characteristics of low cost, small model, fast processing speed, and is suitable for deployment on edge devices.

[0104] In certain embodiments, see Figure 2 , Figure 3 , Fig.13 and Fig.15 , the columns 43 include a central column 431 and an edge column 432, 05: determining whether there is a tray 40 in the image according to the number and distribution of the columns 43, including:

[0105] 051: when there are pillars 43 in the image, determine whether the number of pillars 43 is a preset number of pillars;

[0106] 053: When the number of columns 43 is the preset number of columns, determine the distribution of the central columns 431 and the edge columns 432; and

[0107] 055: Determine whether there is a tray 40 in the image based on the distribution of the central column 431 and the edge column 432.

[0108] The above-mentioned forklift control method can be applied to the control device 10 of the forklift 100, and the processing module 12 is used to: when there are pillars 43 in the image, determine whether the number of pillars 43 is a preset number of pillars; when the number of pillars 43 is a preset number of pillars, determine the distribution of the central pillars 431 and the edge pillars 432; and determine whether there is a pallet 40 in the image according to the distribution of the central pillars 431 and the edge pillars 432.

[0109] It is understandable that in the field of warehousing and logistics, the most commonly used pallet 40 is a three-column pallet, that is, the preset number of columns is three. Therefore, it is necessary to determine whether the pallet 40 exists in the image based on the number of columns 43 and the distribution of the columns 43. For example, when the number of columns 43 is three and the distribution of the center column 431 and the edge column 432 conforms to the appearance characteristics of a three-column pallet, the processing module 12 determines that the pallet 40 exists in the image. The center column 431 and the edge column 432 have the same shape and size, and the only difference is that the center column 431 and the edge column 432 are located at different positions on the pallet 40. The method for determining the existence of the pallet 40 will be described in more detail below.

[0110] See also Figure 2 , Figure 3 , Fig.14 and Fig.15 In some embodiments, the distance between the central column 431 and the edge column 432 is the distance between the geometric center 4311 of the central column 431 and the geometric center 4321 of the edge column 432. 055: Determining whether there is a tray 40 in the image according to the distribution of the central column 431 and the edge column 432 includes:

[0111] 0551: In the case where the preset number of pillars is three, the central pillar 431 is located between two edge pillars 432, and the distance between the central pillar 431 and any one edge pillar 432 is equal to the distance between the central pillar 431 and another edge pillar 432, it is determined that the tray 40 exists in the image; and

[0112] 0553: When the preset number of pillars is not three, or the central pillar 431 is not located between two edge pillars 432, or the distance between the central pillar 431 and any edge pillar 432 is not equal to the distance between the central pillar 431 and another edge pillar 432, it is determined that the tray 40 does not exist in the image.

[0113] The forklift control method can be applied to the control device 10 of the forklift 100, and the processing module 12 is used to: determine that the pallet 40 exists in the image when the preset number of pillars is three, the central pillar 431 is located between two edge pillars 432, and the distance between the central pillar 431 and any one edge pillar 432 is equal to the distance between the central pillar 431 and the other edge pillar 432; and determine that the pallet 40 does not exist in the image when the preset number of pillars is not three, or the central pillar 431 is not located between two edge pillars 432, or the distance between the central pillar 431 and any one edge pillar 432 is not equal to the distance between the central pillar 431 and the other edge pillar 432.

[0114] Please refer to Figure 3 as well as Fig.15 It is understandable that in order to ensure the accuracy of the processing module 12 in determining whether the tray 40 exists in the image, the positions of the central column 431 and the edge column 432 need to be determined based on the positions of the geometric centers of the central column 431 and the edge column 432 in the image. Fig.15For example, the geometric center of the edge column 432 is 4321, and the geometric center of the center column 431 is 4311. Therefore, when the number of the preset columns 43 is three, the center column 431 is located between two edge columns 432, and the distance between the center column 431 and any one of the edge columns 432 (i.e., the distance between the geometric center 4311 of the center column 431 and the geometric center 4321 of any one of the edge columns 432) is equal to the distance between the center column 431 and another edge column 432 (i.e., the distance between the geometric center 4311 of the center column 431 and the geometric center 4321 of another edge column 432), the processing module 12 determines that the tray 40 exists in the image. When the preset number of pillars 43 is not three, or the central pillar 431 is not located between two edge pillars 432, or the distance between the central pillar 431 and any edge pillar 432 is not equal to the distance between the central pillar 431 and another edge pillar 432, the processing module 12 determines that there is no tray 40 in the image.

[0115] See also Figure 2 and Fig.16 In some embodiments, the control method further comprises:

[0116] 06: When the pallet 40 is present in the image, the position and posture of the forklift 100 are adjusted according to the relative position between the binocular camera 11 and the pallet 40 .

[0117] The forklift control method described above can be applied to the control device 10 of the forklift 100 , and the processing module 12 is used to adjust the position and posture of the forklift 100 according to the relative position of the binocular camera 11 and the pallet 40 when there is a pallet 40 in the image.

[0118] It is understandable that the forks of the forklift 100 are usually arranged at the front end of the forklift 100. In order for the forklift 100 to successfully lift the pallet 40, the processing module 12 needs to determine the relative position of the binocular camera 11 and the pallet 40 based on the image captured by the binocular camera 11 in the dual-camera mode after determining that the pallet 40 exists in the image captured by the binocular camera 11. That is, determine the relative position of the forklift 100 and the pallet 40, and adjust the position and posture of the forklift 100 according to the relative position of the binocular camera 11 and the pallet 40.

[0119] See also Figure 2 , Fig.15 , Fig.17 and Fig.18 In some embodiments, the column 43 includes a central column 431. 06: According to the relative position of the binocular camera 11 and the pallet 40, the position and posture of the forklift 100 are adjusted, including:

[0120] 061: Detect the line between the center point Oc of the binocular camera 11 and the geometric center 4311 (Ob2) of the central column 431 and the preset axis Y b2 The angle α between them; and

[0121] 063: When the angle α is not equal to the preset angle value, adjust the position and posture of the forklift 100 until the angle α is equal to the preset angle value.

[0122] The forklift control method can be applied to the control device 10 of the forklift 100, and the processing module 12 is used to detect the connection between the center point Oc of the connection line of the binocular camera 11 and the geometric center 4311 of the central column 431 and the preset axis Y. b2 and when the angle α is not equal to the preset angle value, adjusting the position and posture of the forklift 100 until the angle α is equal to the preset angle value.

[0123] Specifically, Fig.18 4 is a top view of the forklift 100 and a cross-sectional view of the pallet 40. After obtaining the image captured by the binocular camera 11, the processing module 12 determines the relative position of the forklift 100 and the pallet 40 by detecting the angle α between the line connecting the center point of the binocular camera 11 and the geometric center 4311 of the center column 431 and the preset axis. In the process of capturing the image, the binocular camera 11 can determine the coordinate axis of the target point in the image according to the preset positioning program, and then obtain the preset axis. The preset axis is usually set to Fig.18 The axis Y b2 For example, when the preset angle value is set to zero, if the angle α is not equal to zero, the processing module 12 needs to adjust the position and posture of the forklift 100 until the angle α is equal to zero, thereby ensuring that the forklift 100 can face the pallet 40 and the two forks of the forklift 100 can just fit into the gap between the center column 431 and the edge column 432 of the pallet 40.

[0124] See also Figure 2 , Fig.15 , Fig.19 and Fig. 20 In some embodiments, the column 43 includes a central column 431, and the binocular distance measurement algorithm includes a binocular distance measurement formula. 07: Obtaining the distance between the tray 40 and the binocular camera 11 includes:

[0125] 071: Process the image according to a preset parallax algorithm to obtain the parallax between the binocular camera 11 and the geometric center 4311 of the central column 431; and

[0126] 073: Obtain the focal length and setting distance of the binocular camera 11, substitute the focal length, setting distance and parallax into the binocular distance measurement formula, and obtain the distance between the tray 40 and the binocular camera 11.

[0127] The above-mentioned forklift control method can be applied to the control device 10 of the forklift 100, and the processing module 12 is used to: process the image according to a preset parallax algorithm to obtain the parallax between the binocular camera 11 and the geometric center 4311 of the central column 431; and obtain the focal length and setting distance of the binocular camera 11, substitute the focal length, setting distance and parallax into the binocular distance measurement formula to obtain the distance between the pallet 40 and the binocular camera 11.

[0128] Specifically, the processing module 12 processes the image according to a preset parallax algorithm to obtain the parallax between the binocular camera 11 and the geometric center of the central column 431, and then substitutes the focal length, set distance and parallax into the binocular distance measurement formula to obtain the distance between the tray 40 and the binocular camera 11. More specifically, the binocular camera 11 is similar to the human eye. The human eye can perceive the distance of an object because the two eyes present different images of the same object, also known as "parallax." The farther the object is, the smaller the parallax; conversely, the larger the parallax. The size of the parallax corresponds to the distance between the object and the eye, which is why 3D movies can give people a three-dimensional sense of hierarchy. Please refer to Fig. 20 , O b2 is the geometric center 4311 of the center column 431, O L With O R They are the two optical centers of the binocular camera 11, point O b2 The two points imaged by the two camera sensors are O b1 and O b2 '(the imaging plane of the camera is rotated and placed in front of the lens), f is the focal length of the camera, B is the distance between the center points of the two cameras (i.e., the setting distance), Z is the distance between the tray 40 and the binocular camera 11, and point O is set b1 to b2 ' is L, then

[0129] L=B-(X L -X R );

[0130] According to the principle of similar triangles, we can get:

[0131]

[0132] After deformation, we can get:

[0133]

[0134] Among them, "X L -XR ” is the parallax, so the processing module 12 can obtain the distance between the tray 40 and the binocular camera 11 as long as it obtains the parallax, the focal length f and the setting distance B.

[0135] See also Figure 2 , Fig.21 , Fig. 22 , Fig.23 and Fig.24 In some embodiments, 08: controlling the movement of the forklift 100 according to the distance of the pallet 40 includes:

[0136] 081: Get the length of fork 42;

[0137] 082: According to the length of the fork 42 and the distance between the pallet 40 and the binocular camera 11, the distance between the end of the fork 42 close to the pallet 40 and the pallet 40 is obtained;

[0138] 083: Control the forklift 100 to move, and detect the distance between the end of the fork 42 close to the pallet 40 and the pallet 40 in real time until the distance between the end of the fork 42 close to the pallet 40 and the pallet 40 becomes zero;

[0139] 084: Obtain the preset safe penetration distance; and

[0140] 085: Control the forklift 100 to keep moving in the original direction until the distance the forklift 100 moves in the original direction is greater than or equal to the preset safe penetration distance.

[0141] The above-mentioned forklift control method can be applied to the control device 10 of the forklift 100, and the processing module 12 is used to: obtain the length of the fork 42; obtain the distance between the end of the fork 42 close to the pallet 40 and the pallet 40 according to the length of the fork 42 and the distance between the pallet 40 and the binocular camera 11; control the forklift 100 to move, and detect the distance between the end of the fork 42 close to the pallet 40 and the pallet 40 in real time until the distance between the end of the fork 42 close to the pallet 40 and the pallet 40 becomes zero; obtain a preset safe penetration distance; and control the forklift 100 to keep moving in the original direction until the distance that the forklift 100 keeps moving in the original direction is greater than or equal to the preset safe penetration distance.

[0142] Specifically, the length of the fork 42 is a parameter of the forklift 100 itself. After obtaining the length of the fork 42, the processing module 12 can obtain the distance Dc between the end of the fork 42 close to the pallet 40 and the pallet 40 according to the length of the fork 42 and the distance between the pallet 40 and the binocular camera 11. The processing module 12 controls the forklift 100 to move so that the forklift 100 moves from Fig.21 The status becomes Fig. 22and detects the distance Dc between the end of the fork 42 close to the pallet 40 and the pallet 40 in real time until the distance Dc between the end of the fork 42 close to the pallet 40 and the pallet 40 becomes zero, that is, the relative position of the forklift 100 and the pallet 40 becomes Fig. 22 The processing module 12 will continue to control the forklift 100 to keep moving in the original direction until the distance the forklift 100 keeps moving in the original direction is greater than or equal to the preset safe penetration distance, that is, the absolute value of Dc is greater than or equal to the preset safe penetration distance (that is, the forklift 100 reaches Fig.23 At this time, the forks 42 of the forklift 100 are inserted deeper and can safely lift the entire pallet 40.

[0143] In some embodiments, the calibration method of the binocular camera 11 can be Zhang Zhengyou's chessboard calibration method.

[0144] In summary, the present application provides a forklift control method, a forklift 100 control device, a vehicle computer, a forklift, and a computer-readable storage medium, which processes the image captured by the binocular camera 11 of the forklift 100 through the acquired target detection model to determine whether there is a column 43 in the image. In the case where there is a column 43 in the image, it is determined whether there is a pallet 40 in the image based on the number and distribution of the columns 43. In the case where there is a pallet 40 in the image, the distance between the pallet 40 and the binocular camera 11 is determined according to the binocular ranging algorithm, so as to control the movement of the forklift 100 according to the distance between the pallet 40 and the binocular camera 11, so as to ensure that the forklift can accurately lift the pallet 40. The present application replaces the depth camera with a lower-cost binocular camera 11, and also achieves the purpose of the forklift 100 automatically and accurately lifting the pallet 40, greatly reducing the manufacturing cost of the forklift 100.

[0145] In certain embodiments, see Fig.25 The present application also provides a vehicle computer 30, which includes a memory and a processor. The memory is configured to store a computer program. When the processor executes the computer program, it implements the control method in any of the above embodiments.

[0146] For example, when the processor of the vehicle computer 30 executes the computer program stored in the memory, the following control method is implemented:

[0147] 03: Use the binocular camera 11 to capture images;

[0148] 04: Input the image into a preset target detection model for processing to determine whether there is a pillar 43 in the image;

[0149] 05: When there are pillars 43 in the image, determine whether there is a tray 40 in the image according to the number and distribution of the pillars 43;

[0150] 07: When the tray 40 exists in the image, obtain the distance between the tray 40 and the binocular camera 11; and

[0151] 08: Control the movement of the forklift 100 according to the distance between the pallet 40 and the binocular camera 11.

[0152] For another example, when the processor of the vehicle computer 30 executes the computer program stored in the memory, the following control method is implemented:

[0153] 041: Input the image into the backbone network, and perform downsampling processing through multiple downsampling modules to obtain multiple downsampling feature maps, where the multiple downsampling feature maps include a first downsampling feature map, a second downsampling feature map, and a third downsampling feature map;

[0154] 043: Input the first down-sampled feature map, the second down-sampled feature map, and the third down-sampled feature map into a feature fusion network to obtain a first fused feature map, a second fused feature map, and a third fused feature map; and

[0155] 045: Input the first fused feature map, the second fused feature map and the third fused feature map into the result transformation network to determine whether there is a pillar 43 in the image.

[0156] For another example, when the processor of the vehicle computer 30 executes the computer program stored in the memory, it can also implement the control methods in 0431, 0432, 0433, 04331, 043311, 043312, 043313, 043314, 043315, 04333, 0434, 0435, 0451, 0453, 0455, 051, 053, 055, 0551, 0553, 06, 061, 063, 071, 073, 081, 082, 083, 084 and 085.

[0157] In certain embodiments, see Fig.25 The present application also provides a forklift 100, including the control device 10 of the forklift 100 in any of the above embodiments or the vehicle computer 30 in any of the above embodiments.

[0158] See also Fig.26 In some embodiments, the present application further provides a computer-readable storage medium 200 on which a computer program 202 is stored, and when the program is executed by a processor, the control method in any of the above embodiments is implemented.

[0159] For example, when the computer program 202 is executed by the processor 20, the following control method is implemented:

[0160] 03: Use the binocular camera 11 to capture images;

[0161] 04: Input the image into a preset target detection model for processing to determine whether there is a pillar 43 in the image;

[0162] 05: When there are pillars 43 in the image, determine whether there is a tray 40 in the image according to the number and distribution of the pillars 43;

[0163] 07: When the tray 40 exists in the image, obtain the distance between the tray 40 and the binocular camera 11; and

[0164] 08: Control the movement of the forklift 100 according to the distance between the pallet 40 and the binocular camera 11.

[0165] For another example, when the computer program 202 is executed by the processor 20, the following control method is implemented:

[0166] 041: Input the image into the backbone network, and perform downsampling processing through multiple downsampling modules to obtain multiple downsampling feature maps, where the multiple downsampling feature maps include a first downsampling feature map, a second downsampling feature map, and a third downsampling feature map;

[0167] 043: Input the first down-sampled feature map, the second down-sampled feature map, and the third down-sampled feature map into a feature fusion network to obtain a first fused feature map, a second fused feature map, and a third fused feature map; and

[0168] 045: Input the first fused feature map, the second fused feature map and the third fused feature map into the result transformation network to determine whether there is a pillar 43 in the image.

[0169] In the computer-readable storage medium 200 in the present application, the image captured by the binocular camera 11 of the forklift 100 is processed by the acquired target detection model to determine whether there is a column 43 in the image. In the case where there is a column 43 in the image, it is determined whether there is a pallet 40 in the image based on the number and distribution of the columns 43. In the case where there is a pallet 40 in the image, the distance between the pallet 40 and the binocular camera 11 is determined according to the binocular ranging algorithm, so that the movement of the forklift 100 is controlled according to the distance between the pallet 40 and the binocular camera 11 to ensure that the forklift can accurately lift the pallet 40. The present application replaces the depth camera with a lower-cost binocular camera 11, and also achieves the purpose of the forklift 100 automatically and accurately lifting the pallet 40, greatly reducing the manufacturing cost of the forklift 100.

[0170] In the description of this specification, the descriptions with reference to the terms "certain embodiments", "in an example", "exemplarily", etc., mean that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0171] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0172] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A forklift control method, characterized in that: The forklift (100) is provided with a binocular camera (11), and the control method comprises: Using the binocular camera (11) to capture images; Inputting the image into a preset target detection model for processing to determine whether there is a pillar (43) in the image; In the case where the pillars (43) exist in the image, determining whether a tray (40) exists in the image according to the number and distribution of the pillars (43); When the tray (40) exists in the image, obtaining the distance between the tray (40) and the binocular camera (11); and The movement of the forklift (100) is controlled according to the distance between the pallet (40) and the binocular camera (11).

2. The control method according to claim 1, characterized in that: The preset target detection model includes a backbone network, a feature fusion network and a result conversion network, wherein the backbone network includes a plurality of downsampling modules, and the image is input into the preset target detection model for processing to determine whether there is a pillar (43) in the image, including: Input the image into the backbone network, and perform downsampling processing by the multiple downsampling modules to obtain multiple downsampling feature maps, wherein the multiple downsampling feature maps include a first downsampling feature map, a second downsampling feature map, and a third downsampling feature map; Inputting the first down-sampled feature map, the second down-sampled feature map, and the third down-sampled feature map into the feature fusion network to obtain a first fused feature map, a second fused feature map, and a third fused feature map; and The first fused feature map, the second fused feature map and the third fused feature map are input into the result transformation network to determine whether the pillar exists in the image (43).

3. The control method according to claim 2, characterized in that: The feature fusion network includes a preset improvement module, and the first down-sampled feature map, the second down-sampled feature map, and the third down-sampled feature map are input into the feature fusion network to obtain a first fused feature map, a second fused feature map, and a third fused feature map, including: After upsampling the third down-sampled feature map, the up-sampled feature map is fused and added with the second down-sampled feature map to obtain a first transition map; After upsampling the first transition map, the first transition map is fused and added with the first downsampled feature map to obtain a second transition map; Inputting the second transition map, the first transition map and the third down-sampled feature map into the preset improvement module to obtain the first fusion feature map, the third transition map and the fourth transition map; After downsampling the first fused feature map, the first fused feature map is fused and added with the third transition map to obtain the second fused feature map; and After downsampling the second fused feature map, it is fused and added with the fourth transition map to obtain the third fused feature map.

4. The control method according to claim 3, characterized in that: The step of inputting the second transition map, the first transition map, and the third down-sampled feature map into the preset improvement module to obtain the first fused feature map, the third transition map, and the fourth transition map comprises: Using the preset improvement module to perform feature extraction processing on the first transition map, the second transition map and the third down-sampled feature map to obtain an eighth transition map corresponding to the first transition map, the second transition map and the third down-sampled feature map respectively; and The eighth transition map corresponding to the third down-sampling feature map is fused with the third down-sampling feature map to obtain the fourth transition map, the eighth transition map corresponding to the first transition map is fused with the first transition map to obtain the third transition map, and the eighth transition map corresponding to the second transition map is fused with the second transition map to obtain the first fused feature map.

5. The control method according to claim 4, characterized in that: The preset improvement module includes a plurality of first-class convolutional layers, a plurality of second-class convolutional layers, a spatial information collection layer, a normalization layer, an activation layer, and an assignment weight layer; the preset improvement module is used to perform feature extraction processing on the third down-sampled feature map, the second transition map, and the first transition map to obtain an eighth transition map corresponding to the third down-sampled feature map, the second transition map, and the first transition map, respectively, including: Inputting the third down-sampled feature map, the first transition map and the second transition map as input maps into the preset improvement module respectively; After the input image is segmented, it is processed by the first type of convolutional layer and the second type of convolutional layer to obtain a fifth transition image and a sixth transition image; The fifth transition map and the sixth transition map are fused and added, and then processed by the second type of convolutional layer to obtain a seventh transition map; After fusing and adding the fifth transition map and the seventh transition map, the fifth transition map and the seventh transition map are sequentially processed by the spatial information collection layer, the two first-type convolutional layers, the normalization layer, the activation layer, the two first-type convolutional layers, and the assignment weight layer to obtain a first assignment weight map and a second assignment weight map; and The fifth transition map is weightedly fused with the first assignment weight map, and the seventh transition map is weightedly fused with the second assignment weight map, and then processed by the first type of convolutional layer to obtain an eighth transition map.

6. The control method according to claim 4, characterized in that: The step of inputting the first fused feature map, the second fused feature map and the third fused feature map into the result conversion network to determine whether the pillar (43) exists in the image comprises: Obtaining pixel values ​​of each pixel point in the first fused feature map, the second fused feature map, and the third fused feature map; Based on the pixel values ​​of each pixel in the first fused feature map, the second fused feature map and the third fused feature map, a probability value of each pixel in the first fused feature map, the second fused feature map and the third fused feature map being a column (43) is calculated and used as a weight value; and Determine whether the pillar exists in the image based on the weight value of each pixel point in the first fusion feature map, the second fusion feature map and the third fusion feature map (43).

7. The control method according to claim 1, characterized in that: The columns (43) include a central column (431) and edge columns (432), and determining whether a tray (40) exists in the image according to the number and distribution of the columns includes: In the case where the pillars (43) exist in the image, determining whether the number of the pillars (43) is a preset number of pillars; When the number of the columns (43) is the preset number of columns, determining the distribution of the central columns (431) and the edge columns (432); and Whether the tray (40) exists in the image is determined based on the distribution of the central column (431) and the edge columns (432).

8. The control method according to claim 7, characterized in that: The distance between the central column and the edge column is the distance between the geometric center of the central column and the geometric center of the edge column, and the determining whether the tray (40) exists in the image according to the distribution of the central column (431) and the edge column (432) includes: In the case where the number of the preset pillars is three, the central pillar (431) is located between two edge pillars (432), and the distance between the central pillar (431) and any one of the edge pillars (432) is equal to the distance between the central pillar (431) and another one of the edge pillars (432), determining that the tray (40) exists in the image; and When the number of the preset pillars is not three, or the central pillar (431) is not located between two edge pillars (432), or the distance between the central pillar (431) and any one of the edge pillars (432) is not equal to the distance between the central pillar (431) and another one of the edge pillars (432), it is determined that the tray (40) does not exist in the image.

9. The control method according to claim 1, characterized in that: The control method further comprises: When a pallet (40) exists in the image, the position and posture of the forklift (100) are adjusted according to the relative position of the binocular camera (11) and the pallet (40).

10. The control method according to claim 9, characterized in that: The column (43) includes a central column (431), and the position and posture of the forklift (100) are adjusted according to the relative position of the binocular camera (11) and the pallet (40), including: Detecting the angle between a line connecting the center point of the line connecting the binocular camera (11) and the geometric center (4311) of the central column (431) and a preset axis; and When the included angle is not equal to the preset angle value, the position and posture of the forklift (100) are adjusted until the included angle is equal to the preset angle value.

11. The control method according to claim 10, characterized in that: The preset angle value is zero.

12. The control method according to claim 1, characterized in that: The column (43) includes a central column (431), the binocular distance measurement algorithm includes a binocular distance measurement formula, and the distance between the tray (40) and the binocular camera (11) is obtained, including: Processing the image according to a preset parallax algorithm to obtain the parallax between the binocular camera (11) and the geometric center (4311) of the central column (431); and The focal length and setting distance of the binocular camera (11) are obtained, and the focal length, the setting distance and the parallax are substituted into the binocular distance measurement formula to obtain the distance between the tray (40) and the binocular camera (11).

13. The control method according to claim 12, characterized in that: The binocular ranging formula is: Wherein, Z is the distance between the tray (40) and the binocular camera (11), f is the focal length, B is the setting distance, and "XL-XR" is the parallax.

14. The control method according to claim 1, characterized in that: The controlling the movement of the forklift (100) according to the distance of the pallet (40) comprises: Get the length of the fork (42); According to the length of the fork (42) and the distance between the pallet (40) and the binocular camera (11), the distance between the end of the fork (42) close to the pallet (40) and the pallet (40) is obtained; Controlling the forklift (100) to move, and detecting in real time the distance between the end of the fork (42) close to the pallet (40) and the pallet (40), until the distance between the end of the fork (42) close to the pallet (40) and the pallet (40) becomes zero; Obtain the preset safe penetration distance; and The forklift (100) is controlled to keep moving in the original direction until the distance the forklift (100) keeps moving in the original direction is greater than or equal to the preset safe penetration distance.

15. The control method according to claim 1, characterized in that: Before using the binocular camera (11) to capture an image, the method further comprises: Get the preset camera calibration algorithm; The binocular camera (11) is calibrated according to the preset camera calibration algorithm.

16. A control device for a forklift, characterized in that: The control device comprises a binocular camera (11) and a processing module, wherein the binocular camera (11) is used to capture images; The processing module is used to: input the image into a preset target detection model for processing to determine whether there is a pillar (43) in the image; In the case where the pillars (43) exist in the image, determining whether a tray (40) exists in the image according to the number and distribution of the pillars (43); When the tray (40) exists in the image, obtaining the distance between the tray (40) and the binocular camera (11); and The movement of the forklift (100) is controlled according to the distance between the pallet (40) and the binocular camera (11).

17. A vehicle computer, characterized in that: The vehicle computer (30) comprises a memory and a processor, the memory is configured to store a computer program, and the processor implements the control method according to any one of claims 1 to 15 when executing the computer program.

18. A forklift, characterized in that: It comprises the control device (10) of claim 16 and the vehicle computer (30) of claim 17.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor (20), the control method described in any one of claims 1 to 15 is implemented.

Images