Pallet access method, device, computer equipment and storage medium

Abstract

The present application relates to a pallet access method, device, computer equipment, storage medium and computer program product. The method comprises: collecting fisheye images based on a fisheye lens; cropping the fisheye images according to a morphological algorithm to obtain a cropped fisheye image; detecting whether there is a pallet in the cropped fisheye image; when there is a pallet, determining the position information of the pallet based on the pixel position of the pallet in the cropped fisheye image; accessing the pallet based on the position information. The method can acquire information by collecting fisheye images through a fisheye lens instead of a laser sensor, thereby ensuring the detection range and improving the perception range of the forklift; cropping the fisheye image according to a morphological algorithm to obtain a cropped fisheye image, detecting whether there is a pallet in the cropped fisheye image, reducing the total amount of information in the fisheye image, and facilitating direct analysis of the corresponding fisheye image with less algorithm complexity.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a pallet access method, device, computer equipment, storage medium and computer program product. Background Art

[0002] With the transformation and upgrading of traditional manufacturing industries, object storage industries such as smart factories and unmanned warehouses have gradually emerged, and transportation equipment such as automated forklifts have been widely used in many automated production lines.

[0003] At present, the mainstream pallet identification solution for logistics storage equipment is to use laser sensors to obtain point cloud data, and obtain the position of the pallet by analyzing the point cloud data, so as to realize the storage or insertion of pallets by transportation equipment.

[0004] However, when the laser sensor does not meet the application conditions, pallet perception and positioning cannot be performed based on the laser sensor. Summary of the invention

[0005] Based on this, it is necessary to provide a pallet access method, device, computer equipment, computer-readable storage medium and computer program product that can replace the sensor with a fisheye lens to address the above technical problems.

[0006] In a first aspect, the present application provides a pallet access method. The method comprises:

[0007] Collect fisheye images based on fisheye lens;

[0008] Cropping the fisheye image according to a morphological algorithm to obtain a cropped fisheye image;

[0009] Detecting whether there is a pallet in the cropped fisheye image;

[0010] When the pallet exists, determining the position information of the pallet based on the pixel position of the pallet in the cropped fisheye image;

[0011] The pallet is accessed based on the location information.

[0012] In one embodiment, the step of cropping the fisheye image according to a morphological algorithm to obtain a cropped fisheye image includes:

[0013] Selecting the morphological algorithm based on a preset shape corresponding to the target detection model;

[0014] Cropping the fisheye image according to a selected morphological algorithm to obtain a cropped fisheye image;

[0015] The detecting whether there is a pallet in the cropped fisheye image includes:

[0016] Based on the object detection model, it is detected whether there is a pallet in the cropped fisheye image.

[0017] In one embodiment, the target detection model is generated based on the yoloR network.

[0018] In one embodiment, the collecting of fisheye images based on a fisheye lens includes:

[0019] During the operation of the forklift robot, real-time fisheye images are collected based on the fisheye lens;

[0020] Before cropping the fisheye image according to the morphological algorithm to obtain the cropped fisheye image, the method further includes:

[0021] When the forklift robot receives at least one of a pallet storage instruction and a pallet insertion instruction, a pallet identification instruction is generated.

[0022] In one embodiment, determining the position information of the pallet based on the pixel position of the pallet in the cropped fisheye image includes:

[0023] Calculating an estimated distance between the pallet and the forklift truck based on a pixel position of the pallet in the cropped fisheye image;

[0024] The coordinate position information of the pallet is calculated based on the estimated distance.

[0025] In one embodiment, the method further comprises:

[0026] Collecting model training images and model test images based on fisheye lenses in different environments, wherein the model training images and the model test images respectively include marked modeling pallets;

[0027] Training a pre-trained target detection model based on the model training image to obtain a target detection model;

[0028] Calculating the recognition accuracy of the target detection model based on the model test image and the corresponding modeling stack;

[0029] When the recognition accuracy does not meet the preset condition, the target detection model continues to be trained until the model converges.

[0030] In a second aspect, the present application also provides a pallet storage and access device. The device comprises:

[0031] An image acquisition module, used for acquiring fisheye images based on a fisheye lens;

[0032] An image cropping module, used for cropping the fisheye image according to a morphological algorithm to obtain a cropped fisheye image;

[0033] A pallet recognition module, used to detect whether there is a pallet in the cropped fisheye image;

[0034] a pallet position determination module, configured to determine position information of the pallet based on a pixel position of the pallet in the cropped fisheye image when the pallet exists;

[0035] A pallet access module is used to access the pallet based on the position information.

[0036] In one embodiment, the device further comprises:

[0037] A model data acquisition module, used for acquiring model training images and model test images based on fisheye lenses in different environments, wherein the model training images and the model test images respectively include marked modeling pallets;

[0038] A model training module, used to train a pre-trained target detection model based on the model training image to obtain a target detection model;

[0039] A model testing module, used to calculate the recognition accuracy of the target detection model based on the model test image and the corresponding modeling stack;

[0040] The model convergence module is used to continue training the target detection model until the model converges when the recognition accuracy does not meet the preset conditions.

[0041] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of stack access in any of the above embodiments are implemented.

[0042] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of stack access in any of the above embodiments are implemented.

[0043] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of stack access in any of the above embodiments are implemented.

[0044] The above-mentioned pallet access method, device, computer equipment, storage medium and computer program product, based on the way of collecting fisheye images by fisheye lens, replaces laser sensor to obtain information, ensures the detection range, and improves the perception range of forklift; crops the fisheye image according to the morphological algorithm to obtain the cropped fisheye image, detects whether there is a pallet in the cropped fisheye image, reduces the total amount of information in the fisheye image, and facilitates direct analysis of the corresponding fisheye image with less algorithm complexity; then determines the position information of the pallet based on the pixel position of the pallet in the cropped fisheye image; accesses the pallet based on the position information, so as to perform operations such as accessing the pallet by using the fisheye lens instead of the laser sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 An application environment diagram of a stack access method in an embodiment;

[0046] Figure 2 A schematic diagram of a flow chart of a stack access method in one embodiment;

[0047] Figure 3 A schematic diagram of a process for training a target detection model in one embodiment;

[0048] Figure 4 A schematic diagram of a process flow of a pallet access device in another embodiment;

[0049] Figure 5 is a structural block diagram of a pallet access device in one embodiment;

[0050] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] The stack access method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 collects fisheye images based on a fisheye lens; crops the fisheye image according to a morphological algorithm to obtain a cropped fisheye image; detects whether there is a pallet in the cropped fisheye image; when the pallet exists, determines the position information of the pallet based on the pixel position of the pallet in the cropped fisheye image; and accesses the pallet based on the position information.

[0053] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0054] The intelligent vehicle-mounted device may be a forklift or other transport equipment carrying a corresponding control system, and the server may be a remote control device for the pallet access method.

[0055] In one embodiment, Figure 2 As shown, a stack access method is provided, which is applied to Figure 1 The terminal 102 in the example is used as an example to illustrate, and the following steps are included:

[0056] Step 202: Capture a fisheye image based on a fisheye lens.

[0057] A fisheye lens is a lens with a focal length within the preset fisheye lens focal length set. A fisheye lens is a wide-angle lens with a relatively small focal length, and its focal length is negatively correlated with the viewing angle. When the focal length of a fisheye lens is less than a certain focal length, the fisheye lens has a larger viewing angle than other lenses, which can cover the viewing angle of the laser sensor, and then replace the laser sensor to obtain the corresponding information for pallet perception and positioning. Moreover, in the field of transportation, laser sensors are expensive, ranging from thousands of yuan to tens of thousands of yuan, making the cost of transportation equipment such as forklifts too high; while the price of fisheye lenses is relatively low, which reduces the product cost of transportation equipment and has a corresponding price advantage.

[0058] Fisheye images are images taken at a certain position by a fisheye lens. Fisheye lenses cause distortion of fisheye images, and the degree of distortion of images taken by fisheye lenses of the same specification has a certain regularity, which is determined by training based on machine learning algorithms and is used to obtain corresponding information for pallet identification and positioning. The degree of distortion of fisheye images is indicated by the fisheye coefficient, which is negatively correlated with the focal length of the fisheye lens.

[0059] In one embodiment, after the terminal collects the fisheye image through the fisheye lens, it does not correct the fisheye image, but directly performs video image analysis on the fisheye image to reduce the complexity of the algorithm, increase the detection range, and improve its perception range.

[0060] In one embodiment, the forklift robot is directly used as a terminal, and the forklift robot performs corresponding business operations on the pallet. The fisheye image is collected based on the fisheye lens, including: during the operation of the forklift robot, the real-time fisheye image is collected based on the fisheye lens.

[0061] During the operation of the forklift robot, it can obtain various information through real-time fisheye images. It can determine the path of the forklift robot to the corresponding pallet and the corresponding obstacles based on the real-time fisheye images. It can generate corresponding commands based on the real-time fisheye images. It can also record and query information according to the real-time fisheye images.

[0062] Correspondingly, before cropping the fisheye image according to the morphological algorithm to obtain the cropped fisheye image, the method also includes: when the forklift robot receives at least one of a pallet storage instruction and a pallet insertion instruction, generating a pallet recognition instruction.

[0063] A pallet storage instruction is an instruction to place a pallet at a certain position; a pallet insertion instruction is an instruction to insert and remove a pallet. The generation process of the pallet storage instruction and the pallet insertion instruction can be the same. They can both be generated when the operator of the forklift robot views certain information in the fisheye image. They can be generated by a program of the terminal based on a preset timing system, or they can be generated by an application of the terminal based on a certain feature in the fisheye image.

[0064] When receiving the first pallet storage instruction, the forklift robot generates a first pallet insertion instruction and a pallet recognition instruction based on the first pallet storage instruction, recognizes and inserts the pallet in the fisheye image, and then places the inserted pallet at a certain location.

[0065] When receiving the second pallet insertion instruction, the forklift robot generates a corresponding pallet recognition instruction, and after identifying and inserting the pallet in the fisheye image, carries the corresponding pallet until the second pallet storage instruction is sent to the forklift robot, so as to place the pallet to a certain location based on the second pallet storage instruction.

[0066] When the forklift robot receives a pallet placement instruction or a pallet insertion instruction, it generates a pallet recognition instruction. The pallet recognition instruction is based on real-time fisheye image recognition, which can be used to determine whether there are pallets around the forklift robot, which can be used to determine the location of pallets in the surrounding environment, and which can also be used to determine the pallet to be recognized from multiple pallets.

[0067] Step 204 , cropping the fisheye image according to a morphological algorithm to obtain a cropped fisheye image.

[0068] The morphological algorithm may be a method for image cropping based on a binary morphological algorithm, which performs operations such as corrosion and expansion on the collected fisheye image to obtain a corresponding cropped fisheye image. The cropped fisheye image is a fisheye image generated by cropping the collected fisheye image according to the morphological algorithm, and has the pallet information corresponding to the collected fisheye image.

[0069] After the terminal crops the fisheye image according to the morphological algorithm, it can crop an image of a specific shape from the fisheye image, which removes information in the fisheye image that is not related to pallet recognition, so as to improve the accuracy and efficiency of pallet recognition. For example: when the edge of the fisheye image involves the forklift robot itself, the terminal will remove the edge of the image and delete information that is not related to the pallet; when the pallets that the forklift robot can access are all below a certain horizontal plane, the fisheye image above the horizontal plane can be cropped.

[0070] In one embodiment, the fisheye image is cropped according to a morphological algorithm to obtain a cropped fisheye image, including: selecting a morphological algorithm based on a preset shape corresponding to a target detection model, and cropping the fisheye image according to the selected morphological algorithm to obtain a cropped fisheye image.

[0071] During the training process of the target detection model, a morphological algorithm is selected based on a certain shape. Before detecting whether there is a pallet in the cropped fisheye image, a morphological algorithm is selected based on a preset shape corresponding to the target detection model.

[0072] For example, when training an object detection model, a morphological algorithm is used to crop the largest inscribed square from a fisheye image. When detecting a pallet, a morphological algorithm is used to crop the largest inscribed square from the fisheye image to obtain a cropped fisheye image.

[0073] Correspondingly, detecting whether there is a pallet in the cropped fisheye image includes detecting whether there is a pallet in the cropped fisheye image based on the target detection model.

[0074] In one embodiment, the target detection model is generated based on the yoloR network.

[0075] The yoloR network refers to one of the (You Only Look Once, YOLO) algorithms, whose Chinese names can be a glance target detection algorithm, regression detection algorithm, and only look once algorithm. The target detection model trained by this algorithm has the advantages of both high efficiency and high accuracy. Among them, the w6 model of the yoloR network has higher training efficiency and more accurate recognition effect.

[0076] The yoloR network integrates implicit knowledge and explicit knowledge through a network, so that the model contains a unified representation. This unified representation allows sub-representation to be applied to other tasks. Among them, explicit knowledge is the knowledge observed by the model, while implicit knowledge is the knowledge that the model cannot observe and is irrelevant to the observation.

[0077] Step 206 , detecting whether there is a pallet in the cropped fisheye image.

[0078] In one embodiment, the process of detecting whether there is a stack in the cropped fisheye image by the terminal is based on a trained target detection model. The target detection model can be trained based on the EfficientDet network, based on the Faster R-CNN network, based on the DETR network, based on the centerNet network, or based on the yolo network. The yoloR network in the yolo network is the most efficient and accurate deep learning network among these target detection network models.

[0079] In one embodiment, detecting whether there is a pallet in the cropped fisheye image is a process of performing target detection on the pallet based on a target detection model. After feature extraction is performed on the cropped fisheye image, based on the extracted features, it is determined whether there is a pallet in each anchor point frame of the target detection model. When a corresponding pallet exists in any anchor point frame, it is determined that there is a pallet in the cropped fisheye image.

[0080] In one embodiment, different pallets have various labels. When the label corresponding to the pallet is detected, one or more of the pallet information such as the pallet type, pallet specification and pallet style can be identified to achieve accurate identification of the pallet. Compared with a variety of commodities, the specifications and styles of pallets are relatively few, and the shapes of most pallets are relatively similar. The target detection model can be trained more conveniently through the deep information network training platform, and the target detection model has the characteristics of high precision and high efficiency.

[0081] Step 208 : When a pallet exists, determine the position information of the pallet based on the pixel position of the pallet in the cropped fisheye image.

[0082] The cropped fisheye image is composed of multiple pixels, and there is a corresponding positional relationship between each pixel. Although there is an essential difference between the pixel position in the cropped fisheye image and the actual position of the pallet, when certain data based on the cropped fisheye image and the forklift robot are input into the neural network model as a training set, the output distance can be used to instruct the robot in a certain coordinate system to execute and complete the corresponding action.

[0083] For example, when training a target detection model, a fisheye image with a distortion coefficient of k1 is searched to train the target detection model. Then, a cropped fisheye image with a cropping coefficient of k1 can be directly positioned based on the target detection model without correcting the image distortion.

[0084] In one embodiment, determining the position information of the pallet based on the pixel position of the pallet in the cropped fisheye image includes: calculating the estimated distance between the pallet and the forklift truck based on the pixel position of the pallet in the cropped fisheye image; and calculating the coordinate position information of the pallet based on the estimated distance.

[0085] Step 210, access the stack based on the location information.

[0086] The location information is used to control transportation equipment such as forklift robots to operate the pallet. The terminal determines the distance and direction of the pallet relative to the transportation equipment based on the location information, and locates the pallet based on the distance and direction to store or remove the pallet.

[0087] In the above-mentioned pallet access method, the fisheye image is collected based on the fisheye lens, which replaces the laser sensor to obtain information, thereby increasing the detection range and improving the forklift perception range; the fisheye image is cropped according to the morphological algorithm to obtain the cropped fisheye image, and it is detected whether there is a pallet in the cropped fisheye image, thereby reducing the total amount of information in the fisheye image, and facilitating the direct analysis of the corresponding fisheye image with less algorithm complexity; then, based on the pixel position of the pallet in the cropped fisheye image, the position information of the pallet is determined; the pallet is accessed based on the position information, and the fisheye lens is used instead of the laser sensor to access the pallet, so that the pallet can be sensed, positioned and operated when the laser sensor does not meet the application conditions, thereby realizing the forklift robot positioning the pallet and accurately identifying multiple pallets.

[0088] In a specific embodiment, a fisheye camera is installed in front of a forklift robot, and the fisheye image is preprocessed by a morphological method to obtain a maximum inscribed square image; the trained target detection model is loaded, and the yoloR algorithm is used for forward reasoning to detect whether there is a pallet in the image. If there is a pallet, a pallet coordinate message is sent to the forklift robot. The forklift robot locates the position of the pallet in front through the returned coordinate message and performs related business operations such as pallet insertion and removal.

[0089] Specifically, after the forklift robot equipped with the fisheye camera is deployed in the warehouse, the forklift robot obtains the front image in real time through the fisheye camera during operation; when the forklift robot needs to insert a pallet, it sends a message to the fisheye camera for real-time identification of the pallet. If the fisheye camera detects the pallet, it sends a message to the forklift robot to locate the pallet. The forklift robot inserts and removes the pallet based on the positioning information.

[0090] In one embodiment, Figure 3 As shown, when detecting whether there is a pallet in the cropped fisheye image based on the target detection model, the method further includes:

[0091] Step 302 : collecting model training images and model test images based on fisheye lenses in different environments, wherein the model training images and the model test images respectively include marked modeling pallets.

[0092] In one embodiment, the terminal collects model training images and model test images based on fisheye lenses in different environments, including collecting original images for modeling, cropping original images for modeling, and converting data sets. In the process of collecting original images, the fisheye camera is installed on a traditional forklift or other transportation equipment, and the traditional forklift is pushed to simulate the insertion and removal of pallets to collect data in real time and obtain the original images for modeling.

[0093] Cropping the original image for modeling is the process of labeling the data set. It uses the labelImg labeling tool to use a square frame to crop the original image into the largest inscribed square or other preset shape image, and then annotate the pallets one by one, and the annotated label is pallet, and the corresponding txt format label file is saved. All data is divided into model training images and model test image sets for pallet detection at a ratio of 5:1.

[0094] Dataset format conversion is to convert the annotated txt file into a corresponding format file, such as xml, json format file, according to the required format of the training data, to obtain model training images and model test images.

[0095] Step 304: Train the pre-trained target detection model based on the model training image to obtain the target detection model.

[0096] When training the pre-training target detection model based on the model training image, first set the training parameters of the pre-training target detection model through the terminal: select Pytorch as the deep learning network training platform, use the yoloR network as the backbone network, select the w6 model, set the width to 1280, the height to 1280, the batch_size to 64, the classes to 1, the anchors to 29, 46, 43, 48, 40, 85, 88, 61, 78, 183, 160, 150, 132, 260, 270, 270, 223, 613, 456, 588, 800, 450, 956, 764, use the pre-training target detection model such as the open source pre-training database model, and set the model initialization parameters.

[0097] In the process of training the model, the learning rate lr is set to 0.00261 as the initial learning rate, the threshold of the learning strategy change of the network iteration (burn_in) is set to 1000, and the maximum number of training iterations (max_batches) is set to 300200. Start training until the average value of the observed prediction value (ap value) tends to be stable and stop training. Save the training to obtain the target detection model that has not been tested.

[0098] Step 306 , calculating the recognition accuracy of the target detection model based on the model test image and the corresponding modeling stack.

[0099] Step 308: When the recognition accuracy does not reach the preset condition, continue to train the target detection model until the model converges.

[0100] Specifically, based on the training parameters corresponding to the preset detection model, the stack training images are trained until the average value change trend of the modeling stack is predicted to be less than a threshold value, thereby obtaining a first target detection model corresponding to the preset detection model; based on the stack test image, the prediction accuracy of the first detection model is tested; based on the result of the accuracy test, the average precision corresponding to the first detection model is calculated; when the average precision is less than the preset threshold value, at least one of the following processing is performed: adjusting the training parameters or increasing the stack training images, and training is performed based on the processed data until the average precision is greater than the threshold value, thereby obtaining a second target detection model; setting the region of interest and the corresponding shape in the second target detection model to set the morphological algorithm;

[0101] For example, in the process of testing the model, the test parameters are set first, the threshold threshod is set to 0.75, the intersection-and-union ratio threshold iou_threshod = 0.5, the training object obj_ist = ['pallet], and the average precision mAP indicator is used to judge the test data for the model. If mAP is less than 0.99, the training parameter settings are modified or the data set is increased for retraining until the average precision mAP requirement is greater than 0.99.

[0102] Therefore, a unique fisheye image-based pallet recognition algorithm is provided, which enables the forklift robot to locate the pallet and realize the AI ​​recognition function of the forklift, while also being able to perform accurate recognition of multiple pallets.

[0103] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0104] Based on the same inventive concept, the embodiment of the present application also provides a pallet access device for implementing the pallet access method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more pallet access device embodiments provided below can refer to the limitations on the pallet access method above, and will not be repeated here.

[0105] In one embodiment, Figure 4As shown, a pallet access device is provided, including: an image acquisition module 402, an image cropping module 404, a pallet identification module 406, a pallet position determination module 408 and a pallet access module 410, wherein:

[0106] An image acquisition module 402 is used to acquire fisheye images based on a fisheye lens;

[0107] An image cropping module 404 is used to crop the fisheye image according to a morphological algorithm to obtain a cropped fisheye image;

[0108] A pallet recognition module 406 is used to detect whether there is a pallet in the cropped fisheye image;

[0109] A pallet position determination module 408, configured to determine position information of the pallet based on a pixel position of the pallet in the cropped fisheye image when the pallet exists;

[0110] The pallet access module 410 is used to access the pallet based on the location information.

[0111] In one embodiment, the image cropping module 404 includes:

[0112] A morphological algorithm selection unit, configured to select the morphological algorithm based on a preset shape corresponding to the target detection model;

[0113] An image cropping unit, used for cropping the fisheye image according to a selected morphological algorithm to obtain a cropped fisheye image;

[0114] Correspondingly, the pallet identification module 406 includes:

[0115] A pallet recognition unit is used to detect whether there is a pallet in the cropped fisheye image based on the target detection model.

[0116] In one embodiment, the target detection model is generated based on the yoloR network.

[0117] In one embodiment, the image acquisition module 402 includes:

[0118] A real-time data acquisition unit, used to acquire real-time fisheye images based on the fisheye lens during the operation of the forklift robot;

[0119] Correspondingly, the device further comprises:

[0120] The pallet identification instruction generating module is used to generate a pallet identification instruction when the forklift robot receives at least one of a pallet storage instruction and a pallet insertion instruction.

[0121] In one embodiment, the pallet position determination module 408 includes:

[0122] A distance estimation unit, configured to calculate an estimated distance between the pallet and the forklift truck based on a pixel position of the pallet in the cropped fisheye image;

[0123] A coordinate position generating unit is used to calculate the coordinate position information of the pallet based on the estimated distance.

[0124] In one embodiment, if Figure 5 As shown, the device also includes:

[0125] A model data acquisition module 502, used for acquiring model training images and model test images based on fisheye lenses in different environments, wherein the model training images and the model test images respectively include marked modeling pallets;

[0126] A model training module 504 is used to train a pre-trained target detection model based on the model training image to obtain a target detection model;

[0127] A model testing module 506, configured to calculate the recognition accuracy of the target detection model based on the model testing image and the corresponding modeling stack;

[0128] The model convergence module 508 is used to continue training the target detection model until the model converges when the recognition accuracy does not meet the preset conditions.

[0129] Each module in the above-mentioned stack access device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0130] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a stack access method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0131] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0132] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0133] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0134] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0135] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0136] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0137] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0138] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A pallet access method, characterized in that: The method comprises: During the operation of the forklift robot, a real-time fisheye image is collected based on a fisheye lens; the real-time fisheye image is uncorrected; Based on the real-time fisheye image, determine the path and obstacles of the forklift robot to the corresponding pallet; Cropping the fisheye image according to a morphological algorithm to obtain a cropped fisheye image; After the forklift robot receives at least one of a pallet storage instruction and a pallet insertion instruction and generates a pallet recognition instruction, it detects whether there is a pallet in the cropped fisheye image; the pallet is detected and located based on a target detection model, and the real-time fisheye image and the fisheye image used for training the target detection model have the same distortion coefficient; When the pallet exists, determining the position information of the pallet based on the pixel position of the pallet in the cropped fisheye image; The pallet is accessed based on the location information.

2. The method according to claim 1, characterized in that The step of cropping the fisheye image according to the morphological algorithm to obtain a cropped fisheye image includes: Selecting the morphological algorithm based on a preset shape corresponding to the target detection model; Cropping the fisheye image according to a selected morphological algorithm to obtain a cropped fisheye image; The detecting whether there is a pallet in the cropped fisheye image comprises: Based on the object detection model, it is detected whether there is a pallet in the cropped fisheye image.

3. The method according to claim 2, characterized in that The target detection model is generated based on the yoloR network.

4. The method according to claim 1, characterized in that The method further comprises: generating corresponding commands according to the real-time fisheye image; and / or, Information is recorded and queried according to the real-time fisheye image.

5. The method according to claim 1, characterized in that The determining the position information of the pallet based on the pixel position of the pallet in the cropped fisheye image includes: Calculating an estimated distance between the pallet and the forklift truck based on a pixel position of the pallet in the cropped fisheye image; The coordinate position information of the pallet is calculated based on the estimated distance.

6. The method according to claim 2, characterized in that The method further comprises: Collecting model training images and model test images based on fisheye lenses in different environments, wherein the model training images and the model test images respectively include marked modeling pallets; Training a pre-trained target detection model based on the model training image to obtain a target detection model; Calculating the recognition accuracy of the target detection model based on the model test image and the corresponding modeling stack; When the recognition accuracy does not meet the preset condition, the target detection model continues to be trained until the model converges.

7. A pallet storage and retrieval device, characterized in that: The device comprises: An image acquisition module is used to acquire real-time fisheye images based on a fisheye lens during the operation of the forklift robot; based on the real-time fisheye images, determine the path and obstacles from the forklift robot to the corresponding pallet; the real-time fisheye images are uncorrected; An image cropping module, used for cropping the fisheye image according to a morphological algorithm to obtain a cropped fisheye image; a pallet recognition module, configured to detect whether a pallet exists in the cropped fisheye image after the forklift robot receives at least one of a pallet storage instruction and a pallet insertion instruction and generates a pallet recognition instruction; the pallet is detected and located based on a target detection model, and the real-time fisheye image and the fisheye image used for training the target detection model have the same distortion coefficient; a pallet position determination module, configured to determine position information of the pallet based on a pixel position of the pallet in the cropped fisheye image when the pallet exists; A pallet access module is used to access the pallet based on the position information.

8. The device according to claim 7, characterized in that The device also includes: A model data acquisition module, used for acquiring model training images and model test images based on fisheye lenses in different environments, wherein the model training images and the model test images respectively include marked modeling pallets; A model training module, used to train a pre-trained target detection model based on the model training image to obtain a target detection model; A model testing module, used to calculate the recognition accuracy of the target detection model based on the model test image and the corresponding modeling stack; The model convergence module is used to continue training the target detection model until the model converges when the recognition accuracy does not meet the preset conditions.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images