Roadway type stacker operation roadway obstacle detection method

Abstract

The invention relates to a roadway type stacker operation roadway obstacle detection method, which realizes dead-angle-free image acquisition and accurate positioning through a binocular camera and a laser distance measuring sensor, and performs obstacle identification and positioning in combination with a YOLOv3 deep learning model. The roadway type stacking machine comprises a binocular camera, a laser distance measuring sensor, a supporting piece, a stacking machine track and a light reflecting plate, and high-precision detection of obstacles is achieved through the steps of binocular camera calibration, image collection, obstacle image marking, model training and testing and the like. According to the method, the operation safety of the roadway type stacking machine is improved, the collision risk is reduced, the distance between the obstacle and the stacking machine can be judged in real time, and a powerful guarantee is provided for intelligent operation of the roadway type stacking machine.

Description

Technical Field

[0001] The invention belongs to the technical field of logistics warehousing equipment, and in particular relates to a method for detecting obstacles in an aisle of an aisle stacker. Background Art

[0002] In the production of cigarette industry, the importance of the aisle stacker (hereinafter referred to as "stacker") as a widely used logistics host equipment is self-evident. The stacker shuttles at high speed in the aisle, responsible for the tasks of storing and picking up goods. Its operating efficiency and safety are directly related to the smoothness and stability of the entire production line.

[0003] During the operation of the stacker, the laser distance sensor is the key detection mechanism to ensure the normal operation of the stacker and plays a vital role. The traditional laser distance sensor measures the relative position between the stacker and the shelf by emitting laser and receiving the laser signal reflected by the reflector, thereby achieving accurate positioning and obstacle avoidance. Figure 3 and Figure 4 As shown, 16 is a stacker, 15 and 17 are the first shelf 15 and the second shelf 17 arranged on both sides of the stacker, the lane is the space between the two rows of shelves, 9 is a laser ranging sensor, 10 is a reflector, and 14 is a stacker track.

[0004] However, this traditional detection method has exposed some problems in practical applications. The traditional laser distance sensor is installed at the bottom of the stacker. During the operation of the equipment, the laser distance sensor can only detect obstacles on the stacker track that are blocked in front of the laser distance sensor and can reflect the laser. Due to the limitations of the installation position and detection range of the traditional laser distance sensor, it is easy to miss detection. During the production operation, when the pallets or goods on the shelves are not stacked in place, that is, there are Figure 3 When there are obstacles shown, such as the first obstacle 11 or the second obstacle 18, these obstacles may extend into the operating space of the stacker and collide with the stacker, resulting in equipment damage and production interruption.

[0005] At the same time, Figure 4 As shown in the figure, the laser distance sensor needs a specific reflector to reflect the laser signal when working, which means that its detection range is mainly concentrated in a specific area directly above the stacker track. Once the obstacle is outside this area or cannot provide sufficient reflective surface, the laser distance sensor may not work properly, resulting in detection failure.

[0006] Although the photoelectric sensor can detect whether there is an obstacle in front by detecting the obstruction of infrared light, its detection principle is: when there is an obstacle in front of the stacker and it blocks the infrared light, the switch signal of the photoelectric sensor changes, and the system program sends out an alarm and a stop signal to stop the stacker from running and prevent the stacker from colliding. However, this method can only detect obstacles that are large in size or that block the sensor signal. For some smaller obstacles, or obstacles that cannot block the sensor signal, such as Figure 3 As shown in the third obstacle 13 and the fourth obstacle 18 in the figure, the photoelectric beam sensor also has the risk of missing detection.

[0007] In addition, photoelectric sensors are installed on the stacker to detect obstacles, such as Figure 5 As shown, 1 is the first detection phototube, 5 is the second detection phototube, 2 is the third detection phototube, 6 is the fourth detection phototube, 3 is the fifth detection phototube, 7 is the sixth detection phototube, 4 is the seventh detection phototube, 8 is the eighth detection phototube, 1 and 5, 2 and 6, 3 and 7, 4 and 8 respectively constitute four groups of detection phototubes. When there are obstacles blocking the range of these four groups of phototubes, the photoelectric sensor will generate a switch signal change and transmit this change to the stacker control system. The control system determines the existence of an obstacle based on this signal change, and issues an obstacle alarm and shuts down through the system program to prevent the stacker from colliding with the obstacle.

[0008] Although the use of the above-mentioned photoelectric sensors for obstacle detection has improved the safety of the stacker to a certain extent, it still has the following disadvantages:

[0009] (1) Limited detection range: Photoelectric sensors can usually only detect specific areas and cannot cover the entire area where the stacker may collide. This means that in some areas, obstacles may not be detected in time, increasing the risk of collision.

[0010] (2) Inconvenient installation: The four detection photoelectric tubes in the photoelectric sensor need to be installed on the wall or related devices. This installation method not only increases the complexity of installation, but may also be limited by the structure and position of the wall or device, resulting in limited detection range or poor detection effect.

[0011] (3) It is difficult to ensure the accuracy of the shooting: Since the stacker is located in a large high-bay warehouse, the distance between each group of photoelectric tubes is relatively far. At the same time, the stacker is working at a high speed. These factors increase the difficulty of ensuring the accuracy of each group of photoelectric tubes. Once the shooting is not accurate, it may lead to false alarms or missed alarms, increasing the failure rate of the equipment.

[0012] (4) Difficulty in fault diagnosis: When the stacker is located in a high-bay warehouse and far away from the control host, once a fault occurs, the control system can usually only display an obstacle alarm signal, but cannot provide detailed information about the fault site. This makes fault diagnosis and maintenance difficult, increasing downtime and maintenance costs.

[0013] (5) Detection blind spots: Since the upper part of the stacker has a small coverage area, it is difficult to install photoelectric detection devices in these areas. Therefore, these areas become detection blind spots, increasing the risk of collision. Especially in complex environments such as high-bay warehouses, detection blind spots may be more serious.

[0014] In summary, traditional laser distance sensors and photoelectric sensors have certain limitations in detecting obstacles during the operation of stackers, and cannot fully meet the safety and reliability requirements of the cigarette industry. Therefore, it is necessary to develop a new method for detecting obstacles in the aisle of aisle stackers to improve the safety and stability of the stackers in the aisles.

[0015] The present invention is proposed to solve the above-mentioned problems. Summary of the invention

[0016] The present invention provides a method for detecting obstacles in an aisle of an aisle stacker, wherein the aisle stacker comprises: a first binocular camera 19, a second binocular camera 20, a laser ranging sensor 9, a support 21, a stacker track 14 and a reflector 10, wherein the laser ranging sensor 9 is located at the bottom of the aisle stacker, and a stacker track 14 is arranged below the aisle stacker, the reflector 10 is located directly in front of the laser ranging sensor 9 and is connected to one end of the stacker track 14, the support 21 is located directly above the aisle stacker, the first binocular camera 19 and the second binocular camera 20 are respectively arranged in front and at the rear of the aisle stacker, wherein the second binocular camera 20 is located on the support 21, and the number of the first binocular camera 19 and the second binocular camera 20 is one or more, and the method comprises the following steps:

[0017] (1) Calibrate and optimize parameters of the first binocular camera 19 and the second binocular camera 20;

[0018] (2) Using a first binocular camera 19 and a second binocular camera 20, which are installed at different heights in front and behind the aisle stacker, image acquisition of all height shelves and the ground on both sides of the aisle stacker is completed during the aisle stacker's back-and-forth shuttle movement, thereby achieving image acquisition and detection of the aisle in which the aisle stacker is operating without blind spots;

[0019] (3) Adjust the aisle stacker to different positions, place obstacles such as pallets and goods in the aisle where the aisle stacker is running and collect images, move pallets at different positions on the shelf outward to a position where they can collide with the aisle stacker and collect images, adjust the contrast and brightness of the collected images, and expand the image set size to 4000 images;

[0020] (4) Marking the obstacle images on the left and right sides of the binocular camera obtained in step (3), and dividing the images into a training set, a validation set, and a test set after marking; wherein the images in the training set and the validation set are used for YOLOv3 deep learning model training, and the images in the test set are used for model effect testing after model training;

[0021] (5) Use TensorFlow 2.0 in the Python environment to build a deep learning platform, set training parameters, and use the image set completed in step (4) for training. During the training process, observe the changes in the loss value (avg loss). When the loss value (avg loss) remains stable within multiple rounds and no longer decreases, select the weight file at that moment as the optimal weight, import the weight file into the YOLOv3 model, and test the model;

[0022] (6) Set the obstacle detection accuracy target value to 95%. If the detection accuracy is lower than the target value during the model testing process, repeat steps (3), (4) and (5), and increase the number of images of the corresponding misdetected targets in step (3) according to the test results, and retrain the model until the requirements are met. The calculation formula for the average accuracy (AP) of obstacle detection is as follows:

[0023]

[0024] Where: TP is the number of samples with debris and detected in the test sample image, FP is the number of samples with debris and not detected in the test sample image, and FN is the number of samples with no debris and detected in the test sample image;

[0025] (7) After the training of step (6) is completed, the YOLOv3, binocular camera image acquisition, posture calculation, PLC communication and other program deployments are completed on the Jetson Xavier NX platform, the relevant hardware is installed and debugged, the aisle obstacle detection system of the aisle stacker is completed, and the detection system is used for detection;

[0026] (8) During the detection process, the image obtained by the left camera in the binocular camera is used to input the obstacle target detection model, and the obstacle target detection and recognition results are output. At the same time, the image depth information is calculated through the binocular camera imaging parallax principle analysis, and the obstacle image is combined with the obstacle detection frame corner points calculated in the obstacle target recognition results to calculate the position and posture of the obstacle image, and the distance between the obstacle and the aisle stacker is determined in real time.

[0027] Preferably, the specific steps of calibrating and optimizing the parameters of the first binocular camera 19 and the second binocular camera 20 in step (1) are as follows:

[0028] (a) Prepare a checkerboard of Zhang Zhengyou calibration method, measure the size of the checkerboard, fix the posture of the first binocular camera 19 and the second binocular camera 20, shoot the checkerboard 10 times from different angles, and obtain a calibration image;

[0029] (b) detecting the feature points in the image and the corner points of the calibration plate to obtain the pixel coordinate values ​​of the corner points of the calibration plate, and calculating the physical coordinate values ​​of the corner points of the calibration plate according to the known chessboard size and the origin of the world coordinate system;

[0030] (c) solving the internal parameter matrix and the external parameter matrix of the first binocular camera 19 and the second binocular camera 20 according to the pixel coordinate values ​​and the spatial coordinate values ​​of the feature corner points;

[0031] (d) Solve the distortion parameters of the camera;

[0032] (e) Use the LM (Levenberg-Marquardt) algorithm to optimize the distortion parameters in step (d).

[0033] Preferably, the method for calculating the distance between the obstacle and the aisle stacker in step (8) is as follows:

[0034] The pixel coordinate system UOV of the image is in pixels, O is the origin, the left direction is U+, and the downward direction is V-;

[0035] Coordinate system X of the imaging plane I O I Y I It is based on the imaging plane of the camera, which is the plane of the camera's CCD or CMOS sensor. The intersection of the imaging plane and the center line perpendicular to the camera lens is the origin of the coordinate system, and its unit is mm. Let the origin of the camera's imaging plane coordinate system be O I , the coordinates of this point in the pixel coordinate system UOV are (u0, v0), then any point (u, v) in the pixel coordinate system UOV and the imaging plane coordinate system X I O I Y IThe relationship between the corresponding points (x1, y1) in is as follows:

[0036]

[0037] Among them, dx and dy represent the length and width of each pixel in the image on the imaging plane, that is, in an image, the area is dx*dy (in mm). 2 ) is a pixel in the image; rewrite equations (4) and (5) into matrix form:

[0038]

[0039] The camera's coordinate system O C -X C Y C Z C The coordinate origin of the image coincides with the projection center of the image. C The axis coincides with the vertical line of the center of the imaging lens, X C Axis and Y C The axes are respectively related to the X I Axis and Y I Axis parallel, from the origin of the camera's coordinate system O C To the origin O of the imaging plane coordinate system I The distance is the focal length f of the camera; through the perspective projection model, it can be obtained that the object point P (X C , Y C , Z C ) and the coordinate system X of the imaging plane I O I Y I The conversion between the imaging points P(X1, Y1) under is as follows:

[0040]

[0041] The transformation relationship of the coordinate system is expressed in matrix form:

[0042]

[0043] The camera's internal parameter matrix is ​​as follows:

[0044]

[0045] Multiply both sides of equation (10) by Z C have to:

[0046]

[0047] World Coordinate System O W -XW Y W Z W , is a coordinate system used to describe the position of objects in the real environment. Its setting is related to the specific application scenario. The camera coordinate system and the world coordinate system are transformed by the rotation matrix R and the translation vector T. At a certain point in space (X W , Y W , Z W ) and a point in the camera coordinate system (X C , Y C , Z C ) are as follows:

[0048]

[0049] Where M2 is the camera's external parameter matrix, M2 is only related to the relative position of the camera and the world coordinate system, R is a 3×3 orthogonal unit matrix, and T is a three-dimensional translation vector;

[0050] The conversion relationship between the pixel coordinate system of the image and the world coordinate system is as follows:

[0051]

[0052] Through formula (13), the pose parameters of the obstacle photographed in the current detection image can be estimated by three-dimensional reconstruction technology, and the distance between the obstacle and the aisle stacker can be calculated.

[0053] Preferably, after the labeling in step (4) is completed, the images are divided into a training set, a validation set and a test set in a ratio of 8:1:1.

[0054] Preferably, after step (8) determines the distance between the obstacle and the aisle stacker in real time, the following steps are also included: based on the distance between the obstacle and the aisle stacker and in combination with the collision distance setting conditions, a collision alarm signal and a field detection image are output in real time to the aisle stacker control system to control the operation of the aisle stacker.

[0055] Beneficial effects of the present invention:

[0056] Improved obstacle detection range for aisle stackers:

[0057] (1) The present invention uses a binocular camera as the core component of obstacle detection. The binocular camera can capture a wider field of view, thereby ensuring that the aisle stacker can promptly detect and avoid obstacles during operation. Compared with the traditional through-beam photoelectric sensor, the obstacle detection range of the aisle stacker is improved, which not only improves the operating efficiency of the aisle stacker, but also reduces the risk of collision caused by failure to detect obstacles, thereby significantly improving the operating safety of the aisle stacker.

[0058] (2) The present invention uses a binocular camera to capture images in the tunnel in real time, and after image processing algorithm analysis, it can accurately identify obstacles and calculate the distance information between the obstacles and the tunnel stacker. This information can provide a reliable reference for the control system of the tunnel stacker, helping the tunnel stacker to make a timely response when encountering obstacles and avoid collision accidents. Compared with the through-beam photoelectric sensor, the binocular camera has higher stability and reliability, can operate stably in various complex environments, and ensure the safe operation of the tunnel stacker.

[0059] (3) The present invention uses a binocular camera as a detection mechanism, which solves the problem of complex installation and adjustment of the through-beam photoelectric sensor and unstable detection system. The through-beam photoelectric sensor needs precise alignment during installation and adjustment, and is easily affected by environmental factors (such as dust, light, etc.), resulting in unstable detection system. The binocular camera does not require a complicated installation and adjustment process, and only needs to be installed in a suitable position on the aisle stacker. The binocular camera uses advanced image processing technology and can automatically adapt to different lighting conditions and environmental changes to ensure the stability and reliability of the detection system.

[0060] (4) It is worth mentioning that the present invention can output collision alarm signals and on-site detection images to the aisle stacker control system in real time according to the distance between the obstacle and the aisle stacker in combination with the collision distance setting conditions, and control the operation of the aisle stacker, which enables maintenance personnel to understand the conditions in the aisle in real time through remote monitoring. Once a problem or abnormal situation is found, maintenance personnel can quickly take measures to deal with it, thereby improving the convenience and efficiency of maintenance. In addition, by collecting and analyzing image data in the aisle, the operating status of the aisle stacker can also be monitored and evaluated, providing strong support for subsequent maintenance and optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a schematic flow chart of a method for detecting obstacles in an aisle of an aisle stacker according to the present invention.

[0062] Figure 2 It is a schematic structural diagram of the lane type stacker of the present invention.

[0063] Figure 3 It is a structural schematic diagram of an existing aisle stacker and corresponding shelves and aisles in the warehouse.

[0064] Figure 4 It is a schematic diagram of the structure of an existing aisle stacker including a laser ranging sensor.

[0065] Figure 5 The present invention is a schematic diagram of the structure of an existing aisle stacker including a photoelectric counter-radiation sensor.

[0066] Figure 6 Schematic diagram of the pixel coordinate system UOV of the image.

[0067] Figure 7 is the coordinate system X of the imaging plane I O I Y I Schematic diagram.

[0068] Figure 8 is the camera coordinate system O C -X C Y C Z C Schematic diagram.

[0069] The meanings of the reference numerals are as follows:

[0070] 1-first detection phototube, 2-third detection phototube, 3-fifth detection phototube, 4-seventh detection phototube, 5-second detection phototube, 6-fourth detection phototube, 7-sixth detection phototube, 8-eighth detection phototube, 9-laser ranging sensor, 10-light-emitting board, 11-first obstacle, 12-stacked goods, 13-third obstacle, 14-stacker track, 15-first shelf, 16-stacker, 17-second shelf, 18-second obstacle, 19-first binocular camera, 20-second binocular camera, 21-support. DETAILED DESCRIPTION

[0071] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is specifically described below in conjunction with embodiments, but the embodiments of the present invention are not limited thereto.

[0072] The operating methods without specific conditions in the examples are usually carried out according to conventional conditions and conditions described in technical manuals, or according to conditions recommended by the manufacturers. The general equipment and materials used are all commercially available unless otherwise specified.

[0073] Example 1

[0074] In this embodiment, the height of the tobacco finished product warehouse is: 12.8m; the target size of the industrial camera selected is: 2 / 3 inch, the target size is 8.8mm wide * 6.6mm high; the shelf spacing is 1900mm, the pallet width is 1400mm, the shortest working distance for obstacle detection is set to 3000mm, the stacker maintenance platform is 1200mm high from the ground, the height of the electric cabinet installed on the maintenance platform is 1900mm, and the height of the top of the electric cabinet from the ground is 3100mm.

[0075] The image acquisition and obstacle ranging range requirements are as follows:

[0076] (1) The camera must be able to clearly capture a rectangular area within the width of the shelf interval and the height of the tobacco finished product warehouse. That is, the camera needs to cover a field of view of: 1900 mm wide and 12800 mm high.

[0077] (2) Obstacles cannot block the pallet in the horizontal area, and the vertical area must be fully covered. The obstacle distance measurement range in the horizontal direction extends 700mm to the left and right from the center of the image. For safety reasons, it is set to 750mm. That is, the distance measurement range is: the area with a width of 1500mm and a height of 12800mm within the field of view covered by the camera.

[0078] There must be no obstacles within the width of the pallet, that is, within the horizontal width of 1400mm from the center of the stacker.

[0079] Calculate according to the camera field of view calculation formula

[0080]

[0081] The size of the camera's vertical field of view is defined as H, and the size of the camera's horizontal field of view is defined as V.

[0082] When the focal length of the camera is 8mm:

[0083]

[0084] When the focal length is 5mm, the camera is:

[0085]

[0086] When the focal length is 3mm, the camera is:

[0087]

[0088] According to the size of the stacker, to ensure that obstacles on the ground and on the top can be detected, both cameras need to be 3mm focal length cameras. Since the horizontal field of view of the camera is 8800mm, each camera needs to cover a field of view of 6400mm at a vertical height. The installation positions of the two cameras are as follows:

[0089] (1) On the electrical cabinet (low position) The camera bracket can be installed on the top of the electrical cabinet at a height of 3200 mm from the ground.

[0090] (2) The camera support can be installed on the stacker's lifting column (at a high position) at a position 9600mm above the ground.

[0091] The installation according to the above scheme can realize image recognition detection of stacker obstacles without blind spots.

[0092] A method for detecting obstacles in an aisle of an aisle stacker, wherein the aisle stacker, Figure 2 As shown, it includes: a first binocular camera 19, a second binocular camera 20, a laser ranging sensor 9, a support 21, a stacker track 14 and a reflector 10, wherein the laser ranging sensor 9 is located at the bottom of the aisle stacker, and the stacker track 14 is arranged below the aisle stacker, the reflector 10 is located directly in front of the laser ranging sensor 9 and is connected to one end of the stacker track 14, the support 21 is located directly above the aisle stacker, and the first binocular camera 19 and the second binocular camera 20 are respectively arranged in front and behind the aisle stacker, wherein the second binocular camera 20 is located on the support 21, and the number of the first binocular camera 19 and the second binocular camera 20 is 1.

[0093] The method comprises the following steps:

[0094] (1) Calibrate and optimize the parameters of the first binocular camera 19 and the second binocular camera 20. The specific steps are as follows:

[0095] (a) Prepare a checkerboard of Zhang Zhengyou calibration method, measure the size of the checkerboard, fix the postures of the first binocular camera 19 and the second binocular camera 20, shoot the checkerboard 10 times from different angles, and obtain a calibration image.

[0096] (b) The feature points in the image and the corner points of the calibration plate are detected to obtain the pixel coordinate values ​​of the corner points of the calibration plate. According to the known chessboard size and the origin of the world coordinate system, the physical coordinate values ​​of the corner points of the calibration plate are calculated.

[0097] (c) The intrinsic parameter matrix and the extrinsic parameter matrix of the first binocular camera 19 and the second binocular camera 20 are solved according to the pixel coordinate values ​​and the spatial coordinate values ​​of the feature corner points.

[0098] (d) Solve the distortion parameters of the camera.

[0099] (e) Use the LM (Levenberg-Marquardt) algorithm to optimize the distortion parameters in step (d).

[0100] (2) A first binocular camera 19 and a second binocular camera 20 are installed at different heights in front of and behind the aisle stacker to complete image acquisition of all height shelves and the ground in the tobacco finished product warehouse on both sides of the aisle stacker during the back-and-forth movement of the aisle stacker, thereby realizing image acquisition and detection of the aisle in which the aisle stacker is operating without blind spots.

[0101] (3) Adjust the aisle stacker to different positions, place obstacles such as pallets and goods in the aisle where the aisle stacker is operating and collect images, move pallets at different positions on the shelf outward to a position where they can collide with the aisle stacker and collect images, adjust the contrast and brightness of the collected images, and expand the image set size to 4000 images.

[0102] (4) Mark the obstacle images on the left and right sides of the binocular camera obtained in step (3), and divide the images into a training set, a validation set, and a test set in a ratio of 8:1:1. The images in the training set and the validation set are used for YOLOv3 deep learning model training, and the images in the test set are used for model effect testing after model training.

[0103] (5) Use TensorFlow 2.0 in the Python environment to build a deep learning platform, set training parameters, and use the image set completed in step (4) for training. During the training process, observe the changes in the loss value (avg loss). When the loss value (avg loss) remains stable within multiple rounds and no longer decreases, select the weight file at that moment as the optimal weight, import the weight file into the YOLOv3 model, and test the model.

[0104] (6) Set the obstacle detection accuracy target value to 95%. If the detection accuracy is lower than the target value during the model testing process, repeat steps (3), (4) and (5), and increase the number of images of the corresponding misdetected targets in step (3) according to the test results, and retrain the model until the requirements are met. The calculation formula for the average accuracy (AP) of obstacle detection is as follows:

[0105]

[0106] Where TP is the number of samples with and without debris in the test sample image, FP is the number of samples with and without debris in the test sample image, and FN is the number of samples with and without debris in the test sample image.

[0107] (7) After the training in step (6) is completed, the YOLOv3, binocular camera image acquisition, pose calculation, PLC communication and other program deployments are completed on the Jetson Xavier NX platform, and the relevant hardware is installed and debugged. The aisle obstacle detection system for the aisle stacker is completed, and the detection system is used for detection.

[0108] (8) During the detection process, the image obtained by the left camera in the binocular camera is used to input the obstacle target detection model, and the obstacle target detection and recognition results are output. At the same time, the image depth information is calculated through the binocular camera imaging parallax principle analysis, and the obstacle image is combined with the obstacle detection frame corner points calculated in the obstacle target recognition results to calculate the position and posture of the obstacle image, and the distance between the obstacle and the aisle stacker is determined in real time.

[0109] The distance calculation method between the obstacle and the aisle stacker is as follows:

[0110] like Figure 6 As shown, the pixel coordinate system UOV of the image is in pixels, O is the origin, the left direction is U+, and the downward direction is V-;

[0111] like Figure 7 As shown, the coordinate system X of the imaging plane I O I Y I It is based on the imaging plane of the camera, which is the plane of the camera's CCD or CMOS sensor. The intersection of the imaging plane and the center line perpendicular to the camera lens is the origin of the coordinate system, and its unit is mm. Let the origin of the camera's imaging plane coordinate system be O I , the coordinates of this point in the pixel coordinate system UOV are (u0, v0), then any point (u, v) in the pixel coordinate system UOV and the imaging plane coordinate system X I O I Y I The relationship between the corresponding points (x1, y1) in is as follows:

[0112]

[0113] Among them, dx and dy represent the length and width of each pixel in the image on the imaging plane, that is, in an image, the area is dx*dy (in mm). 2 ) is a pixel in the image; rewrite equations (4) and (5) into matrix form:

[0114]

[0115] like Figure 8As shown, the camera coordinate system O C -X C Y C Z C The coordinate origin of the image coincides with the projection center of the image. C The axis coincides with the vertical line of the center of the imaging lens, X C Axis and Y C The axes are respectively related to the X I Axis and Y I Axis parallel, from the origin of the camera's coordinate system O C To the origin O of the imaging plane coordinate system I The distance is the focal length f of the camera; through the perspective projection model, it can be obtained that the object point P (X C , Y C , Z C ) and the coordinate system X of the imaging plane I O I Y I The conversion between the imaging points P(X1, Y1) under is as follows:

[0116]

[0117] The transformation relationship of the coordinate system is expressed in matrix form:

[0118]

[0119] The camera's internal parameter matrix is ​​as follows:

[0120]

[0121] Multiply both sides of equation (10) by Z C have to:

[0122]

[0123] World Coordinate System O W -X W Y W Z W , is a coordinate system used to describe the position of objects in the real environment. Its setting is related to the specific application scenario. The camera coordinate system and the world coordinate system are transformed by the rotation matrix R and the translation vector T. At a certain point in space (X W , Y W , Z W ) and a point in the camera coordinate system (X C , Y C , Z C ) are as follows:

[0124]

[0125] Where M2 is the camera's external parameter matrix, M2 is only related to the relative position of the camera and the world coordinate system, R is a 3×3 orthogonal unit matrix, and T is a three-dimensional translation vector;

[0126] The conversion relationship between the pixel coordinate system of the image and the world coordinate system is as follows:

[0127]

[0128] Through formula (13), the pose parameters of the obstacle photographed in the current detection image can be estimated by three-dimensional reconstruction technology, and the distance between the obstacle and the aisle stacker can be calculated.

[0129] (9) Based on the distance between the obstacle and the aisle stacker and the collision distance setting conditions, a collision alarm signal and a field detection image are output in real time to the aisle stacker control system to control the operation of the aisle stacker.

[0130] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for detecting obstacles in an aisle of an aisle stacker, the aisle stacker comprising: A first binocular camera (19), a second binocular camera (20), a laser distance measuring sensor (9), a support (21), a stacker track (14) and a reflector (10), wherein the laser distance measuring sensor (9) is located at the bottom of the aisle stacker, a stacker track (14) is arranged below the aisle stacker, the reflector (10) is located directly in front of the laser distance measuring sensor (9) and is connected to one end of the stacker track (14), the support (21) is located directly above the aisle stacker, the first binocular camera (19) and the second binocular camera (20) are arranged in front and in the rear of the aisle stacker, respectively, wherein the second binocular camera (20) is located on the support (21), the number of the first binocular camera (19) and the second binocular camera (20) is one or more, and the method comprises the following steps: (1) calibrating and optimizing parameters of a first binocular camera (19) and a second binocular camera (20); (2) using a first binocular camera (19) and a second binocular camera (20) installed at different heights in front and behind the aisle stacker, completing image acquisition of all height shelves and the ground on both sides of the aisle stacker during the aisle stacker's back-and-forth shuttle movement, thereby achieving image acquisition and detection of the aisle in which the aisle stacker is operating without blind spots; (3) Adjust the aisle stacker to different positions, place obstacles such as pallets and goods in the aisle where the aisle stacker is running and collect images, move pallets at different positions on the shelf outward to a position where they can collide with the aisle stacker and collect images, adjust the contrast and brightness of the collected images, and expand the image set size to 4000 images; (4) Marking the obstacle images on the left and right sides of the binocular camera obtained in step (3), and dividing the images into a training set, a validation set, and a test set after marking; wherein the images in the training set and the validation set are used for YOLOv3 deep learning model training, and the images in the test set are used for model effect testing after model training; (5) Use TensorFlow 2.0 in the Python environment to build a deep learning platform, set training parameters, and use the image set completed in step (4) for training. During the training process, observe the changes in the loss value (avg loss). When the loss value (avgloss) remains stable within multiple rounds and no longer decreases, select the weight file at that moment as the optimal weight, import the weight file into the YOLOv3 model, and test the model; (6) Set the obstacle detection accuracy target value to 95%. If the detection accuracy is lower than the target value during the model testing process, repeat steps (3), (4) and (5), and increase the number of images of the corresponding misdetected targets in step (3) according to the test results, and retrain the model until the requirements are met. The calculation formula for the average accuracy (AP) of obstacle detection is as follows: Where: TP is the number of samples with debris and detected in the test sample image, FP is the number of samples with debris and not detected in the test sample image, and FN is the number of samples with no debris and detected in the test sample image; (7) After the training of step (6) is completed, the YOLOv3, binocular camera image acquisition, posture calculation, PLC communication and other program deployments are completed on the Jetson Xavier NX platform, the relevant hardware is installed and debugged, the aisle obstacle detection system of the aisle stacker is completed, and the detection system is used for detection; (8) During the detection process, the image obtained by the left camera in the binocular camera is used to input the obstacle target detection model, and the obstacle target detection and recognition results are output. At the same time, the image depth information is calculated through the binocular camera imaging parallax principle analysis, and the obstacle image is combined with the obstacle detection frame corner points calculated in the obstacle target recognition results to calculate the position and posture of the obstacle image, and the distance between the obstacle and the aisle stacker is determined in real time.

2. The method according to claim 1, characterized in that The specific steps of calibrating and optimizing the parameters of the first binocular camera (19) and the second binocular camera (20) in step (1) are as follows: (a) preparing a checkerboard of Zhang Zhengyou calibration method, measuring the size of the checkerboard, fixing the postures of the first binocular camera (19) and the second binocular camera (20), photographing the checkerboard 10 times from different angles, and obtaining a calibration image; (b) detecting the feature points in the image and the corner points of the calibration plate to obtain the pixel coordinate values ​​of the corner points of the calibration plate, and calculating the physical coordinate values ​​of the corner points of the calibration plate according to the known chessboard size and the origin of the world coordinate system; (c) solving the internal parameter matrix and the external parameter matrix of the first binocular camera (19) and the second binocular camera (20) according to the pixel coordinate values ​​and the spatial coordinate values ​​of the feature corner points; (d) Solve the distortion parameters of the camera; (e) Use the LM (Levenberg-Marquardt) algorithm to optimize the distortion parameters in step (d).

3. The method according to claim 1, characterized in that The method for calculating the distance between the obstacle and the aisle stacker in step (8) is as follows: The pixel coordinate system UOV of the image is in pixels, O is the origin, the left direction is U+, and the downward direction is V-; Coordinate system X of the imaging plane I O I Y I It is based on the imaging plane of the camera, which is the plane of the camera's CCD or CMOS sensor. The intersection of the imaging plane and the center line perpendicular to the camera lens is the origin of the coordinate system, and its unit is mm. Let the origin of the camera's imaging plane coordinate system be O I , the coordinates of this point in the pixel coordinate system UOV are (u0, v0), then any point (u, v) in the pixel coordinate system UOV and the imaging plane coordinate system X I O I Y I The relationship between the corresponding points (x1, y1) in is as follows: Among them, dx and dy represent the length and width of each pixel in the image on the imaging plane, that is, in an image, the area is dx*dy (in mm). 2 ) is a pixel in the image; rewrite equations (4) and (5) into matrix form: The camera's coordinate system O C -X C Y C Z C The coordinate origin of the image coincides with the projection center of the image. C The axis coincides with the vertical line of the center of the imaging lens, X C Axis and Y C The axes are respectively related to the X I Axis and Y I Axis parallel, from the origin of the camera's coordinate system O C To the origin O of the imaging plane coordinate system I The distance is the focal length f of the camera; through the perspective projection model, it can be obtained that the object point P (X C , Y C , Z C ) and the coordinate system X of the imaging plane I O I Y I The conversion between the imaging points P(X1, Y1) under is as follows: The transformation relationship of the coordinate system is expressed in matrix form: The camera's internal parameter matrix is ​​as follows: Multiply both sides of equation (10) by Z C have to: World coordinate system Q W -X W Y W Z W , is a coordinate system used to describe the position of objects in the real environment. Its setting is related to the specific application scenario. The camera coordinate system and the world coordinate system are transformed by the rotation matrix R and the translation vector T. At a certain point in space (X W , Y W , Z W ) and a point in the camera coordinate system (X C , Y C , Z C ) are as follows: Where M2 is the camera's external parameter matrix, M2 is only related to the relative position of the camera and the world coordinate system, R is a 3×3 orthogonal unit matrix, and T is a three-dimensional translation vector; The conversion relationship between the pixel coordinate system of the image and the world coordinate system is as follows: Through formula (13), the pose parameters of the obstacle photographed in the current detection image can be estimated by three-dimensional reconstruction technology, and the distance between the obstacle and the aisle stacker can be calculated.

4. The method according to claim 1, characterized in that After the labeling in step (4) is completed, the images are divided into training set, validation set and test set in a ratio of 8:1:

1.

5. The method according to claim 1, characterized in that After step (8) determines the distance between the obstacle and the aisle stacker in real time, the following steps are also included: according to the distance between the obstacle and the aisle stacker and in combination with the collision distance setting conditions, a collision alarm signal and a field detection image are output in real time to the aisle stacker control system to control the operation of the aisle stacker.

Images