A dangerous chemical product stacking safety distance detection method adaptable to different illumination conditions

Abstract

The present application relates to the technical field of image data processing, and discloses a dangerous chemical product stacking safety distance detection method adaptable to different illumination conditions, which extracts feature points by using a machine learning algorithm, and does not involve using experience to specify what kind of point is a feature point in the feature point extraction process. The method can be used under low illumination or even infrared camera, and cannot successfully extract feature points due to not considering low illumination like traditional experience algorithms. The method overcomes the problem that irrelevant points are also extracted when machine learning is used to extract feature points, so that subsequent three-dimensional reconstruction work will not be caused by the existence of irrelevant points, and the reconstructed three-dimensional graphics is inconsistent with the real shape.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a method for detecting the safe distance of hazardous chemical stacks that can adapt to different lighting conditions. Background Technology

[0002] When storing hazardous chemicals in a warehouse, it is necessary to control the distance between them and other locations to avoid dangers caused by excessive close proximity. These safe distances generally include stack distance, wall distance, top distance, column distance, and aisle distance, also known as the "five distances".

[0003] During loading and unloading, warning lines are generally used to control the distance between stacks of hazardous chemicals. However, due to negligence during loading and unloading or falling during storage, some hazardous chemicals may sometimes cross the warning line. Therefore, warehouses need to be inspected frequently to eliminate this risk. However, manual inspection is not only labor-intensive and time-consuming, but also unreliable in many cases because people's subjective perception of distance is inaccurate. Furthermore, the warning line itself may sometimes be misaligned due to subjective or objective reasons.

[0004] For this reason, some methods have emerged that use machine vision (extracting feature points from two photos taken by a binocular camera, such as the corner points of an object, and then using the paired feature points on the two photos to measure the distance to obtain depth information, i.e., the distance between the feature points and the camera, and using the feature points and depth information to perform 3D reconstruction) to automatically determine the five distances of hazardous chemicals. Examples include CN109741306A - Image processing method for stacking hazardous chemicals in warehouses, CN113034490A - Method for monitoring the safe distance of stacking in chemical warehouses, CN114120113A - A binocular image matching method suitable for detecting the stacking distance of hazardous chemicals, and CN114219857A - A method for measuring the safe distance of stacking hazardous chemicals.

[0005] However, these methods share a common drawback: they are dependent on lighting conditions. They perform poorly in low light (even when visible to the human eye) and are completely ineffective under infrared cameras. This necessitates sufficient lighting during inspection, requiring adequate visible light sources in the warehouse, and the lights must remain on during the five-distance inspection. This increases the barrier to entry and warehouse energy consumption.

[0006] The inventors discovered in their research that existing five-distance detection methods for hazardous chemicals, regardless of the method used, all involve the extraction of feature points in the image. The extraction of feature points is carried out using feature detection methods designed based on a large amount of experience. However, these algorithms cannot take into account various factors such as lighting and camera type (infrared camera vs. visible light camera) during their design. Therefore, they identify very few feature points when the lighting is poor, which is insufficient to complete the subsequent three-dimensional reconstruction.

[0007] Besides traditional feature point extraction algorithms, some machine learning-based algorithms for feature point extraction have emerged, which are not based on empirical design. However, these algorithms have a major problem: many of the identified feature points are unnecessary. For example, they may also identify gaps in floor tiles or stains on walls, which limits their practical applications.

[0008] Compared to machine vision (which identifies the three-dimensional shape of objects), object recognition and contour recognition in two-dimensional graphics are simpler (because they do not involve depth information in the image) and are already widely used in software such as Photoshop. Summary of the Invention

[0009] This invention provides a method for detecting the safe distance of hazardous chemical stacks that can adapt to different lighting conditions.

[0010] The technical problem to be solved is that existing automatic detection methods for safe distances in hazardous chemical stacks have poor recognition performance under low light conditions.

[0011] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: a method for detecting the safe distance of hazardous chemical stacks that can adapt to different lighting conditions, comprising the following steps:

[0012] Step 1: Take photos of the interior of the hazardous chemical storage warehouse from at least two different angles;

[0013] Step 2: Extract image features from each photo to obtain feature points and descriptors for each feature point; the image feature extraction algorithm used is a self-supervised machine learning algorithm.

[0014] Step 3: Mark the outline of the hazardous chemical stacks in each photo. If a feature point in a photo falls outside the outline, remove that feature point.

[0015] Step 4: Match the feature points obtained in Step 2 to obtain feature point pairs;

[0016] Step 5: Use the feature point pairs from Step 4 to perform 3D reconstruction, and then monitor whether the five distances of the hazardous chemical stacks in the 3D model meet the requirements.

[0017] Furthermore, in step one, a calibrated binocular camera is used for shooting.

[0018] Furthermore, the binocular camera was calibrated using the Zhang Zhengyou calibration method.

[0019] Furthermore, in step two, the algorithm used is the SuperPoint feature extraction network.

[0020] Furthermore, in step four, the SuperGule feature matching network is used for feature point matching.

[0021] Furthermore, step three is as follows: Use an object detection algorithm to identify hazardous chemical stacks, use regular geometric shapes to frame each hazardous chemical stack, and then remove feature points located outside the frame.

[0022] Furthermore, in step three, the target detection algorithm used is YOLOv5.

[0023] Furthermore, in step three, a rectangular frame with each side tangent to the edge of the hazardous chemical stack is used for selection.

[0024] Furthermore, the floor of the hazardous chemical storage warehouse is a solid-color floor without grid lines or decorations.

[0025] Furthermore, in step five, the 3D model is a colorless point cloud model.

[0026] Compared with existing technologies, the present invention provides a method for detecting the safe distance of hazardous chemical stacks that can adapt to different lighting conditions, and has the following advantages:

[0027] In this invention, feature points are extracted using machine learning algorithms. The feature point extraction process does not rely on experience to define what constitutes a feature point. Unlike traditional empirical algorithms, which fail to extract feature points under low light conditions, this method is effective even in low-light environments and can be used with infrared cameras. Furthermore, by utilizing mature 2D image target and contour recognition methods to identify the contours of hazardous chemical stacks and then discarding feature points outside the contours, the invention overcomes the problem of machine learning extracting irrelevant points. This ensures that subsequent 3D reconstruction will not result in inconsistencies between the reconstructed 3D graphics and the actual shape due to the presence of irrelevant points. Attached Figure Description

[0028] Figure 1 This is a flowchart of a method for detecting the safe distance of hazardous chemical stacks that can adapt to different lighting conditions, according to the present invention. Detailed Implementation

[0029] like Figure 1 As shown, a method for detecting the safe distance of hazardous chemical stacks that can adapt to different lighting conditions includes the following steps:

[0030] Step 1: Take photos of the interior of the hazardous chemical storage warehouse from at least two different angles;

[0031] Note that the photos here can be a single set of photos or multiple sets of photos taken consecutively, i.e., a video. The reason for requiring photos from at least two angles is to obtain depth information from the image. To use the human eye as an example, at least two eyes are needed to perceive the distance of objects.

[0032] Step 2: Extract image features from each photo to obtain feature points and descriptors for each feature point; the image feature extraction algorithm used is a self-supervised machine learning algorithm.

[0033] Self-supervised training algorithms are a new type of object detection algorithm that uses auxiliary tasks to mine its own supervisory information from large-scale unsupervised data. The network is then trained using this constructed supervisory information, thereby learning valuable representations for tasks such as object classification.

[0034] The reason for choosing a self-supervised machine learning algorithm here is to ensure the feasibility of the solution. This field is quite niche, with limited training sets available, and existing sets do not consider recognition under low light or even infrared camera conditions. Furthermore, creating a new training set is very costly because it requires interdisciplinary expertise in both machine learning and chemical safety to label the data. Using a self-supervised training algorithm eliminates the need for labeled training sets; unlabeled sets can be obtained simply by taking a few sets of photos under different lighting conditions, making it very easy to implement. There are even pre-trained models available online that can be directly transferred and used.

[0035] Step 3: Mark the outline of the hazardous chemical stacks in each photo. If a feature point in a photo falls outside the outline, remove that feature point.

[0036] This step involves two tasks: first, identifying where the hazardous chemical stacks are in the image, and then drawing the outline of the identified hazardous chemical stacks. Both of these are very mature technologies, so I won't go into details here.

[0037] Step 4: Match the feature points obtained in Step 2 to obtain feature point pairs;

[0038] In layman's terms, if a feature point is the eight corner points of a cube, then each of these points needs to be identified as the same thing in different photos; this is called a feature point pair. With a feature point pair, the distance between the feature point and the camera can be calculated, which is equivalent to obtaining depth information. Using feature points and depth information, 3D graphics can be reconstructed.

[0039] Step 5: Use the feature point pairs from Step 4 to perform 3D reconstruction, and then monitor whether the five distances of the hazardous chemical stacks in the 3D model meet the requirements.

[0040] The positions of the warehouse walls, ceilings, pillars, and passageways remain constant. Therefore, we only need to determine the position and shape of the hazardous chemical stacks to create a 3D model of the entire warehouse containing the hazardous chemical stacks. Then, we can easily calculate the five distances in the 3D model.

[0041] This application also compares the performance of the method in this application with that of two traditional algorithms (SIFT and ORB), as detailed below:

[0042] Experimental hardware environment: Intel(R) Core(TM) i5-10400 CPU@2.9GHz; Experimental software environment: Windows 10 64-bit operating system, OpenCV 3.4.2.16; Deep learning framework: PyTorch 1.6.0. The evaluation of SuperPoint was performed using the pre-trained model provided at https: / / github.com / MagicLeapResearch / SuperPointPretrainedNetwork, and the evaluation of traditional algorithms was performed using the default OpenCV parameters.

[0043] The experiment involved four cubes placed in an experimental room to simulate a stack of hazardous chemicals. The cubes were photographed using a binocular camera under three different lighting conditions. The average brightness of the three sets of images was obtained using the Python image processing library PIL, and denoted as well as well-lit, poorly lit, and extremely poorly lit conditions, respectively. The average brightness of each set of images was 155.62, 85.88, and 31.125 pixel values ​​(grayscale values), respectively.

[0044] In step one, a calibrated binocular camera was used for imaging. The binocular camera was calibrated using the Zhang Zhengyou calibration method. The acquired image size was 640×480, and the calibration board used a 258mm×258mm checkerboard grid, with each checkerboard cell having a side length of 25.8mm. The calibration results are shown in Tables 1, 2, and 3. The units for focal length and principal point position are pixels, and the unit for translation matrix is ​​mm.

[0045] Table 1. Calibration results of the left camera's intrinsic parameters.

[0046]

[0047] Table 2. Calibration results of the right camera's intrinsic parameters.

[0048]

[0049] Table 3. Calibration results of binocular camera extrinsic parameters

[0050]

[0051] The calibration results show that the rotation matrix is ​​approximately the identity matrix, indicating that the binocular system has virtually no rotation, which facilitates the establishment of a binocular vision model. The translation matrix results show that the focal distance between the two cameras is 101.1429 mm, which is close to the actual measured value of 100 mm, indicating that the calibration results are relatively accurate and can be used for subsequent measurement calculations.

[0052] In step two, the algorithm used is the SuperPoint feature extraction network.

[0053] The SuperPoint feature extraction network is an existing algorithm, and its details can be found in the following literature, so it will not be repeated here.

[0054] Detone D, Malisiewicz T, Rabinovich A. SuperPoint: Self-Supervised Interest Point Detection and Description[C] / / 2018 IEEE / CVFConference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE, 2018.

[0055] The feature points extracted using the SuperPoint algorithm include not only the points on the edges and corners of the cube, but also many points in the floor seams and at the base of the walls. These points are invalid for 3D reconstruction and will cause interference. These points will be discarded in step three.

[0056] Correspondingly, the SIFT algorithm extracts feature points that are mostly on the squares, while the ORB algorithm extracts feature points that are some outside the squares.

[0057] Step three is as follows: Use an object detection algorithm to identify the hazardous chemical stacks, and use regular geometric shapes to frame each stack, then remove feature points outside the frames. The regular geometric shapes can be rectangles, circles, ellipses, and various regular polygons.

[0058] In step three, the target detection algorithm used is YOLOv5. This is also a very commonly used target detection algorithm, and will not be described in detail here.

[0059] The reason for using regular geometric shapes as the outline of the hazardous chemical stack, rather than drawing an amorphous outline around it, is that in practical use, we found that drawing an amorphous outline around the stack not only involves a large amount of computation but also moves some feature points that are actually on the stack outside the outline. In this embodiment, a rectangular frame with each side tangent to the edge of the hazardous chemical stack is used. The selection box has only four tangent points with the hazardous chemicals in the image, which effectively avoids moving feature points on the stack outside the outline while also preventing irrelevant points from being included. Even if one or two irrelevant points are included, they will become outliers, which will not be considered during 3D reconstruction and will be directly excluded.

[0060] If the feature points near the tangent point are relatively dense in actual use, the rectangle can be enlarged by a fixed number of pixels to serve as the selection box.

[0061] Of course, if the floor of the hazardous chemical storage warehouse is a solid color floor without grid lines or decorations, many irrelevant points can be eliminated from the source.

[0062] In step four, the SuperGule feature matching network is used for feature point matching.

[0063] The SuperGule feature matching network is also an existing algorithm, and its details can be found in the following literature, which will not be repeated here.

[0064] Sarlin PE, Detone D, Malisiewicz T, et al. SuperGlue: LearningFeature Matching With Graph Neural Networks[C] / / 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020.

[0065] Since irrelevant feature points were removed in step three, the feature points obtained in this step are on the edges and corners of the cube. The feature point pairs are shown in the following figure:

[0066] Table 4. Matching results of different algorithms after removing irrelevant feature points.

[0067]

[0068] In Table 4, only the algorithm in this application removes irrelevant feature points. The two traditional algorithms used for comparison do not remove irrelevant feature points. The matching accuracy in the table refers to the proportion of all feature point pairs in which the same feature point is correctly matched in the images taken by the left and right cameras.

[0069] As shown in the figure, even after removing many irrelevant points, the number of feature point pairs extracted by the algorithm in this application is greater than that of the two traditional algorithms, which is more beneficial to the subsequent 3D reconstruction work. If irrelevant points are not removed, the number of feature point pairs is even greater, ranging from 310, 279 and 391 in well-lit areas to extremely poor lighting, with matching accuracies of 98.71%, 99.20% and 98.46%, respectively.

[0070] In the subsequent step five, these numerous feature points can be used to successfully complete the 3D reconstruction. Since the color information of the image is not required, a colorless point cloud model can be built to quickly complete the 3D reconstruction.

[0071] To ensure accurate ranging, we also compared the three algorithms at feature points 550 mm away from the camera, using the feature points to measure the error between the ranging results and the actual distance. The results are as follows:

[0072] Table 5 Comparison of Distance Measurement Errors

[0073]

[0074] It is evident that the method described in this application also achieves higher ranging accuracy.

[0075] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for detecting the safe distance of hazardous chemical stacks that can adapt to different lighting conditions, characterized in that: Includes the following steps: Step 1: Take photos of the interior of the hazardous chemical storage warehouse from at least two different angles; Step 2: Extract image features from each photo to obtain feature points and descriptors for each feature point; the image feature extraction algorithm used is a self-supervised machine learning algorithm. In step two, the algorithm used is the SuperPoint feature extraction network. The feature points extracted by the SuperPoint algorithm include points in the floor seams and wall corners. These points are invalid for 3D reconstruction and will cause interference, so they will be discarded in step three. Step 3: Mark the outline of the hazardous chemical stacks in each photo. If a feature point in a photo falls outside the outline, remove that feature point. Step 3 is as follows: Use object detection algorithms to identify hazardous chemical stacks, use regular geometric shapes to frame each hazardous chemical stack, and then remove feature points located outside the frames; Step 4: Match the feature points obtained in Step 2 to obtain feature point pairs; Step 5: Use the feature point pairs from Step 4 to perform 3D reconstruction, and then monitor whether the five distances of the hazardous chemical stacks in the 3D model meet the requirements; during 3D reconstruction, exclude outliers located within the boxes from Step 3.

2. The method for detecting the safe distance of hazardous chemical stacks adaptable to different lighting conditions according to claim 1, characterized in that: In step one, a calibrated binocular camera is used for shooting.

3. The method for detecting the safe distance of hazardous chemical stacks adaptable to different lighting conditions according to claim 2, characterized in that: The binocular camera was calibrated using the Zhang Zhengyou calibration method.

4. The method for detecting the safe distance of hazardous chemical stacks adaptable to different lighting conditions according to claim 1, characterized in that: In step four, the SuperGule feature matching network is used for feature point matching.

5. The method for detecting the safe distance of hazardous chemical stacks adaptable to different lighting conditions according to claim 4, characterized in that: In step three, the target detection algorithm used is YOLOv5.

6. The method for detecting the safe distance of hazardous chemical stacks adaptable to different lighting conditions according to claim 4, characterized in that: In step three, a rectangular frame with each side tangent to the edge of the hazardous chemical stack is used for selection.

7. The method for detecting the safe distance of hazardous chemical stacks adaptable to different lighting conditions according to claim 1, characterized in that: The floor of the hazardous chemical storage warehouse is a solid color floor without grid lines or decorations.

8. The method for detecting the safe distance of hazardous chemical stacks adaptable to different lighting conditions according to claim 1, characterized in that: In step five, the 3D model is a colorless point cloud model.

Images