Hoisting operation safety detection method and device based on binocular vision and storage medium

Abstract

The invention relates to a hoisting operation safety detection method and device based on binocular vision and a storage medium, and belongs to the technical field of constructional engineering.The method comprises the steps that a target image is segmented from hoisting operation images collected by a binocular camera; the target detection object comprises a lifting arm, a lifting object and a pedestrian; feature point comparison is carried out on the target detection object based on an ORB feature point matching algorithm, and target feature points are determined; determining a target distance between the target detection object and the binocular camera based on the target feature point; and carrying out safety detection on the hoisting operation based on the target distance. According to the hoisting operation safety detection method based on binocular vision, the distance between the hoisting arm, the hoisting object and the pedestrian and the binocular camera can be obtained respectively, so that whether the pedestrian is within the hoisting operation range or not can be judged according to the target distance, safety detection on the hoisting operation is achieved, the hoisting operation is efficiently monitored in real time, and the safety of the hoisting operation is improved. And the safety in the operation process is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of construction engineering, and in particular to a binocular vision-based hoisting operation safety detection method, device and storage medium. Background Art

[0002] With the continuous acceleration of urbanization, the use of lifting operations in the field of construction projects is increasing. During lifting operations, the safety of personnel within the operating radius is extremely important. Due to the limitations of the blind spots of the driver and workers, safety accidents may occur due to improper observation during lifting operations. If the lifting driver cannot observe the dangerous situation within the operating radius in time, the hoisted objects may fall off, endangering the safety of the workers.

[0003] Surveillance cameras are widely used in the construction field because of their many advantages such as low cost, strong anti-interference, long service life and good visualization effect. The current methods for detecting safe hoisting operations mainly include installing distance measuring sensors and installing cameras for image recognition to determine the relationship between targets. Both methods have certain limitations. Among them, installing distance measuring sensors is expensive and is greatly affected by the use site and environment. Direct image acquisition and image processing usually use a deep learning-based target detection algorithm to complete the detection of cranes and pedestrians, and output whether the pedestrian is within the scope of the hoisting operation by judging whether the rectangular frames of the crane and the pedestrian overlap. Since the image only contains two-dimensional information, it is impossible to determine the positional relationship between the crane, especially the boom and the pedestrian, which will trigger a large number of false alarm messages in the project, resulting in poor detection effect and low safety. Summary of the invention

[0004] In view of this, it is necessary to provide a hoisting operation safety detection method, device and storage medium based on binocular vision to solve the problems of poor detection effect and low safety of existing safety detection algorithms.

[0005] In order to solve the above problems, the present invention provides a hoisting operation safety detection method based on binocular vision, comprising: Segmenting a target image from an image of a hoisting operation captured by a binocular camera; the target image is an image containing a target detection object; the target detection object includes a crane arm, a hoisted object, and a pedestrian; Perform feature point comparison on the target detection object based on the ORB feature point matching algorithm to determine the target feature points; Based on the target feature points, determining a target distance between the target detection object and the binocular camera; Based on the target distance, a safety check is performed on the lifting operation.

[0006] In a possible implementation, segmenting a target image from an image of a hoisting operation captured by a binocular camera includes: Inputting the image of the lifting operation into a trained target detection model to obtain a labeled image output by the target detection model; the labeled image is an image that labels the target detection object; The labeled image is input into a trained target segmentation model to obtain the target image output by the target segmentation model; the target detection model and the target segmentation model are trained by the YOLOV8 algorithm.

[0007] In a possible implementation, determining the target distance between the target detection object and the binocular camera based on the target feature point includes: Based on the target feature points, determining a disparity value of the binocular camera; The target distance is determined based on a baseline distance, a focal length and the disparity value of the binocular camera.

[0008] In a possible implementation, the target distance is expressed as follows:

[0009] in, Indicates the target distance, represents the disparity value, represents the baseline distance, Indicates focal length.

[0010] In a possible implementation, the target distance includes a first distance between the boom and the binocular camera, a second distance between the suspended object and the binocular camera, and a third distance between the pedestrian and the binocular camera; The safety detection of the hoisting operation based on the target distance includes: When the difference between the first distance and the third distance is smaller than a preset threshold or the difference between the second distance and the third distance is larger than a preset threshold, a safety warning message is issued.

[0011] In a possible implementation, before segmenting the target image from the image of the hoisting operation captured by the binocular camera, the method further includes: The binocular camera is calibrated and corrected.

[0012] The present invention also provides a binocular vision-based hoisting operation safety detection device, comprising: A segmentation module is used to segment a target image from the image of the hoisting operation captured by the binocular camera; the target image is an image containing a target detection object; the target detection object includes a crane arm, a hoisted object, and a pedestrian; A first determination module is used to compare the feature points of the target detection object based on the ORB feature point matching algorithm to determine the target feature points; A second determination module is used to determine the target distance between the target detection object and the binocular camera based on the target feature point; The detection module is used to perform safety detection on the hoisting operation based on the target distance.

[0013] On the other hand, the present invention also provides an electronic device, including a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the binocular vision-based hoisting operation safety detection method described in any of the above implementation methods.

[0014] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the binocular vision-based hoisting operation safety detection method described in any of the above-mentioned implementation methods is implemented.

[0015] On the other hand, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the binocular vision-based hoisting operation safety detection method described in any of the above implementation methods.

[0016] The beneficial effects of the present invention are as follows: the binocular vision-based hoisting operation safety detection method provided by the present invention, by segmenting a target image containing target detection objects such as a crane arm, a hoisted object and a pedestrian from the image of the hoisting operation captured by a binocular camera, thereby performing feature point comparison on the target detection object according to an ORB feature point matching algorithm to obtain target feature points, and determining the target distance between the target detection object and the binocular camera according to the target feature points, that is, the distance between the crane arm, the hoisted object and the pedestrian and the binocular camera can be obtained respectively, so that whether the pedestrian is within the range of the hoisting operation can be judged according to the target distance, thereby realizing safety detection of the hoisting operation, and real-time and efficient monitoring of the hoisting operation to ensure safety during the operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1A method flow chart of an embodiment of a hoisting operation safety detection method based on binocular vision provided by the present invention; Figure 2 A schematic diagram of a chessboard provided by the present invention; Figure 3 A schematic diagram of the front side of the binocular vision system provided by the present invention; Figure 4 A schematic diagram of the back side of the binocular vision system provided by the present invention; Figure 5 A schematic diagram of using a binocular vision system to collect images of a calibration plate provided by the present invention; Figure 6 A schematic diagram of the binocular vision system provided by the present invention before stereo correction; Figure 7 A schematic diagram of a binocular vision system after stereo correction provided by the present invention; Figure 8 A flowchart of the binocular vision system provided by the present invention; Fig. 9 A schematic diagram of the binocular vision system provided by the present invention for calculating target distance; Fig.10 A schematic structural diagram of an embodiment of a hoisting operation safety detection device based on binocular vision provided by the present invention; Fig.11 A schematic structural diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0020] In the description of the embodiments of the present invention, unless otherwise specified, "multiple" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone.

[0021] The descriptions of "first", "second", etc. involved in the embodiments of the present invention are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the technical features defined as "first" or "second" may explicitly or implicitly include at least one of the features.

[0022] Figure 1 A method flow chart of an embodiment of the hoisting operation safety detection method based on binocular vision provided by the present invention, as shown in Figure 1 As shown in the figure, the hoisting operation safety detection method based on binocular vision includes: S101, segmenting a target image from an image of a hoisting operation captured by a binocular camera; the target image is an image containing a target detection object; the target detection object includes a crane arm, a hoisted object, and a pedestrian; S102, performing feature point comparison on the target detection object based on the ORB feature point matching algorithm to determine the target feature points; S103, determining a target distance between the target detection object and the binocular camera based on the target feature points; S104: Perform safety detection on the hoisting operation based on the target distance.

[0023] Compared with the prior art, the binocular vision-based safety detection method for lifting operations provided in an embodiment of the present invention segments a target image including target detection objects such as a crane arm, a hanging object and a pedestrian from the image of the lifting operation captured by a binocular camera, thereby performing feature point comparison on the target detection object according to an ORB feature point matching algorithm to obtain target feature points, and determining the target distance between the target detection object and the binocular camera according to the target feature points, that is, the distance between the crane arm, the hanging object and the pedestrian and the binocular camera can be obtained respectively, so that whether the pedestrian is within the range of the lifting operation can be judged according to the target distance, thereby realizing safety detection of the lifting operation, and real-time and efficient monitoring of the lifting operation to ensure safety during the operation.

[0024] In some embodiments of the present invention, before segmenting the target image from the image of the hoisting operation captured by the binocular camera, the method further includes: The binocular camera is calibrated and corrected.

[0025] It is understandable that before collecting images of hoisting operations through a binocular camera, it is necessary to select a camera, make a calibration plate, build a binocular vision system, and calibrate and correct the binocular camera.

[0026] First, it is necessary to select cameras and make calibration plates. In order to ensure that the image information collected by the two cameras is the same, this embodiment selects two Basler series Basler acA1920-40gc area array industrial cameras, and uses Basler LensC23-3520-5M-P f35mm camera lenses, and prepares a 10*7 chessboard with each grid size of 100*100mm. Figure 2 As shown, Figure 2 A schematic diagram of a chessboard provided by the present invention.

[0027] Secondly, a binocular vision system needs to be built for image acquisition. Figure 3 A schematic diagram of the front side of the binocular vision system provided by the present invention, Figure 4 A schematic diagram of the back side of the binocular vision system provided by the present invention.

[0028] The two cameras are placed horizontally and fixed by buckles. The distance between the two cameras is b. In this embodiment, b is 300 mm. Figure 5 The schematic diagram of the present invention for collecting the image of the calibration plate using the binocular vision system is as follows: Figure 5 As shown, the two cameras are kept parallel to construct a parallel binocular vision system, and the calibration plate is placed in the camera field of view. After ensuring that the calibration plate completely covers all positions in the image area, multiple groups of images are saved. In this embodiment, a total of 42 groups are collected.

[0029] Then, calibrate the stereo camera. You can use the stereoCamera Calibrator stereo vision calibration tool in the MatLab2021a software toolbox to calibrate and calculate the radial distortion coefficient and tangential distortion to obtain the intrinsic parameter matrix, distortion coefficient, rotation R and translation T matrix of the two cameras. The intrinsic parameter matrix contains the focal length and optical center information of the camera. The composition of the intrinsic parameter matrix is ​​as follows:

[0030] in, , is the focal length of the camera, , Represents the optical center of the camera.

[0031] The distortion coefficient includes radial distortion k and tangential distortion p, and its composition is as follows:

[0032] Finally, the binocular vision system is stereo rectified. Figure 6 This is a schematic diagram of the binocular vision system provided by the present invention before stereo correction, Figure 7 This is a schematic diagram of the binocular vision system after stereo correction provided by the present invention, such as Figure 6 and Figure 7 As shown in the figure, there will be certain errors in the construction and installation of the binocular vision system, which will cause the position relationship between the two cameras to be not completely parallel, affecting the subsequent matching and measurement. Therefore, it is necessary to calibrate the entire binocular vision system to eliminate image distortion and use the epipolar constraint to make the epipolar lines of the left and right images exactly on the same horizontal line, so as to improve the speed of the subsequent matching algorithm. The Bouguet algorithm is used for stereo correction. It is assumed that the external parameter matrix of the binocular camera obtained by calibration is R and T ,but:

[0033] in, , They are the three-dimensional coordinates of a point in the same world coordinate system in the left camera coordinate system and the right camera coordinate system, thus completing the stereo correction process of the binocular vision system.

[0034] The binocular vision-based hoisting operation safety detection method provided by the embodiment of the present invention can accurately calculate the external parameters (rotation matrix and translation vector) between the binocular cameras by calibrating and correcting the binocular cameras, so that stereo vision reconstruction can be performed more accurately and the accuracy of measurement and positioning can be improved; the camera's internal parameters (camera matrix and distortion coefficient) can be obtained, so that the influence of lens distortion can be effectively eliminated, making the image more realistic and accurate.

[0035] In some embodiments of the present invention, segmenting a target image from an image of the hoisting operation captured by a binocular camera includes: Inputting the image of the lifting operation into a trained target detection model to obtain a labeled image output by the target detection model; the labeled image is an image that labels the target detection object; The labeled image is input into a trained target segmentation model to obtain the target image output by the target segmentation model; the target detection model and the target segmentation model are trained by the YOLOV8 algorithm.

[0036] Before segmenting the target image from the image of the hoisting operation captured by the binocular camera, it is necessary to use the binocular camera to capture the image of the hoisting operation, which includes target detection objects such as the crane, the hoisted object, the boom and pedestrians, and the image resolution is 1920*1080. The present invention uses a binocular camera to collect a total of 40,000 pieces of hoisting operation data on different construction sites, including data under various weather conditions, in order to improve the generalization ability and robustness of the model.

[0037] After data collection is completed, the YOLOV8 target detection algorithm based on deep learning can be used to detect the target objects that need to be detected in the lifting operation.

[0038] Specifically, the LabelImg annotation tool can be used to annotate the target to be detected. The annotation box is a rectangular box, and the label starts from 0. After the data is annotated, write code to divide the model training set, validation set, and test set in a ratio of 7:2:1, and perform model training on a Linux server equipped with two 3080TI 20G video memory to output the target detection model.

[0039] By inputting the image of the hoisting operation into the target detection model through the trained target detection model, the labeled image output by the target detection model can be obtained. It can be understood that the target detection objects such as the crane, the hoisted object, the boom and the pedestrian are respectively labeled with rectangular boxes in the labeled image.

[0040] After the image annotation is completed, the target area of ​​the rectangular frame, especially during hoisting operations, after the boom is extended, the pixels of the non-boom area in the rectangular area are larger than the boom itself, forming a strong background interference, which affects the subsequent calculation of the boom distance.

[0041] Therefore, the YOLOV8 instance segmentation algorithm can be used to segment the boom area. Before that, it is necessary to write code and combine the target detection model to crop the rectangular box after target detection to obtain an image that only contains the boom area and the background.

[0042] In this example, 10,000 crane area data were selected and Labelme was used to annotate the cranes. The annotation was done by selecting points and drawing to form a closed polygonal area. After the data annotation was completed, the code was written to divide the model training set, validation set, and test set in a ratio of 7:2:1. The model was trained on a Linux server equipped with two 3080TI 20G video memory cards, and the target segmentation model was output.

[0043] By inputting the labeled image into the target segmentation model through the trained target segmentation model, the target image output by the target detection model can be obtained. The target image is the segmented image, that is, the image obtained by segmenting the rectangular frame part in the labeled image.

[0044] The binocular vision-based hoisting operation safety detection method provided in the embodiment of the present invention completes the detection of targets such as hoisting and pedestrians by combining deep learning technology target detection and instance segmentation algorithm. When calculating whether there is a person standing under the boom, the actual area of ​​the boom is determined by adding an instance segmentation algorithm to reduce background interference, thereby providing strong technical support for efficient and accurate judgment of the safety of hoisting operations.

[0045] In some embodiments of the present invention, determining the target distance between the target detection object and the binocular camera based on the target feature point includes: Based on the target feature points, determining a disparity value of the binocular camera; The target distance is determined based on a baseline distance, a focal length and the disparity value of the binocular camera.

[0046] In some embodiments of the present invention, the expression of the target distance is as follows:

[0047] in, Indicates the target distance, represents the disparity value, represents the baseline distance, Indicates focal length.

[0048] After completing target detection and instance segmentation, the ORB feature point matching algorithm can be used to compare feature points of the same target object to obtain the target feature points. The ORB feature point matching algorithm is an algorithm that can quickly extract and describe feature points in an image. It is mainly composed of the improved FAST feature detection algorithm and the BRIEF feature description algorithm. FAST feature detection and BRIEF feature description are achieved through coarse extraction, point screening, feature point scale invariance and rotation invariance. And complete the corresponding relationship mapping of the same feature point in the left and right images of the binocular camera.

[0049] Figure 8 The work flow chart of the binocular vision system provided by the present invention is as follows: Figure 8 As shown in the figure, in the binocular vision system, the main process includes four steps: camera calibration, image correction, stereo matching and triangulation positioning.

[0050] The embodiment of the present invention adopts the ORB feature point detection algorithm in the stereo matching part, which improves the feature matching accuracy and speed. Without generating a disparity map through stereo matching, the binocular vision principle is used to calculate the final disparity value. According to the binocular vision principle, the disparity value represents the pixel difference of the same pixel point in the left and right images.

[0051] Therefore, through the ORB feature point matching algorithm in the above steps, the matching of the same pixel on the left and right can be completed, and according to the definition of the disparity value Complete the disparity calculation, where Indicates the horizontal coordinate of the target feature point under the left camera, Indicates the horizontal coordinate of the target feature point under the right camera.

[0052] In a binocular vision system, the target distance can be calculated by constructing similar triangles. Fig. 9 The schematic diagram of the binocular vision system provided by the present invention for calculating the target distance is as follows: Fig. 9 As shown, the optical center positions of the left and right cameras are marked as points A and B, the position of the target object is E, and the image coordinates of the target object E under the left and right cameras are respectively expressed as , According to the principle of similar triangles, we can get:

[0053]

[0054] After simplification:

[0055] The target distance takes a positive value, so the above formula can be transformed as follows:

[0056] in, It is called parallax, which indicates the difference in the horizontal coordinates of the image coordinates of the target object in the left and right images. They represent the baseline distance and camera focal length respectively, which can be directly measured and are fixed values.

[0057] In some embodiments of the present invention, the target distance includes a first distance between the boom and the binocular camera, a second distance between the hanging object and the binocular camera, and a third distance between the pedestrian and the binocular camera; The safety detection of the hoisting operation based on the target distance includes: When the difference between the first distance and the third distance is smaller than a preset threshold or the difference between the second distance and the third distance is larger than a preset threshold, a safety warning message is issued.

[0058] Through the binocular vision algorithm, the target distance between the corresponding target detection object and the camera in the hoisting operation can be completed. Therefore, the target distance includes the first distance between the boom and the binocular camera d 1. The second distance between the hanging object and the binocular camera d 2, and the third distance between the pedestrian and the stereo camera d 3.

[0059] After the target distance is calculated, it is determined whether there are pedestrians within the lifting operation radius, that is, whether the distance difference between the hoisted object and the boom and the pedestrians is within the preset threshold during the lifting operation. Within the alarm range, when the preset threshold is exceeded When a warning message is given.

[0060] Right now or When it is determined that there are pedestrians within the lifting operation radius, a safety warning message will be issued.

[0061] The preset threshold It can be set according to the actual warning distance.

[0062] The binocular vision-based hoisting operation safety detection method provided in the embodiment of the present invention determines whether there are pedestrians within the hoisting operation radius through the target distance, performs safety detection on the hoisting operation, realizes real-time and efficient monitoring of the hoisting operation, and ensures safety during the operation.

[0063] Exemplarily, the binocular vision-based hoisting operation safety detection method provided by an embodiment of the present invention includes: Step S1, perform preparation work, select two camera devices of the same model, and draw a checkerboard.

[0064] Step S2: Building a binocular vision system.

[0065] Place two cameras parallel to each other and fix them, and measure the baseline distance between the two cameras. b , the left camera is used to synchronously trigger the right camera, and the chessboard is used for image acquisition. After ensuring that both cameras can fully acquire the information of the chessboard, the distance between the chessboard and the camera is continuously adjusted to acquire multiple sets of data containing images from the left and right cameras.

[0066] Step S3: camera calibration.

[0067] Use Matlab to perform binocular vision calibration and obtain the camera's internal and external parameters and distortion coefficients.

[0068] Step S4: stereo calibration.

[0069] There will be certain errors in the construction and installation of the binocular vision system, which will cause the positional relationship between the two cameras to be completely parallel. The stereo correction algorithm is used to make the left and right images on the same horizontal line, which is convenient for improving the speed of the subsequent matching algorithm.

[0070] Step S5: target detection algorithm.

[0071] When using the target detection algorithm based on YOLOV8, you need to first collect images in the lifting operation scene. The images include target information such as pedestrians and crane arms. After the image is collected, use the LabelImg annotation tool to annotate the targets in the image. After the annotation is completed, write code to divide the detection image data set into training set, validation set and test set in a ratio of 7:2:1, and train the YOLOV8 target detection network on a 3080Ti 20G graphics card.

[0072] Step S6: boom segmentation algorithm.

[0073] Using the instance segmentation algorithm based on YOLOV8, in step S5, the target detection will only detect a rectangular area of ​​the arm, which contains a large amount of background area. In the later calculation, it will affect the calculation of the arm distance. When the detection target includes the arm, it is necessary to use the segmentation algorithm to remove the background area of ​​the arm. The Labelme annotation software is used to annotate the small image of the detected arm and mark the area of ​​the selected arm. After the annotation is completed, the code is written to divide the segmented data set into training set, validation set and test set according to the ratio of 7:2:1, and the YOLOV8 instance segmentation network is trained on the 3080Ti 20G graphics card.

[0074] Step S7: feature point matching.

[0075] Traditional feature matching uses a stereo matching method, which has a large amount of calculation and difficult parameter adjustment. Unlike the traditional binocular vision stereo matching algorithm, the present invention combines a deep learning target detection algorithm and a feature point matching algorithm to replace stereo matching. After completing steps S6 and S7, the binocular camera video data is connected to the algorithm model, and the images captured by the left and right cameras at the same time and in the same direction are detected.

[0076] After target detection and segmentation, the ORB (Oriented Fast and Rotated BRIEF) feature matching algorithm is used to perform similarity analysis on the targets of the left and right cameras. In order to speed up the matching speed, while ensuring that the detected targets are of the same category, the ORB algorithm is used to find the position of the same target in the left and right cameras and obtain the corresponding key point information.

[0077] Step S8: parallax calculation.

[0078] In step S7, multiple sets of feature point correspondences are obtained. The key point coordinate information of the left camera is subtracted from the key point information of the right camera to form a disparity array, and the final disparity value is obtained by the median method. .

[0079] Step S9: triangulation positioning distance calculation.

[0080] After the camera is calibrated in step S2, the camera's intrinsic parameter matrix is ​​obtained. The intrinsic parameter matrix contains the focal length of the camera in the x and y directions. fx , fy , the baseline distance between the two cameras is measured in step S1 b , according to the principle of binocular vision, calculate the target distance d :

[0081] According to the distance calculation formula, calculate the distance between pedestrians and crane arms respectively.

[0082] Step S10: Safety assessment of the hoisting operation.

[0083] Through the distance relationship between pedestrians and the crane arm, it is determined whether the pedestrian is within the lifting operation radius to complete the construction of the entire system.

[0084] The present invention improves the target matching speed and accuracy by constructing a binocular vision system and improving the existing binocular vision calculation method; it completes the detection of targets such as hoisting and pedestrians by combining deep learning technology target detection and instance segmentation algorithm; when calculating whether there is a person standing under the boom, the actual area of ​​the boom is determined by adding an instance segmentation algorithm to reduce background interference, thereby providing strong technical support for efficient and accurate judgment of the safety of hoisting operations, and has great engineering significance.

[0085] In order to better implement the hoisting operation safety detection method based on binocular vision in the embodiment of the present invention, on the basis of the hoisting operation safety detection method based on binocular vision, the embodiment of the present invention also provides a hoisting operation safety detection device based on binocular vision. Fig.10 The structure diagram of an embodiment of the hoisting operation safety detection device based on binocular vision provided by the present invention is as follows Fig.10 As shown, the hoisting operation safety detection device 1000 based on binocular vision includes: The segmentation module 1010 is used to segment a target image from the image of the hoisting operation captured by the binocular camera; the target image is an image containing a target detection object; the target detection object includes a crane arm, a hoisted object, and a pedestrian; A first determination module 1020 is used to compare the feature points of the target detection object based on the ORB feature point matching algorithm to determine the target feature points; A second determination module 1030 is used to determine a target distance between the target detection object and the binocular camera based on the target feature point; The detection module 1040 is used to perform safety detection on the hoisting operation based on the target distance.

[0086] The binocular vision-based hoisting operation safety detection device 1000 provided in the above embodiment can implement the technical solution described in the above embodiment of the binocular vision-based hoisting operation safety detection method. The specific implementation principles of the above modules or units can refer to the corresponding contents in the embodiment of the binocular vision-based hoisting operation safety detection method, which will not be repeated here.

[0087] like Fig.11 As shown, the present invention also provides an electronic device 1100. The electronic device 1100 includes a processor 1101, a memory 1102 and a display 1103. Fig.11Only some components of the electronic device 1100 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0088] In some embodiments, the processor 1101 can be a central processing unit (CPU), a microprocessor or other data processing chip, which is used to run the program code or process data stored in the memory 1102, such as the binocular vision-based hoisting operation safety detection method in the present invention.

[0089] In some embodiments, processor 1101 may be a single server or a server group. The server group may be centralized or distributed. In some embodiments, processor 1101 may be local or remote. In some embodiments, processor 1101 may be implemented in a cloud platform. In some embodiments, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-cloud, etc., or any combination thereof.

[0090] In some embodiments, the memory 1102 may be an internal storage unit of the electronic device 1100, such as a hard disk or memory of the electronic device 1100. In other embodiments, the memory 1102 may also be an external storage device of the electronic device 1100, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the electronic device 1100.

[0091] Furthermore, the memory 1102 may include both an internal storage unit of the electronic device 1100 and an external storage device. The memory 1102 is used to store application software installed in the electronic device 1100 and various data.

[0092] In some embodiments, the display 1103 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an organic light-emitting diode (OLED) touch device, etc. The display 1103 is used to display information on the electronic device 1100 and to display a visual user interface. The components 1101-1103 of the electronic device 1100 communicate with each other via a system bus.

[0093] In one embodiment, when the processor 1101 executes the binocular vision-based hoisting operation safety detection program in the memory 1102, the following steps may be implemented: Segmenting a target image from an image of a hoisting operation captured by a binocular camera; the target image is an image containing a target detection object; the target detection object includes a crane arm, a hoisted object, and a pedestrian; Perform feature point comparison on the target detection object based on the ORB feature point matching algorithm to determine the target feature points; Based on the target feature points, determining a target distance between the target detection object and the binocular camera; Based on the target distance, a safety check is performed on the lifting operation.

[0094] It should be understood that: when the processor 1101 executes the binocular vision-based hoisting operation safety detection program in the memory 1102, in addition to the above functions, other functions can also be implemented. For details, please refer to the description of the corresponding method embodiment above.

[0095] Furthermore, the embodiment of the present invention does not specifically limit the type of the electronic device 1100 mentioned, and the electronic device 1100 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic devices may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 1100 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0096] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the steps or functions of the binocular vision-based lifting operation safety detection method provided in the above-mentioned method embodiments.

[0097] Accordingly, an embodiment of the present application also provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the program or instructions are executed by a processor, it can implement the steps or functions of the binocular vision-based hoisting operation safety detection method provided in the above-mentioned method embodiments.

[0098] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0099] The above is a detailed introduction to the binocular vision-based hoisting operation safety detection method, device and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for detecting safety of hoisting operations based on binocular vision, characterized in that: include: Segmenting a target image from an image of the hoisting operation captured by a binocular camera; the target image is an image containing a target detection object; The target detection objects include a crane arm, a crane object and a pedestrian; Perform feature point comparison on the target detection object based on the ORB feature point matching algorithm to determine the target feature points; Based on the target feature points, determining a target distance between the target detection object and the binocular camera; Based on the target distance, a safety check is performed on the lifting operation.

2. The binocular vision-based hoisting operation safety detection method according to claim 1 is characterized in that: The step of segmenting a target image from an image of the hoisting operation captured by a binocular camera comprises: Inputting the image of the lifting operation into a trained target detection model to obtain a labeled image output by the target detection model; the labeled image is an image that labels the target detection object; The labeled image is input into a trained target segmentation model to obtain the target image output by the target segmentation model; the target detection model and the target segmentation model are trained by the YOLOV8 algorithm.

3. The binocular vision-based hoisting operation safety detection method according to claim 1 is characterized in that: The step of determining the target distance between the target detection object and the binocular camera based on the target feature point includes: Based on the target feature points, determining a disparity value of the binocular camera; The target distance is determined based on a baseline distance, a focal length and the disparity value of the binocular camera.

4. The binocular vision-based hoisting operation safety detection method according to claim 3 is characterized in that: The expression of the target distance is as follows: in, Indicates the target distance, represents the disparity value, represents the baseline distance, Indicates focal length.

5. The binocular vision-based hoisting operation safety detection method according to claim 1 is characterized in that: The target distance includes a first distance between the boom and the binocular camera, a second distance between the hanging object and the binocular camera, and a third distance between the pedestrian and the binocular camera; The safety detection of the hoisting operation based on the target distance includes: When the difference between the first distance and the third distance is smaller than a preset threshold or the difference between the second distance and the third distance is larger than a preset threshold, a safety warning message is issued.

6. The binocular vision-based hoisting operation safety detection method according to claim 1 is characterized in that: Before segmenting the target image from the image of the hoisting operation collected by the binocular camera, the method further includes: The binocular camera is calibrated and corrected.

7. A hoisting operation safety detection device based on binocular vision, characterized in that: include: A segmentation module is used to segment a target image from an image of a hoisting operation captured by a binocular camera; The target image is an image containing the target detection object; The target detection objects include a crane arm, a crane object and a pedestrian; A first determination module is used to compare the feature points of the target detection object based on the ORB feature point matching algorithm to determine the target feature points; A second determination module is used to determine the target distance between the target detection object and the binocular camera based on the target feature point; The detection module is used to perform safety detection on the hoisting operation based on the target distance.

8. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the binocular vision-based hoisting operation safety detection method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the binocular vision-based hoisting operation safety detection method as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the binocular vision-based hoisting operation safety detection method as described in any one of claims 1 to 6 is implemented.

Images