Air duct hook detection method, device, electronic equipment and storage medium

Abstract

The present invention provides a method, device, electronic device and storage medium for detecting air duct hooks, wherein the method comprises: performing image detection on an image data stream to obtain a two-dimensional image detection result; performing joint calibration on a point cloud data stream and an image data stream to obtain a joint calibration result; performing filtering and target segmentation on the point cloud data stream to obtain a point cloud three-dimensional detection result, and converting the point cloud three-dimensional detection result into a projection result of a three-dimensional point cloud to a two-dimensional image based on the joint calibration result; determining a fusion result based on the overlap between a detection frame of the two-dimensional image detection result and a detection frame of the projection result, and determining the target air duct hook category and the target spatial position of the target air duct hook category based on the fusion result. On the one hand, the use of a multi-sensor fusion method can more accurately and efficiently obtain the target air duct hook category and the target spatial position; on the other hand, it can replace manual observation, reduce operation risks, and significantly improve work efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a method, a device, an electronic device and a storage medium for detecting a hook of an air duct. Background Art

[0002] At present, all major railway marshalling yards have been equipped with comprehensive automation systems, which centrally manage the control equipment of each yard and achieve a high degree of informatization and integration. However, the removal of air ducts and hooks in marshalling yards still mainly relies on manual work, which is labor-intensive and repetitive, resulting in low work efficiency and high labor costs.

[0003] In the process of detecting hooks of air ducts in railway marshalling yards, existing technical means mainly face the following problems: 1) Low detection efficiency: Manual visual inspection is labor-intensive and slow, and cannot meet the detection needs of large-scale production; and the single sensor system requires multiple inspections and calibrations, which increases the complexity and maintenance cost of the system. 2) Insufficient accuracy: Although a single sensor (such as photoelectric, ultrasonic, etc.) can provide some information, it lacks comprehensive processing capabilities and cannot maintain stable detection accuracy under different environmental conditions. 3) High safety risks: There are high safety risks in relying on manual operation, especially in busy or high-risk working environments, which are prone to accidents. Summary of the invention

[0004] The present invention provides a method, a device, an electronic device and a storage medium for detecting an air duct hook, so as to solve the defects of low detection efficiency, insufficient accuracy and high safety risk of air duct hook detection in the prior art.

[0005] The present invention provides a method for detecting a duct hook, comprising the following steps: Obtaining a point cloud data stream scanned by a laser radar and an image data stream captured by a camera, and performing image detection on the image data stream to obtain a two-dimensional image detection result; Performing joint calibration on the point cloud data stream and the image data stream to obtain a joint calibration result; Filtering and segmenting the point cloud data stream to obtain a point cloud three-dimensional detection result, and converting the point cloud three-dimensional detection result into a projection result of a three-dimensional point cloud to a two-dimensional image based on the joint calibration result; A fusion result is determined based on the degree of overlap between the detection frame of the two-dimensional image detection result and the detection frame of the projection result, and a target air duct hook category and a target spatial position of the target air duct hook category are determined based on the fusion result.

[0006] According to a method for detecting air duct hooks provided by the present invention, determining a target air duct hook category and a target spatial position of the target air duct hook category based on the fusion result includes: Determine a spatial position offset based on a first spatial position of a target center in a current fusion result and a second spatial position of a target center in a previous fusion result; the previous fusion result is a fusion result of a preset number of times before the current fusion result; When the spatial position offset is less than a first preset threshold, the target air duct hook category and the target spatial position in the current fusion result are determined.

[0007] According to a method for detecting air duct hooks provided by the present invention, the fusion result is determined based on the overlap degree between the detection frame of the two-dimensional image detection result and the projection result, including: When the overlap degree is greater than a second preset threshold, the duct hook category in the two-dimensional image detection result and the spatial position of the duct hook category in the point cloud three-dimensional detection result are used as the fusion result.

[0008] According to a method for detecting air duct hooks provided by the present invention, performing image detection on the image data stream to obtain a two-dimensional image detection result includes: Using an image detection model, performing image detection on the image data stream to obtain the two-dimensional image detection result; The training steps of the image detection model include: Acquire sample image data and an initial image detection model, and annotate the image data of the air duct and the retractor in the sample image data to obtain image labels; the sample image data includes image data of the air duct and the retractor in each scene; Inputting the sample image data into the initial image detection model to obtain the image detection prediction result output by the initial image detection model; Based on the difference between the image detection prediction result and the image label, a target loss is determined, and based on the target loss, parameters of the initial image detection model are iterated to obtain the image detection model.

[0009] According to a duct hook detection method provided by the present invention, the joint calibration includes camera intrinsic parameter calibration, sensor time synchronization, and sensor space synchronization.

[0010] According to a method for detecting air duct hooks provided by the present invention, the sensor spatial synchronization includes coarse calibration and fine calibration; The rough calibration is performed based on an automated calibration algorithm; The fine calibration is a manual calibration based on the rough calibration.

[0011] According to a method for detecting air duct hooks provided by the present invention, filtering and target segmenting the point cloud data stream to obtain a point cloud three-dimensional detection result includes: Performing a preprocessing operation on the point cloud data stream to obtain preprocessed point cloud data; the preprocessing operation includes at least one of ROI filtering, NAN point filtering and downsampling filtering; Filtering the ground point cloud data in the pre-processed point cloud data to obtain candidate point cloud data; Performing point cloud angle segmentation on the candidate point cloud data to obtain a point cloud target segmentation result; The point cloud target segmentation result is filtered to obtain the point cloud three-dimensional detection result.

[0012] The present invention also provides a duct hook detection device, comprising the following modules: An acquisition unit is used to acquire a point cloud data stream scanned by a laser radar and an image data stream taken by a camera, and perform image detection on the image data stream to obtain a two-dimensional image detection result; A joint calibration unit, used for jointly calibrating the point cloud data stream and the image data stream to obtain a joint calibration result; A conversion unit, configured to filter and segment the point cloud data stream to obtain a point cloud three-dimensional detection result, and convert the point cloud three-dimensional detection result into a projection result based on the joint calibration result; A determination unit is used to determine a fusion result based on the degree of overlap between the detection frame of the two-dimensional image detection result and the detection frame of the projection result, and to determine a target air duct hook category and a target spatial position of the target air duct hook category based on the fusion result.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned air duct hook detection methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the air duct hook detection method as described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned air duct hook detection methods.

[0016] The duct hook detection method, device, electronic device and storage medium provided by the present invention obtain a joint calibration result by jointly calibrating the point cloud data stream and the image data stream; filtering and target segmenting the point cloud data stream to obtain a point cloud three-dimensional detection result, and converting the point cloud three-dimensional detection result into a projection result of the three-dimensional point cloud to a two-dimensional image based on the joint calibration result; determining the fusion result based on the overlap between the detection frame of the two-dimensional image detection result and the detection frame of the projection result, and determining the target duct hook category and the target spatial position of the target duct hook category based on the fusion result. On the one hand, the use of multi-sensor fusion can more accurately and efficiently obtain the target duct hook category and the target spatial position; on the other hand, it can replace manual observation, reduce operation risks, and significantly improve work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is one of the flow charts of the air duct hook detection method provided by the present invention.

[0019] Figure 2 It is a schematic diagram of the fusion result provided by the present invention.

[0020] Figure 3 This is the second flow chart of the air duct hook detection method provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the air duct hook detection device provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] In the relevant technologies, the removal of air ducts and hooks in the marshaling yard is still mainly done manually. These operations are labor-intensive and repetitive, resulting in low work efficiency and high labor costs. At the same time, manual operations have high safety risks and are prone to cause safety accidents. Although with the continuous advancement of intelligent technology, technologies such as image recognition, industrial robots and laser ranging have gradually matured and have been widely used in various fields, these advanced technologies have not yet been effectively integrated and applied in the detection of air duct hooks in railway marshaling yards.

[0025] In the field of air duct hook detection in railway marshalling yards, the current technical means mainly include the following: 1) Manual visual inspection Manual operators observe the status of the duct and hooks with their naked eyes to determine whether they meet the requirements. This method usually relies on experienced operators to conduct inspections. The advantages are high flexibility and the ability to handle various non-standard situations. The disadvantages are high labor intensity and easy to be affected by factors such as fatigue and lack of concentration, resulting in inconsistency and poor accuracy of inspection results.

[0026] 2) Single sensor detection a. Image recognition technology uses a camera to capture images of air ducts and hooks, and analyzes them through computer vision technology and image processing algorithms. Although it can provide detailed visual information, it lacks three-dimensional position information and is difficult to accurately determine the actual position of the air ducts and hooks. Since image recognition mainly relies on two-dimensional data, it is impossible to achieve accurate automatic removal of air ducts and hooks. In addition, image recognition technology is sensitive to external factors such as lighting changes and environmental impurities, which may lead to unstable and inaccurate detection results; b. Photoelectric sensors use photoelectric sensors to detect the presence and position of air ducts and hooks. Although the technology is mature and the response speed is fast, it is sensitive to changes in ambient light and is easily disturbed by dust and dirt, which affects the accuracy and stability of detection.

[0027] c. Ultrasonic sensor measures the distance between the air duct and the hook through ultrasonic signals. Its advantage is that it is not affected by light and is suitable for different lighting conditions. Its disadvantage is that it is sensitive to environmental noise and surface characteristics of objects, which may lead to measurement errors. It cannot provide accurate three-dimensional position information, which affects the implementation of complex detection tasks.

[0028] d. Laser distance sensor, which uses laser beam to measure the distance between the air duct and the hook. Although the positioning accuracy is high, it lacks texture detail information for target recognition.

[0029] Based on the above problems, the present invention provides a method for detecting a hook of an air duct. Figure 1 FIG. 1 is one of the flow charts of the air duct hook detection method provided by the present invention, such as Figure 1As shown, the method includes step 110 , step 120 , step 130 and step 140 .

[0030] Step 110, obtaining the point cloud data stream scanned by the laser radar and the image data stream taken by the camera, and performing image detection on the image data stream to obtain a two-dimensional image detection result.

[0031] Specifically, the point cloud data stream scanned by the laser radar and the image data stream captured by the camera can be obtained. The point cloud data stream here is the point cloud data required for subsequent target detection.

[0032] After obtaining the image data stream, image detection can be performed on the image data stream to obtain a two-dimensional image detection result. Here, the image detection can use an image detection model, which can be a YOLOv8 model or an SSD (Single Shot MultiBox Detector) model, etc., and the embodiment of the present invention does not specifically limit this.

[0033] Step 120 , jointly calibrate the point cloud data stream and the image data stream to obtain a joint calibration result.

[0034] Specifically, considering that different sensors are not synchronized in time and space, it is necessary to jointly calibrate the point cloud data stream and the image data stream to obtain a joint calibration result.

[0035] Here, joint calibration includes camera intrinsic calibration, sensor time synchronization, and sensor space synchronization. For example, first set the camera resolution to 1920x1080, then prepare a black and white checkerboard grid of an aluminum substrate with 12x9 squares, sample and calibrate the camera image based on the ubuntu20.04 ros environment, and obtain the camera intrinsic parameters including focal length, main point pixel coordinates, and distortion parameters; then perform time synchronization of the lidar and camera, synchronize the time of these two sensors to the ros timestamp of the computing box, and collect data at the same frequency; finally, perform spatial synchronization (i.e., external parameter calibration of the lidar and camera), and perform rough and fine calibration of the two sensors based on the black and white checkerboard calibration board used for the above internal parameter calibration, where the rough calibration is the calibration result obtained by the automated calibration algorithm, and the fine calibration is a manual calibration based on the rough calibration, so as to obtain the final external parameter, i.e., the mapping matrix from the lidar to the camera, and then obtain the joint calibration result.

[0036] Step 130 , filtering and segmenting the point cloud data stream to obtain a point cloud three-dimensional detection result, and converting the point cloud three-dimensional detection result into a projection result of a three-dimensional point cloud to a two-dimensional image based on the joint calibration result.

[0037] Specifically, the point cloud data stream is filtered and the target is segmented to obtain the point cloud 3D detection result. For example, the point cloud data can be subjected to ROI (Region Of Interest) filtering, NAN point filtering and downsampling filtering, and then the ground filtering and angle segmentation algorithm are performed through threshold setting to achieve target segmentation. Finally, the segmented target is filtered to obtain the final point cloud 3D detection result.

[0038] Furthermore, the point cloud 3D detection result can be converted into a projection result of the 3D point cloud to the 2D image based on the joint calibration result. For example, the points above and below the diagonal of the 3D box in the point cloud 3D detection result can be converted into a 2D box projection result on the image plane.

[0039] Step 140, determining a fusion result based on the degree of overlap between the detection frame of the two-dimensional image detection result and the detection frame of the projection result, and determining a target air duct hook category and a target spatial position of the target air duct hook category based on the fusion result.

[0040] Specifically, Figure 2 is a schematic diagram of the fusion result provided by the present invention, such as Figure 2 As shown, after obtaining the two-dimensional image detection result and the projection result, the fusion result can be determined based on the degree of overlap between the detection frame of the two-dimensional image detection result and the detection frame of the projection result.

[0041] That is, the IoU (Intersection over Union) calculation is performed on the detection frame of the two-dimensional image detection result and the detection frame of the projection result. When the overlap (IoU value) between the detection frame of the two-dimensional image detection result and the detection frame of the projection result is greater than the preset threshold, it is considered that the same target is perceived. The air duct hook category in the two-dimensional image detection result is used as the target air duct hook category, and the spatial position of the air duct hook category in the point cloud three-dimensional detection result is used as the target spatial position, that is, the fusion result includes the target air duct hook category and the target spatial position.

[0042] It should be noted that, on the one hand, the use of multi-sensor fusion can more accurately and efficiently obtain the target air duct hook type and target spatial position; on the other hand, the method provided by the embodiment of the present invention can replace manual observation, reduce operational risks, and significantly improve work efficiency.

[0043] The method provided by the embodiment of the present invention jointly calibrates the point cloud data stream and the image data stream to obtain a joint calibration result; filters and segments the point cloud data stream to obtain a point cloud three-dimensional detection result, and converts the point cloud three-dimensional detection result into a projection result of the three-dimensional point cloud to a two-dimensional image based on the joint calibration result; determines the fusion result based on the degree of overlap between the detection frame of the two-dimensional image detection result and the detection frame of the projection result, and determines the target air duct hook category and the target spatial position of the target air duct hook category based on the fusion result. On the one hand, the use of multi-sensor fusion can more accurately and efficiently obtain the target air duct hook category and the target spatial position; on the other hand, the method provided by the embodiment of the present invention can replace manual observation, reduce operational risks, and significantly improve work efficiency.

[0044] This method realizes the automated detection of air ducts and hooks in railway marshaling yards, and can provide accurate category identification and three-dimensional position information. This provides rich and accurate data for the automatic removal of air ducts and hooks, thereby significantly improving work efficiency while reducing safety risks.

[0045] Based on the above embodiment, the step 140 of determining the target air duct hook category and the target spatial position of the target air duct hook category based on the fusion result includes: Step 141, determining a spatial position offset based on a first spatial position of a target center in a current fusion result and a second spatial position of a target center in a previous fusion result; the previous fusion result is a fusion result of a preset number of times before the current fusion result; Step 142, when the spatial position offset is less than a first preset threshold, determine the target air duct hook category and the target spatial position in the current fusion result.

[0046] Specifically, the spatial position offset may be determined based on a first spatial position of the target center in the current fusion result and a second spatial position of the target center in the previous fusion result.

[0047] Here, the spatial position includes the coordinates of the center of the target and the length, width and height information of the target detection box.

[0048] Here, the previous fusion result is the fusion result of the previous preset number of times before the current fusion result. The previous preset number of times refers to the previous 4 times, or may be the previous 5 times, etc., and the embodiment of the present invention does not make any specific limitation on this.

[0049] Then, when the spatial position offset is less than the first preset threshold, the target air duct hook category and the target spatial position in the current fusion result are determined.

[0050] It can be understood that when the spatial position offset is less than the first preset threshold, the target duct hook category and the target spatial position in the current fusion result are determined. On the one hand, the tracking stability can be improved. By comparing the target positions in consecutive frames, the stable movement pattern of the target can be identified. If the position offset of the target is within the set threshold range, it means that the performance of the target in the consecutive frames is stable, which increases the reliability of the tracking results. On the other hand, it reduces false alarms and missed alarms: In some cases, single-frame detection may produce false alarms or missed alarms due to noise, lighting changes, occlusion or other interference factors. By comparing the detection results of consecutive frames, those unstable or abnormal detection results can be filtered out, thereby reducing the occurrence of false alarms and missed alarms.

[0051] Based on the above embodiment, the step 140 of determining the fusion result based on the overlap between the detection frame of the two-dimensional image detection result and the detection frame of the projection result includes: When the overlap degree is greater than a second preset threshold, the duct hook category in the two-dimensional image detection result and the spatial position of the duct hook category in the point cloud three-dimensional detection result are used as the fusion result.

[0052] Specifically, when the degree of overlap is greater than a second preset threshold, the duct hook category in the two-dimensional image detection result and the spatial position of the duct hook category in the point cloud three-dimensional detection result are used as the fusion result.

[0053] In the method provided by the embodiment of the present invention, the camera and the laser radar each have different detection characteristics and advantages. The camera can provide rich color, texture and shape information, while the laser radar can provide accurate depth information and three-dimensional spatial structure. By fusing the data of these two sensors and using IoU as the judgment criterion, the accuracy of target detection can be significantly improved and false positives and false negatives can be reduced.

[0054] Based on the above embodiment, the step 110 of performing image detection on the image data stream to obtain a two-dimensional image detection result includes: Step 111, using an image detection model to perform image detection on the image data stream to obtain the two-dimensional image detection result; The training steps of the image detection model include: Acquire sample image data and an initial image detection model, and annotate the image data of the air duct and the retractor in the sample image data to obtain image labels; the sample image data includes image data of the air duct and the retractor in each scene; Inputting the sample image data into the initial image detection model to obtain the image detection prediction result output by the initial image detection model; Based on the difference between the image detection prediction result and the image label, a target loss is determined, and based on the target loss, parameters of the initial image detection model are iterated to obtain the image detection model.

[0055] Specifically, an image detection model may be used to perform image detection on an image data stream to obtain a two-dimensional image detection result.

[0056] The training steps of the image detection model are as follows: First, sample image data and an initial image detection model are obtained, and image data of the airway tube and the retractor in the sample image data are annotated to obtain image labels.

[0057] The sample image data includes image data of air ducts and hooks in various scenarios. For example, the target images and point cloud topic data in different outdoor environments such as sunny days, rainy days, daytime and night are collected through the ROS system and recorded in bag format. After data analysis, image data sets and point cloud data in different environments are obtained, thereby completing the collection of sample image data.

[0058] Here, the parameters of the initial image detection model may be pre-set or randomly generated, and the embodiment of the present invention does not specifically limit this.

[0059] Then, the sample image data can be input into the initial image detection model to obtain the image detection prediction result output by the initial image detection model; Finally, the target loss can be determined based on the difference between the image detection prediction result and the image label, and the parameters of the initial image detection model can be iterated based on the target loss. The initial image detection model after the parameter iteration is used as the image detection model.

[0060] It can be understood that the greater the difference between the image detection prediction result and the image label, the greater the target loss; the smaller the difference between the image detection prediction result and the image label, the smaller the target loss.

[0061] Based on the above embodiment, the step 130 of filtering and segmenting the point cloud data stream to obtain a point cloud three-dimensional detection result includes: Step 131, performing a preprocessing operation on the point cloud data stream to obtain preprocessed point cloud data; the preprocessing operation includes at least one of ROI filtering, NAN point filtering and downsampling filtering; Step 132, filtering out the ground point cloud data in the pre-processed point cloud data to obtain candidate point cloud data; Step 133, performing point cloud angle segmentation on the candidate point cloud data to obtain a point cloud target segmentation result; Step 134, filter the point cloud target segmentation result to obtain the point cloud three-dimensional detection result.

[0062] Specifically, a preprocessing operation is performed on the point cloud data stream to obtain preprocessed point cloud data, wherein the preprocessing operation includes at least one of ROI filtering, NAN point filtering and downsampling filtering.

[0063] Here, ROI filtering is used to extract the region of interest in the 3D point cloud data stream to highlight or enhance the information in the region, while weakening or removing the information in other non-ROI regions. This filtering method is often used in the field of point cloud processing, which can help people pay more attention to and analyze the key parts of the point cloud data.

[0064] NaN point filtering refers to identifying and removing NaN values ​​in a data set. NaN values ​​may be generated due to errors in data collection, transmission, or processing, and they can negatively affect the results of data analysis.

[0065] Downsampling filtering is a multi-rate digital signal processing technique used to reduce the sampling rate of a signal. During the downsampling process, filters are used to reduce the aliasing distortion caused by the reduction in sampling rate. Aliasing refers to the high-frequency signal being mixed into the low-frequency signal during the sampling process, causing signal distortion. To avoid this, a low-pass filter is usually used to remove the high-frequency components in the signal before downsampling.

[0066] Considering that ground point clouds often contain a lot of redundant information, if it is not filtered out, the complexity and computational complexity of target detection will increase. Therefore, the ground point cloud data in the preprocessed point cloud data can be filtered out to obtain candidate point cloud data. Here, the ground point cloud can be filtered out by setting a threshold.

[0067] Furthermore, the candidate point cloud data is segmented by point cloud angle to obtain a point cloud target segmentation result.

[0068] It can be understood that point cloud angle segmentation can be performed based on the angle characteristics of the object, thereby extracting the unique characteristics of the object. This helps to enhance the ability of object recognition, making it possible to more accurately identify different objects and scenes.

[0069] Finally, the point cloud target segmentation result is filtered to obtain the point cloud 3D detection result.

[0070] It should be noted that point cloud data may be affected by various factors during the collection and processing, such as illumination, occlusion, sensor error, etc., resulting in a decrease in data quality. Filter filtering can suppress and compensate for these influencing factors to a certain extent, thereby optimizing the quality of point cloud data and improving the reliability and stability of 3D detection.

[0071] Based on any of the above embodiments, Figure 3 FIG. 2 is a flow chart of the air duct hook detection method provided by the present invention. Figure 3 As shown, the method includes: First, the laser radar and camera are calibrated together. Secondly, a data set is constructed by collecting image and point cloud data. Then, image and point cloud processing algorithms are designed. After algorithm processing, the detection results of each sensor are obtained respectively. Finally, the detection results of the above two sensor algorithms are fused by algorithm processing to achieve the final target recognition and positioning. The steps include the following: Step S1: Joint calibration of lidar and camera; Step S2: Dataset construction; Step S3: image and point cloud algorithm processing; Step S4: sensor fusion algorithm processing; Specifically, first, the laser point cloud data stream and the camera image data stream are obtained, and then the laser point cloud data stream and the camera image data stream are jointly calibrated to obtain synchronized data and a mapping matrix, that is, a joint calibration result.

[0072] Then, the target images and point cloud topic data in different outdoor environments such as sunny days, rainy days, daytime and night are collected through the ROS system and recorded in bag format. After data analysis, image data sets and point cloud data in different environments are obtained, thus completing the construction of the data set.

[0073] Regarding the image processing part, we first use the annotation tool on the image dataset to annotate the target boxes of the air duct and the hook, and then complete the image data annotation. Then we use the YOLOv8 model to train the annotated data. Finally, through model testing, we get the optimal detection model parameters, and apply the image detection model to perform model inference and prediction to get the two-dimensional image detection results.

[0074] Furthermore, the laser point cloud data stream is subjected to point cloud preprocessing, point cloud ground filtering, point cloud angle segmentation and filter filtering to obtain the point cloud three-dimensional detection result.

[0075] Based on the joint calibration results, the point cloud 3D detection results are converted into projection results of 3D point cloud to 2D image.

[0076] Finally, the IoU is calculated based on the two-dimensional image detection results and projection results, and the spatial position offset of the target center in 5 consecutive frames is compared.

[0077] When the spatial position offset is less than the first preset threshold, the target air duct hook category and the target spatial position (three-dimensional position information) in the current fusion result are fused and output stably.

[0078] The air duct hook detection device provided by the present invention is described below. The air duct hook detection device described below and the air duct hook detection method described above can be referenced to each other.

[0079] Based on any of the above embodiments, the present invention provides a duct hook detection device, Figure 4 Schematic diagram of the structure of the air duct hook detection device provided by the present invention. Figure 4 As shown, the device comprises: The acquisition unit 410 is used to acquire the point cloud data stream scanned by the laser radar and the image data stream taken by the camera, and perform image detection on the image data stream to obtain a two-dimensional image detection result; A joint calibration unit 420, configured to perform joint calibration on the point cloud data stream and the image data stream to obtain a joint calibration result; A conversion unit 430 is used to filter and segment the point cloud data stream to obtain a point cloud three-dimensional detection result, and convert the point cloud three-dimensional detection result into a projection result based on the joint calibration result; The determination unit 440 is used to determine a fusion result based on the degree of overlap between the detection frame of the two-dimensional image detection result and the detection frame of the projection result, and determine the target air duct hook category and the target spatial position of the target air duct hook category based on the fusion result.

[0080] The device provided by the embodiment of the present invention jointly calibrates the point cloud data stream and the image data stream to obtain a joint calibration result; filters and segments the point cloud data stream to obtain a point cloud three-dimensional detection result, and converts the point cloud three-dimensional detection result into a projection result of the three-dimensional point cloud to the two-dimensional image based on the joint calibration result; determines the fusion result based on the degree of overlap between the detection frame of the two-dimensional image detection result and the detection frame of the projection result, and determines the target air duct hook category and the target spatial position of the target air duct hook category based on the fusion result. On the one hand, the use of multi-sensor fusion can more accurately and efficiently obtain the target air duct hook category and the target spatial position; on the other hand, it can replace manual observation, reduce operational risks, and significantly improve work efficiency.

[0081] Based on any of the foregoing embodiments, the determining unit 440 is specifically configured to: Determine a spatial position offset based on a first spatial position of a target center in a current fusion result and a second spatial position of a target center in a previous fusion result; the previous fusion result is a fusion result of a preset number of times before the current fusion result; When the spatial position offset is less than a first preset threshold, the target air duct hook category and the target spatial position in the current fusion result are determined.

[0082] Based on any of the foregoing embodiments, the determining unit 440 is specifically configured to: When the overlap degree is greater than a second preset threshold, the duct hook category in the two-dimensional image detection result and the spatial position of the duct hook category in the point cloud three-dimensional detection result are used as the fusion result.

[0083] Based on any of the above embodiments, the acquisition unit 410 is specifically configured to: Using an image detection model, performing image detection on the image data stream to obtain the two-dimensional image detection result; The training steps of the image detection model include: Acquire sample image data and an initial image detection model, and annotate the image data of the air duct and the retractor in the sample image data to obtain image labels; the sample image data includes image data of the air duct and the retractor in each scene; Inputting the sample image data into the initial image detection model to obtain the image detection prediction result output by the initial image detection model; Based on the difference between the image detection prediction result and the image label, a target loss is determined, and based on the target loss, parameters of the initial image detection model are iterated to obtain the image detection model.

[0084] Based on any of the above embodiments, the joint calibration includes camera intrinsic parameter calibration, sensor time synchronization, and sensor space synchronization.

[0085] Based on any of the above embodiments, the sensor spatial synchronization includes coarse calibration and fine calibration; The rough calibration is performed based on an automated calibration algorithm; The fine calibration is a manual calibration based on the rough calibration.

[0086] Based on any of the above embodiments, the conversion unit 430 is specifically configured to: Performing a preprocessing operation on the point cloud data stream to obtain preprocessed point cloud data; the preprocessing operation includes at least one of ROI filtering, NAN point filtering and downsampling filtering; Filtering the ground point cloud data in the pre-processed point cloud data to obtain candidate point cloud data; Performing point cloud angle segmentation on the candidate point cloud data to obtain a point cloud target segmentation result; The point cloud target segmentation result is filtered to obtain the point cloud three-dimensional detection result.

[0087] Figure 5is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute the air duct hook detection method, which includes: obtaining a point cloud data stream scanned by a laser radar and an image data stream captured by a camera, and performing image detection on the image data stream to obtain a two-dimensional image detection result; jointly calibrating the point cloud data stream and the image data stream to obtain a joint calibration result; filtering and target segmenting the point cloud data stream to obtain a point cloud three-dimensional detection result, and converting the point cloud three-dimensional detection result into a projection result of a three-dimensional point cloud to a two-dimensional image based on the joint calibration result; determining a fusion result based on the degree of overlap between the detection frame of the two-dimensional image detection result and the detection frame of the projection result, and determining the target air duct hook category and the target spatial position of the target air duct hook category based on the fusion result.

[0088] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0089] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the air duct hook detection method provided by the above-mentioned methods, and the method includes: obtaining a point cloud data stream scanned by a laser radar and an image data stream taken by a camera, and performing image detection on the image data stream to obtain a two-dimensional image detection result; jointly calibrating the point cloud data stream and the image data stream to obtain a joint calibration result; filtering and target segmenting the point cloud data stream to obtain a three-dimensional point cloud detection result, and converting the three-dimensional point cloud detection result into a projection result of a three-dimensional point cloud to a two-dimensional image based on the joint calibration result; determining a fusion result based on the degree of overlap between the detection frame of the two-dimensional image detection result and the detection frame of the projection result, and determining the target air duct hook category and the target spatial position of the target air duct hook category based on the fusion result.

[0090] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the air duct hook detection method provided by the above-mentioned methods, the method comprising: obtaining a point cloud data stream scanned by a laser radar and an image data stream taken by a camera, and performing image detection on the image data stream to obtain a two-dimensional image detection result; performing joint calibration on the point cloud data stream and the image data stream to obtain a joint calibration result; performing filtering and target segmentation on the point cloud data stream to obtain a three-dimensional point cloud detection result, and converting the three-dimensional point cloud detection result into a projection result of a three-dimensional point cloud to a two-dimensional image based on the joint calibration result; determining a fusion result based on the degree of overlap between the detection frame of the two-dimensional image detection result and the detection frame of the projection result, and determining a target air duct hook category and a target spatial position of the target air duct hook category based on the fusion result.

[0091] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting a duct hook, characterized in that: include: Obtaining a point cloud data stream scanned by a laser radar and an image data stream captured by a camera, and performing image detection on the image data stream to obtain a two-dimensional image detection result; Performing joint calibration on the point cloud data stream and the image data stream to obtain a joint calibration result; Filtering and segmenting the point cloud data stream to obtain a point cloud three-dimensional detection result, and converting the point cloud three-dimensional detection result into a projection result of a three-dimensional point cloud to a two-dimensional image based on the joint calibration result; A fusion result is determined based on the degree of overlap between the detection frame of the two-dimensional image detection result and the detection frame of the projection result, and a target air duct hook category and a target spatial position of the target air duct hook category are determined based on the fusion result.

2. The air duct hook detection method according to claim 1, characterized in that: The determining of the target air duct hook category and the target spatial position of the target air duct hook category based on the fusion result includes: Determine a spatial position offset based on a first spatial position of a target center in a current fusion result and a second spatial position of a target center in a previous fusion result; the previous fusion result is a fusion result of a preset number of times before the current fusion result; When the spatial position offset is less than a first preset threshold, the target air duct hook category and the target spatial position in the current fusion result are determined.

3. The air duct hook detection method according to claim 1, characterized in that: The determining of the fusion result based on the overlap degree between the detection frame of the two-dimensional image detection result and the projection result includes: When the overlap degree is greater than a second preset threshold, the duct hook category in the two-dimensional image detection result and the spatial position of the duct hook category in the point cloud three-dimensional detection result are used as the fusion result.

4. The air duct hook detection method according to claim 1, characterized in that: The performing image detection on the image data stream to obtain a two-dimensional image detection result includes: Using an image detection model, performing image detection on the image data stream to obtain the two-dimensional image detection result; The training steps of the image detection model include: Acquire sample image data and an initial image detection model, and annotate the image data of the air duct and the retractor in the sample image data to obtain image labels; the sample image data includes image data of the air duct and the retractor in each scene; Inputting the sample image data into the initial image detection model to obtain the image detection prediction result output by the initial image detection model; Based on the difference between the image detection prediction result and the image label, a target loss is determined, and based on the target loss, parameters of the initial image detection model are iterated to obtain the image detection model.

5. The method for detecting air duct hooks according to any one of claims 1 to 4, characterized in that: The joint calibration includes camera intrinsic parameter calibration, sensor time synchronization, and sensor space synchronization.

6. The air duct hook detection method according to claim 5, characterized in that: The sensor spatial synchronization includes coarse calibration and fine calibration; The rough calibration is performed based on an automated calibration algorithm; The fine calibration is a manual calibration based on the rough calibration.

7. The method for detecting air duct hooks according to any one of claims 1 to 4, characterized in that: The filtering and target segmentation of the point cloud data stream to obtain a point cloud three-dimensional detection result includes: Performing a preprocessing operation on the point cloud data stream to obtain preprocessed point cloud data; the preprocessing operation includes at least one of ROI filtering, NAN point filtering and downsampling filtering; Filtering the ground point cloud data in the pre-processed point cloud data to obtain candidate point cloud data; Performing point cloud angle segmentation on the candidate point cloud data to obtain a point cloud target segmentation result; The point cloud target segmentation result is filtered to obtain the point cloud three-dimensional detection result.

8. A duct hook detection device, characterized in that: include: An acquisition unit is used to acquire a point cloud data stream scanned by a laser radar and an image data stream taken by a camera, and perform image detection on the image data stream to obtain a two-dimensional image detection result; A joint calibration unit, used for jointly calibrating the point cloud data stream and the image data stream to obtain a joint calibration result; A conversion unit, configured to filter and segment the point cloud data stream to obtain a point cloud three-dimensional detection result, and convert the point cloud three-dimensional detection result into a projection result based on the joint calibration result; A determination unit is used to determine a fusion result based on the degree of overlap between the detection frame of the two-dimensional image detection result and the detection frame of the projection result, and to determine a target air duct hook category and a target spatial position of the target air duct hook category based on the fusion result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the air duct hook detection method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the air duct hook detection method according to any one of claims 1 to 7 is implemented.

Images