Methods, apparatus, systems, devices, and media for target detection and search

Abstract

The application discloses a method, device, system, equipment and medium for target detection and searching, and relates to the technical field of target autonomous searching and positioning. The target detection method comprises the following steps: acquiring an infrared image and a visible light image of a detection target; and inputting the infrared image and the visible light image into a pre-trained Twin-YOLOv8 network model to output a target detection result. The Twin-YOLOv8 network model is an improved YOLOv8 network model with a double-branch detection head, and the double-branch detection head comprises a main branch for predicting a target bounding box and a class confidence and a shielding perception branch for predicting a shielding rate. The application designs the Twin-YOLOv8 network model with the double-branch detection head, so that the target detection has better scale robustness and shielding adaptability, and avoids missing detection caused by incomplete target features in a complex scene.

Description

Technical Field

[0001] This application relates to the field of autonomous target search and positioning technology, specifically to a method, apparatus, system, device, and medium for target detection and search. Background Technology

[0002] Target detection, as the core task of autonomous target search, plays an important role in scenarios such as modern high-altitude rescue, while environmental perception is an important part of target detection.

[0003] For example, fire trucks and aerial rescue drones are typically equipped with visible light or infrared cameras for status monitoring and remote alarms to enhance their sensing capabilities. However, these solutions are often limited to a single sensor, making them ill-suited for complex and ever-changing rescue scenarios. They also exhibit poor environmental adaptability and are highly susceptible to adverse weather conditions (such as rain, fog, and complex lighting). For instance, visible light cameras fail in smoke, while infrared thermal imaging suffers from reduced signal-to-noise ratios under high-temperature backgrounds or strong sunlight interference.

[0004] To address this, existing technologies have proposed target detection algorithms based on the fusion of infrared and visible light images, such as the YOLO series. While these YOLO algorithms improve target detection accuracy, their traditional single detection heads suffer from insufficient scale robustness and occlusion adaptability when facing complex high-altitude rescue scenes. Consequently, in special scenarios such as rain, fog, smoke, and occlusion (rescued personnel being partially obscured by buildings or other objects), they are prone to missed detections due to incomplete target features. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, system, device, and medium for target detection and search, so as to at least partially solve the above-mentioned technical problems.

[0006] To achieve the above objectives, a first aspect of this application provides a dual-light fusion target detection method, comprising: acquiring an infrared image and a visible light image of the target; and inputting the infrared image and the visible light image into a pre-trained Twin-YOLOv8 network model to output a target detection result. The Twin-YOLOv8 network model is an improved YOLOv8 network model with a dual-branch detection head, and the dual-branch detection head includes: a main branch for predicting the target bounding box and class confidence; and an occlusion perception branch for predicting the occlusion rate.

[0007] In this embodiment, the Twin-YOLOv8 network model includes: a backbone network for extracting features from the infrared image and the visible light image to obtain corresponding infrared features and visible light features; an attention enhancement module for performing attention-weighted feature fusion on the infrared features and the visible light features to obtain corresponding fused features; a neck network for performing multi-scale feature enhancement on the fused features to obtain corresponding enhanced features; and a dual-branch detection head, wherein the main branch and the occlusion perception branch are configured for parallel inference to output the target detection result including the target bounding box, the class confidence, and the occlusion rate based on the enhanced features.

[0008] In this embodiment of the application, the Twin-YOLOv8 network model further includes the following modules disposed before the backbone network: an input module for receiving the infrared image and the visible light image; and a preprocessing module for performing image preprocessing on the infrared image and the visible light image received by the input module, and providing the infrared image and the visible light image after image preprocessing to the backbone network.

[0009] In this embodiment, the neck network adopts a Feature Pyramid Network (FPN)-Path Aggregation Network (PAN) structure, and cross-scale attention gating is added to the FPN to enhance the features of small targets, and scale-adaptive convolution is configured in the PAN to enhance the feature details of targets at the corresponding scale.

[0010] In this embodiment, the main branch uses the CIoU loss function to optimize the bounding box regression loss, and / or the occlusion perception branch uses several convolutional layers and labeled occlusion level tags to identify the occlusion rate.

[0011] In this embodiment of the application, the dual-branch detection head is further configured to: correct the category confidence score by combining the occlusion rate before outputting the category confidence score.

[0012] In this embodiment of the application, the target detection method further includes: optimizing and training the Twin-YOLOv8 network model based on an environmental dataset associated with complex scenes and / or a target image dataset associated with various target occlusion scenes.

[0013] A second aspect of this application provides a target search method, comprising: acquiring an infrared image, a visible light image, and point cloud data of a target; processing the infrared image and the visible light image using any of the aforementioned target detection methods to output image data characterizing the target detection result; performing target localization based on the point cloud data and the image data to obtain corresponding target location information; and determining the optimal location point associated with the target search task according to the target location information and a preset search rule.

[0014] A third aspect of this application provides a dual-light fusion target detection apparatus, comprising: a memory configured to store instructions; and a processor configured to retrieve the instructions from the memory and, when executing the instructions, to implement any of the target detection methods described above.

[0015] A fourth aspect of this application provides a target search system, comprising: a perception module for acquiring infrared images, visible light images, and point cloud data of a target; any of the aforementioned target detection devices for acquiring the infrared images and visible light images of the target from the perception module, and outputting image data characterizing the target detection result based on the infrared images and visible light images; a target localization device for performing target localization based on the point cloud data and the image data to obtain corresponding target location information; and a location determination module for determining the optimal location point associated with the target search task according to the target location information and preset search rules.

[0016] The fifth aspect of this application provides a working machine that includes any of the target search systems described above.

[0017] A sixth aspect of this application provides a machine-readable storage medium storing instructions that cause a machine to perform any of the target detection methods or target search methods described above.

[0018] Through the above technical solutions, this application proposes a dual-light fusion target detection method, and specifically designs a Twin-YOLOv8 network model with a dual-branch detection head. Compared with a single detection head, the dual-branch detection head has better scale robustness and occlusion adaptability when facing complex scenes, avoiding missed detections caused by incomplete target features in complex scenes.

[0019] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0020] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings: Figure 1 The schematic diagram illustrates a flow chart of a dual-light fusion target detection method according to an embodiment of this application; Figure 2 A schematic diagram illustrating the structure of a Twin-YOLOv8 network model according to an embodiment of this application is shown. Figure 3 The illustration shows a flowchart of a target search method according to an embodiment of this application; Figure 4(a) schematically illustrates a flowchart of a method for calibrating the extrinsic parameters of a lidar and camera according to an embodiment of this application; Figure 4(b) schematically illustrates the process of effective matching pair screening in the external parameter calibration method of Figure 4(a); Figure 5(a) schematically illustrates a flowchart of a target localization method according to an embodiment of this application; Figure 5(b) schematically illustrates the point cloud data downsampling process performed in the target localization method of Figure 5(a); Figure 6 A schematic diagram of the structure of a dual-light fusion target detection device according to an embodiment of this application is shown; and Figure 7 The schematic diagram illustrates the structure of a target search system according to an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0022] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with relevant laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0023] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0024] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0025] Example 1 Figure 1 The illustration schematically shows a flowchart of a dual-light fusion target detection method according to an embodiment of this application. Figure 1 As shown in the figure, this application provides a dual-light fusion target detection method, which may include the following steps S110-S120.

[0026] Step S110: Acquire infrared and visible light images of the target.

[0027] For example, visible light images (i.e., RGB images) and infrared images are acquired simultaneously using calibrated visible light and infrared cameras.

[0028] In a preferred embodiment, the image is further preprocessed, including distortion correction, image alignment (registration), and size normalization, to obtain pixel-level aligned image pairs.

[0029] Step S120: Input the infrared image and the visible light image into the pre-trained Twin-YOLOv8 network model to output the target detection result.

[0030] The Twin-YOLOv8 network model is an improved YOLOv8 network model with a dual-branch detection head, and the dual-branch detection head includes: a main branch for predicting the target bounding box and class confidence; and an occlusion perception branch for predicting the occlusion rate.

[0031] In a preferred embodiment, such as Figure 2As shown, the Twin-YOLOv8 network model includes: a backbone network 201, used to extract features from the infrared image and the visible light image to obtain corresponding infrared features and visible light features; an attention enhancement module 202, used to perform attention weighted feature fusion on the infrared features and the visible light features to obtain corresponding fused features; a neck network 203, used to perform multi-scale feature enhancement on the fused features to obtain corresponding enhanced features; and a dual-branch detection head 204, wherein the main branch and the occlusion perception branch are configured for parallel inference to output the target detection result including the target bounding box, the class confidence, and the occlusion rate based on the enhanced features.

[0032] In a more preferred embodiment, the Twin-YOLOv8 network model further includes the following modules disposed before the backbone network 201: an input module 205 for receiving the infrared image and the visible light image; and a preprocessing module 206 for performing image preprocessing on the infrared image and the visible light image received by the input module 205, and providing the infrared image and the visible light image after image preprocessing to the backbone network.

[0033] The following examples will provide a detailed introduction to each module of the Twin-YOLOv8 network model.

[0034] 1. Input module 205.

[0035] The input module 205 is configured to receive the infrared image and the visible light image acquired and preprocessed in step S110 above.

[0036] 2. Preprocessing module 206.

[0037] The image preprocessing performed by the preprocessing module 206 includes, but is not limited to, size normalization, illumination normalization, color conversion, and data normalization.

[0038] 3. Backbone network 201 and attention enhancement module 202.

[0039] In this example, a dual-branch shared CSPDarkNet-53 backbone network is constructed, where CSP stands for Cross Stage Partial Connections. CSPDarkNet-53 is an improvement upon DarkNet-53 (the backbone of YOLOv3), and its CSP modules help balance the network's expressive power and computational efficiency. Sharing the CSPDarkNet-53 backbone network between the two branches achieves the diversity benefits of a dual-branch architecture while controlling computational costs.

[0040] Furthermore, the preprocessed visible light images are input into the backbone network. and infrared images To extract texture features and temperature characteristics .

[0041] In the shallow layers of the backbone network (after the first C2f module), the extracted feature maps are input into the attention enhancement module 202. This attention enhancement module 202 is configured to achieve weighted image fusion through attention-guided feature enhancement.

[0042] Specifically, firstly, texture features are calculated. Temperature characteristics correlation matrix Then through Calculate the attention weights It adaptively evaluates the importance of each channel of visible light and infrared features in the current scene and performs weighted summation to achieve information complementarity (such as in smoke, rain and fog scenes, the algorithm will automatically increase the weight of infrared features).

[0043] The formula for the weighted fusion dual-modal feature is as follows:

[0044]

[0045] 4. Neck network 203.

[0046] The neck network employs a Feature Pyramid Network (FPN) - Path Aggregation Network (PAN) structure. Cross-scale attention gating is added to the FPN to enhance small target features, while scale-adaptive convolutions are configured in the PAN to enhance feature details of targets at corresponding scales. Specifically, the FPN helps enhance the semantic information of shallow features, while the PAN helps enhance the localization capability of deep features.

[0047] Specifically, features from various scales are fused. Input an improved FPN+PAN structure.

[0048] First, the newly added cross-scale attention gating is configured as follows: in the top-down path of FPN, high-resolution features (8×8, corresponding to people waiting for rescue nearby in high-altitude rescue scenarios) are compressed through a 1×1 convolution channel and then attention weights are calculated with low-resolution features (32×32, corresponding to people waiting for rescue in the distance) to enhance the transmission of small target features.

[0049] Thus, cross-scale attention gating is equivalent to adding an intelligent feature filter to FPN, which enhances the transmission of small target features. In small target detection tasks such as high-altitude rescue, cross-scale attention gating ensures that those weak but potentially life-saving small target features are not lost or diluted during the transmission to deeper layers, but are instead protected and enhanced in a targeted manner.

[0050] Secondly, in the bottom-up path of PAN, scale-adaptive convolution is introduced (the kernel size is dynamically adjusted according to the scale: 5×5 kernel for 8×8 features, and 3×3 kernel for 32×32 features) to adapt to the feature extraction needs of targets at different scales. Finally, the enhanced multi-scale fused features are output. This improves the feature differentiation between near- and far-range targets.

[0051] In this way, scale-adaptive convolution uses different convolution kernels for feature maps of different scales, allowing the network to obtain the optimal feature representation at different distances (near, medium, and far), thus enabling accurate detection regardless of the size of the target in complex multi-scale rescue scenarios.

[0052] 5. Dual-branch detection head 204.

[0053] In a preferred embodiment, the main branch uses the CIoU loss function to optimize the bounding box regression loss, and / or the occlusion perception branch uses several convolutional layers and labeled occlusion level tags to identify the occlusion rate.

[0054] In this example, a two-branch parallel inference is added to the YOLOv8 detection head. Main branch: Predicts the target bounding box. and category confidence (Personnel and non-personnel), CIoU loss is used to optimize bounding box regression; Occlusion perception branch: occlusion rate is predicted through 3 convolutional layers (output channels 64→32→1). O During training, the "occlusion level" labels (0-3 levels) labeled in the dataset are used for supervision by MSE loss to achieve quantitative identification of "partial occlusion (30%-50%)" and "severe occlusion (50%-70%)".

[0055] In a preferred embodiment, the dual-branch detection head is further configured to correct the category confidence score by incorporating the occlusion rate before outputting the category confidence score. For example, the category confidence score is dynamically corrected by incorporating the occlusion rate as shown in the following formula to address the problem of low confidence scores caused by occlusion:

[0056] in, The preset occlusion impact coefficient, >0. For general occlusion, >0; For severe occlusion >η, where η is the set severe occlusion threshold, such as 0.7. Through this dynamic correction, the confidence weight is appropriately increased to avoid missed detections.

[0057] Back Figure 2 Through the cooperation of the above modules, especially the improved dual-branch detection head, the final output is the target bounding box of the personnel target. Corrected confidence level and occlusion rate This provides target status information for the subsequent target location steps.

[0058] Furthermore, the Twin-YOLOv8 network model described above is obtained through neural network training based on sample images. During model training, if heavy data augmentation with simulated occlusion (randomly adding occlusion masks and cut mixes) is used, the ability to recognize partially occluded targets can be improved.

[0059] Therefore, the target detection method in this application embodiment may further include: optimizing and training the Twin-YOLOv8 network model based on an environmental dataset associated with a complex scene and / or a target image dataset associated with various target occlusion scenes.

[0060] Complex scenes include, but are not limited to, rain and fog scenes, smoke scenes, nighttime scenes, and strong light scenes. Using environmental datasets containing synthetic rain, fog, and smoke, and applying weather style transfer techniques to augment the training data, can improve the model's generalization performance under adverse weather conditions.

[0061] Similarly, using target image datasets associated with various target occlusion scenarios to augment the training data can improve the model's generalization performance in multiple occlusion scenarios.

[0062] Furthermore, when the Twin-YOLOv8 network model outputs the target bounding box, category, and confidence score, a data optimization module can be set, such as performing non-maximum suppression. Additionally, a temporary ID can be assigned to each detected target for the final detection result.

[0063] Through the above steps S110-S120, taking a high-altitude rescue scenario as an example, in order to achieve personnel target recognition in special scenarios such as rain, fog, smoke, and obstruction, this application embodiment combines dual-light cameras (visible light camera and infrared camera) to extract image features from infrared images and visible light images respectively, and inputs them into the constructed Twin-YOLOv8 network model for detection, ultimately achieving personnel target recognition.

[0064] In summary, this application proposes a dual-light fusion target detection method, which specifically designs a Twin-YOLOv8 network model with a dual-branch detection head. Compared with a single detection head, the dual-branch detection head has better scale robustness and occlusion adaptability in complex scenes, avoiding missed detections caused by incomplete target features in complex scenes.

[0065] More specifically, the improved Twin-Yolov8 network proposed in this application has advantages over other YOLO algorithms (such as YOLOv5, YOLOv7, and traditional YOLOv8) in the following three aspects: First, a dual-branch parallel detection structure is adopted to specifically extract detailed features from visible light images and temperature features from infrared images. This solves the problem of low recognition accuracy and high false negative and false positive rates of single-modal images in complex environments (such as dense smoke, fog, and darkness at night). The accuracy of identifying trapped personnel is improved by more than 35% compared with the traditional YOLO algorithm.

[0066] Secondly, the feature fusion module has been optimized and an attention mechanism has been introduced to focus on key features such as human body contours and limb movements, effectively filtering out environmental interference (such as building components and debris), and improving the recognition speed by 20%, which can meet the real-time requirements in rescue scenarios.

[0067] Third, a small target enhancement detection unit has been added. In response to the characteristics of people trapped at high altitudes being far away and having small imaging size, the false negative rate of small targets has been greatly reduced. Compared with the traditional YOLO algorithm, the accuracy of small target recognition has been improved by 40%, which can quickly capture trapped people at the edge of high altitudes and in narrow spaces, thus buying precious time for rescue.

[0068] Example 2 High-altitude rescue, especially in scenarios such as high-rise building fires, earthquakes, and mountain emergencies, is a crucial link in ensuring people's safety. Currently, the rescue equipment used in this field mainly includes fire ladder trucks and high-altitude rescue drones. While these devices have improved the height and scope of rescue operations to some extent, their level of intelligence remains significantly insufficient, hindering further improvements in rescue efficiency and safety.

[0069] Traditional aerial work platform rescue operations heavily rely on operator visual observation and manual control. Operators on the ground determine the location of trapped individuals visually or through simple video surveillance, and then, based on experience, maneuver the multi-section boom to bring the rescue platform (such as a safety bucket) closer to the target. This method has several drawbacks: First, under complex visual conditions such as at night, in smoke, or in strong light, operators find it difficult to accurately locate trapped individuals and cannot perceive their detailed condition. Second, the process of manually maneuvering the boom to approach the target is time-consuming, demanding extremely high operator skills and psychological fortitude. In emergencies, misjudgment can easily lead to collisions between the boom and building structures, causing secondary risks. The target detection method proposed in Example 1 can address the first drawback, but the second drawback arises during the target localization process after target detection.

[0070] In response, Embodiment 2 of this application provides a target search method based on Embodiment 1. Taking a high-altitude rescue scenario as an example, it further performs target positioning and determines the optimal rescue location point where the end of the aerial work platform boom (safety bucket) stops.

[0071] like Figure 3 As shown, the target search method in Embodiment 2 of this application may include the following steps S310-S340.

[0072] Step S310: Acquire infrared image, visible light image and point cloud data of the target.

[0073] For example, a sensing module can be installed on the operating machinery. The sensing module consists of a dual-light camera and a lidar. The dual-light camera is used to simultaneously acquire visible light images and infrared thermal images of the target area; the lidar is used to acquire real-time 3D point cloud data of the target area.

[0074] Step S320: The infrared image and the visible light image are processed using the target detection method of Embodiment 1 to output image data characterizing the target detection result.

[0075] For example, firstly, a Twin-YOLOv8 network with a shared backbone is constructed, with the structure as follows: Figure 2 As shown. Secondly, by enhancing and improving the detection head through attention-guided feature extraction, the target bounding box of the person target is output. Confidence level and occlusion rate For specific implementation details, please refer to Embodiment 1 above, which will not be repeated here.

[0076] Step S330: Target localization is performed based on the point cloud data and the image data to obtain the corresponding target location information.

[0077] For example, based on the camera-LiDAR extrinsic calibration results, the point cloud ROI is extracted by guiding the visual target bounding box, and combined with adaptive DBSCAN clustering and dynamic error compensation, the three-dimensional spatial coordinates of the personnel target are output.

[0078] Step S340: Determine the optimal location point associated with the target search task based on the target location information and preset search rules.

[0079] Among these, preset search rules can serve as constraints for the automatic obstacle avoidance strategy. For example, in a high-altitude rescue scenario, based on the precise three-dimensional coordinates of the trapped person, and according to preset safety rules (such as maintaining a safe distance from building facades and window frames), an optimal rescue location for the aerial ladder truck's safety bucket to stop is automatically calculated and confirmed in three-dimensional space.

[0080] In summary, the target search method of Embodiment 2 of this application can be applied to high-altitude rescue scenarios. By integrating dual-light images, point cloud data, and artificial intelligence algorithms (Twin-YOLOv8 network), it can achieve automatic identification of people trapped at high altitudes, centimeter-level precise positioning, and autonomous confirmation of rescue location.

[0081] Existing target search methods often only feed image information back to the operator, which is at the "auxiliary observation" level and fails to achieve an intelligent decision-making closed loop from perceived information to execution instructions. However, the target search method in Embodiment 2 of this application integrates multimodal perception information and precise three-dimensional positioning, which can intelligently understand personnel intentions and make autonomous decisions and executions, making the rescue process fully intelligent and automated, and realizing safe, efficient and accurate autonomous rescue.

[0082] Accordingly, the target search method of this application significantly improves the reliability of perception, the accuracy of rescue and the efficiency of operation in complex environments. At the same time, it can be combined with automatic obstacle avoidance and intelligent decision-making, which greatly enhances the safety and reliability of the entire rescue process and effectively reduces the difficulty of operation and human risk.

[0083] Example 3 The above-described embodiment two involves target localization based on point cloud data and image data to achieve accurate multimodal data fusion. It is important to note that extrinsic parameter calibration of the LiDAR-camera system is an absolute prerequisite for accurate multimodal data fusion. Failure to calibrate or inaccurate calibration will not only render the fusion ineffective but may also produce misleading results. Especially in high-altitude rescue scenarios, even centimeter-level errors can lead to mission failure.

[0084] However, the most common method for extrinsic parameter calibration of LiDAR-camera systems is to directly use laser points extracted from point cloud data to calculate the geometric parameters of the calibration board, thereby achieving extrinsic parameter calibration of the LiDAR and camera. However, due to factors such as poor radar performance and sparse point clouds, the laser points themselves also have a certain relative error, which makes the final extrinsic parameter calibration result inaccurate.

[0085] Therefore, this application provides a method for calibrating the extrinsic parameters of a lidar and a camera, as shown in FIG4(a), which may include the following steps S410-S430.

[0086] Step S410: Acquire multiple sets of image data and multiple sets of point cloud data collected by the camera and the lidar at different calibration plate positions, forming corresponding multiple sets of data pairs.

[0087] The image data should include calibration board image data, and the point cloud data should include calibration board point cloud. Furthermore, the calibration board is preferably a reflector, which is suitable for efficient LiDAR identification.

[0088] Step S420: For each data pair, perform a valid matching pair filtering between the image data and the point cloud data regarding the specified calibration board feature points.

[0089] As shown in Figure 4(b), the effective matching pair screening may include the following steps S421-S424.

[0090] Step S421: Perform feature matching between the image data and the point cloud data with respect to the specified calibration board feature points to obtain multiple sets of feature matching pairs.

[0091] The specified calibration board feature points can be calibration board corner points, calibration board center points, and calibration board edge line points, etc.

[0092] In a preferred embodiment, before performing the feature matching, the feature extraction processes on the camera side and the LiDAR side are optimized respectively to reduce noise interference and improve the accuracy of feature point extraction. Specific optimization measures include the following three aspects: First, optimize camera-side feature extraction.

[0093] Specifically, the calibration board feature points (i.e. visual feature points) corresponding to the image data are obtained through the following steps: the multiple sets of image data are processed using a feature point detection algorithm to obtain multiple sets of initial feature points of the calibration board; and the initial feature points of the calibration board are processed using a sub-pixel thinning function to obtain sub-pixel level calibration board feature points.

[0094] For example, when the feature points on the calibration board are reflector corners, the CornerSubPix function is used instead of traditional corner detection algorithms (such as Harris and Shi-Tomasi algorithms) for accurate extraction of reflector corners. First, a traditional corner detection algorithm is used to initially detect candidate corner regions of the reflector, obtaining coarse coordinates of the corners (in pixel coordinates, i.e., the initial corners of the reflector). Then, the CornerSubPix function is called, using the candidate corner regions as input, setting the window size to 5×5, the number of iterations to 10, and the iteration precision to 0.01 pixels, to refine the coarse corners at the sub-pixel level, obtaining the precise pixel coordinates of the corners. This method effectively reduces the projection error caused by "pixel-level positioning" in traditional corner detection, improving the positioning accuracy of visual feature points.

[0095] Second, optimize the feature extraction on the lidar side.

[0096] Specifically, the calibration board feature points (i.e., point cloud feature points) corresponding to the point cloud data are obtained through the following steps: noise filtering is performed on the multiple sets of point cloud data; and calibration board feature points corresponding to the effective calibration board point cloud data after noise filtering are obtained.

[0097] The noise filtering includes: taking each point cloud in each group of point cloud data as the center, counting the number of adjacent point clouds within a preset radius; if the number of adjacent point clouds is less than a set threshold, then determining that the corresponding point cloud is a noise point and removing the noise point to obtain an initial effective calibration board point cloud cluster; and performing planar fitting on the point clouds in the initial effective calibration board point cloud cluster to obtain a planar model of the calibration board, and removing the point clouds outside the planar model from the initial effective calibration board point cloud cluster to obtain the final effective calibration board point cloud cluster.

[0098] That is, in order to address the problem that the calibration board point cloud is easily affected by environmental noise (such as ground noise and obstacle reflection points), which leads to the feature point extraction offset, the calibration board point cloud is first preprocessed with "radius filtering + plane fitting" and then the reflector feature points are accurately extracted, thereby effectively filtering the feature point offset caused by noise points.

[0099] For example, radius filtering may include: setting a filtering radius R (adaptively adjusted according to the calibration board size and installation distance, usually 0.1-0.3m), taking each point cloud as the center, counting the number of adjacent point clouds within the radius R; if the number of adjacent point clouds is less than the set threshold (usually 3-5), then the point is determined to be a noise point and removed, retaining the effective point cloud cluster of the calibration board, and initially filtering discrete noise points.

[0100] For example, regarding plane fitting, since the calibration board is a planar structure, its point cloud should satisfy the plane equation ax + by + cz + d = 0 (where x, y, and z are the three-dimensional coordinates of the point cloud). Plane fitting can then include: using the Random Sample Consensus Algorithm (RANSAC) to perform plane fitting on the radius-filtered point cloud, setting the number of iterations to 1000 and the inlier threshold to 0.02m, to obtain a planar model of the calibration board; subsequently, removing point clouds outside the planar model that exceed the inlier threshold (remaining noise points) to ensure that the retained point clouds are all valid point clouds on the calibration board surface.

[0101] In the example where the feature points of the calibration board are the corner points of the reflector, the extraction and offset filtering of the reflector corner points can include: extracting the three-dimensional corner coordinates of the reflector based on the geometric contour constraints of the reflector (corner geometric feature recognition) from the effective point cloud after plane fitting, ensuring that the extracted corner points correspond one-to-one with the reflector corner points extracted from the camera side; at the same time, setting a corner offset threshold T (set according to the size of the reflector), calculating the distance from each effective point cloud to the corresponding corner point, if more than 80% of the point cloud distances are less than T, then the corner point is determined to be the true corner point of the reflector; if more than 20% of the point cloud distances are greater than T, then radius filtering and plane fitting are performed again until the requirements are met, effectively filtering out the corner offset caused by noise points and improving the positioning accuracy of the LiDAR feature points.

[0102] Similarly, in the example where the feature point of the calibration board is the center of the reflector, the calculation and offset filtering of the reflector center can include: calculating the reflector center using the "centroid method" on the effective point cloud after plane fitting, that is, calculating the average of the three-dimensional coordinates of all effective point clouds; at the same time, setting a center offset threshold T (set according to the size of the calibration board, usually 0.05m), calculating the distance from each effective point cloud to the centroid point, if more than 80% of the point cloud distances are less than T, then the centroid point is determined to be the true center of the reflector; if more than 20% of the point cloud distances are greater than T, then radius filtering and plane fitting are repeated until the requirements are met, effectively filtering the center offset caused by noise points and improving the positioning accuracy of the LiDAR feature points.

[0103] Step S422: For each feature matching pair, project the feature points of the calibration board corresponding to the image data onto the three-dimensional space of the lidar to obtain the corresponding projected three-dimensional coordinates.

[0104] Step S423: Calculate the distance difference between the projected three-dimensional coordinates and the calibration plate feature points corresponding to the matching point cloud data.

[0105] Step S424: Count the number of feature matching pairs whose distance difference exceeds a set threshold. If the number exceeds the set threshold, determine that the multiple feature matching pairs of the data pair are invalid matching pairs and remove the data pair. Otherwise, retain the data pair as a valid matching pair.

[0106] For steps S422-S424, taking corner points as an example, for the sub-pixel corner points (pixel coordinate system) extracted from the camera side and the calibration board corner points (LiDAR coordinate system) extracted from the LiDAR side, firstly, based on the camera intrinsic parameters and initial extrinsic parameters, the camera-side sub-pixel corner points are projected onto the LiDAR 3D space to obtain the projected 3D corner point coordinates. Then, the distance difference between the projected 3D coordinates and the corresponding corner point on the LiDAR calibration board is calculated. For example, the corner point position corresponding to the projected 3D corner point coordinates is denoted as P. camera The distance to the corner point determined based on point cloud data is denoted as P. lidar The distance difference is denoted as |P|. camera P lidar | Further, count the number of matching pairs in each feature matching group (e.g., 4 matching pairs for 4 corner points) where the "distance error exceeds a set threshold". The threshold is set according to the calibration scenario, usually 0.1m. If the proportion of matching pairs with distance errors exceeding the threshold exceeds 30%, the data group is determined to be invalid data and is directly discarded, not participating in the subsequent external parameter solution. If the proportion of matching pairs with errors exceeding the threshold is ≤30%, the data group is retained and enters the external parameter iterative optimization stage.

[0107] It should be noted that existing technologies typically involve directly substituting all feature matching pairs into the extrinsic parameter solution after obtaining the feature matching pairs in step S210. However, this introduces invalid matching data into the extrinsic parameter solution. To avoid invalid matching data from participating in the extrinsic parameter solution, this embodiment introduces a mandatory feature matching verification mechanism in steps S220-S240. This mechanism rigorously verifies each set of feature matching data, eliminating data with excessive errors and improving the robustness of feature matching. Specifically, each feature matching pair must undergo a mandatory three-step verification process—"projection calculation → error statistics → validity determination"—before the extrinsic parameter solution can be performed. This process eliminates invalid data and ensures data reliability.

[0108] Step S430: Based on the valid matching pairs determined by traversing all data pairs, perform extrinsic parameter iterative optimization to determine the extrinsic parameters between the camera and the lidar.

[0109] In a preferred embodiment, the LM (Levenberg-Marquardt) nonlinear iterative algorithm is used to perform the extrinsic parameter iterative optimization with a joint loss function associated with the camera-side feature error and the radar-side feature error.

[0110] For example, still considering the case where corner points are used as feature points on the calibration board, we first construct a joint loss function. The joint loss function L is defined as follows, combining the infrared feature projection error (visual-side error) and the lidar feature error (radar-side error): L = α·L1 + β·L2 Where α and β are weighting coefficients that can be adaptively adjusted according to the scene to ensure that the errors on both sides have a balanced impact; L1 is the infrared feature projection error, which is the sum of the squared distances between the coordinates of the camera corner point projected onto the 3D space of the lidar and the lidar corner point; L2 is the lidar feature error, which is the sum of the squared distances between the coordinates of the lidar corner point projected onto the camera pixel coordinate system and the camera sub-pixel corner point.

[0111] Secondly, LM nonlinear iterative optimization is performed. Initial extrinsic parameters (rotation matrix R, translation vector t) are substituted into the joint loss function L, and iterative optimization is performed using the LM algorithm. During the iteration, the rotation matrix R and translation vector t are continuously adjusted until the joint loss function L gradually converges to its minimum value. An iteration termination condition is set: when the difference in the loss function between two adjacent iterations is less than 10^-6, or when the number of iterations reaches 50, the iteration stops, and the optimized extrinsic parameters are output. Compared to the traditional least squares method, the LM algorithm combines the convergence speed of the Gauss-Newton algorithm with the stability of the gradient descent algorithm, effectively avoiding getting trapped in local optima, reducing the angle error in solving the rotation matrix (tested to be reduced to within 0.5°), and improving the accuracy of extrinsic parameter calibration.

[0112] In summary, the extrinsic parameter calibration method of this application, compared with existing calibration algorithms, optimizes the feature extraction process to reduce noise interference, adds a mandatory feature matching verification mechanism to eliminate invalid matching data, and adopts a joint loss function combined with a better nonlinear iterative algorithm to achieve high-precision optimization of extrinsic parameters, ultimately reducing extrinsic parameter calibration errors. Specifically, it reduces errors in the following three aspects, achieving higher-precision multi-sensor fusion: First, the feature extraction process was optimized. On the camera side, the CornerSubPix function was used to replace the traditional corner detection algorithm, achieving sub-pixel-level corner localization and reducing the localization error of visual feature points. On the radar side, the reflector point cloud was preprocessed with "radius filtering + plane fitting" to effectively filter environmental noise points (such as ground noise and obstacle reflection points), avoiding reflector center offset caused by noise points and reducing radar feature point localization error.

[0113] Second, a mandatory feature matching verification mechanism is introduced. By verifying the distance difference between the feature point projected into the three-dimensional space and the lidar feature point, invalid data with excessive errors are eliminated, abnormal data is avoided from participating in the external parameter solution, and the impact of matching errors on the overall calibration accuracy is reduced.

[0114] Third, the LM nonlinear iterative algorithm is used to replace the traditional least squares method. The infrared feature projection error and the lidar feature error are used as the joint loss function to replace the single loss function. This effectively avoids getting trapped in local optima during the iteration process and reduces the angle error of solving the rotation matrix. The angle error is controlled within 0.5°.

[0115] Example 4 Based on the above embodiments two and three, embodiment four of this application further proposes a target localization method.

[0116] As shown in Figure 5(a), the target localization method of this embodiment four may include the following steps S510-S540: Step S510: Based on the point cloud data collected by the lidar and the image data collected by the camera, the external parameter calibration between the lidar and the camera is performed using the external parameter calibration method described in Example 3.

[0117] It is understood that the camera is a dual-light camera, and the image data is essentially the dual-light fusion target detection result processed by the Twin-YOLOv8 network model in the above embodiment 1.

[0118] In addition, the specific external parameter calibration method can be found in Example 3 above, and will not be repeated here.

[0119] Accurate extrinsic parameter calibration ensures the accuracy of the conversion between point cloud data and image data. Using extrinsic parameters (including rotation matrices and translation vectors), point clouds from the LiDAR system, synchronized with the camera image acquisition time, are selected and transformed to uniformly convert the point clouds in the LiDAR coordinate system to the camera coordinate system. That is, a point cloud in the camera's 3D coordinate system is obtained, achieving spatial alignment between the point cloud and the camera image. Precise extrinsic parameters can eliminate attitude errors and installation errors between the LiDAR and the camera. After spatial alignment of the point cloud and the camera image, intrinsic parameters are used to project it onto the pixel coordinate system for subsequent ROI extraction and other steps.

[0120] Step S520: After the external parameter calibration, the target point cloud data collected by the lidar is subjected to downsampling processing based on polar coordinate layering to obtain the target key point cloud.

[0121] Downsampling refers to the process of reducing the number of points in a point cloud without destroying its overall structure.

[0122] In a preferred embodiment, as shown in FIG5(b), the downsampling process based on polar coordinate layering corresponding to step S520 may include the following steps S521-S524: Step S521: Convert the target point cloud data into polar coordinates, which include radial distance, azimuth angle and pitch angle.

[0123] For example, the target point cloud data in the lidar coordinate system Convert to polar coordinate point cloud , where radial distance Characterizing the straight-line distance and azimuth angle from the point cloud to the origin of the lidar. Represents the horizontal angle, and the pitch angle Represents the vertical angle. The corresponding conversion formula involves: , as well as .

[0124] Step S522: Divide the target point cloud data into several radial layers based on the radial distance.

[0125] Continuing from the example above, the point cloud is divided into n radial layers based on the radial distance r. , ... The radial distance range of the i-th layer is ,in (Maximum detection range of lidar), interlayer spacing The settings are adaptively adjusted based on the scene (smaller spacing between layers in the foreground and larger spacing between layers in the background).

[0126] Step S523: Determine the downsampling coefficient that is positively correlated with the point cloud density based on the point cloud density of each radial layer.

[0127] Following the example above, define the downsampling coefficients. With respect to the point cloud density of that layer Positively correlated, point cloud density Defined as the number of point clouds in the i-th layer. With respect to the spatial volume point cloud density of this layer The ratio is calculated using the following formula:

[0128] in , The intervals are divided into azimuth and elevation angles, which are determined by the vertical resolution of the lidar.

[0129] Dynamic downsampling coefficients The calculation formula is:

[0130] Thus, dynamic downsampling coefficients are obtained. .

[0131] Step S524: Based on the determined downsampling coefficients, perform point cloud downsampling on each radial layer.

[0132] Taking the human body as an example, the radial point cloud is downsampled by a determined dynamic downsampling coefficient, which reduces the amount of data while preserving the key details of the human body point cloud.

[0133] In summary, based on the above steps S521-S524, the lidar point cloud exhibits a distribution characteristic of being denser in the near and sparser in the far. However, the embodiments of this application dynamically adjust the downsampling coefficient based on the principle of polar coordinate layering, which can balance the point cloud density and computational efficiency.

[0134] Step S530: Perform point cloud segmentation on the target key point cloud to separate environmental point cloud and entity point cloud.

[0135] For example, the first step is to extract the ROI from the visually guided point cloud, which includes converting the target bounding box B (also known as the detection box B) into a pixel mask. The point cloud will be synchronized based on the camera's intrinsic parameter K. Projected onto the pixel coordinate system, retaining the values ​​falling within the pixel coordinate system. Candidate point cloud within Specifically, in a pixel coordinate system consistent with the camera size, pixels within the detection box B are marked as 1 (possible target area), and pixels outside the detection box B are marked as 0 (background area), thus obtaining a pixel mask M and establishing pixel positions to provide a basis for point cloud screening.

[0136] Secondly, adaptive DBSCAN clustering and point cloud segmentation are performed, including: calculating the candidate point cloud density. Dynamically set the DBSCAN neighborhood radius Clustering yields point cloud clusters To further eliminate interference from environmental point clouds, taking the human body as an example, the clusters output by DBSCAN are filtered based on the prior features of human body size to obtain human body point clouds.

[0137] Step S540: Output the location information of the entity point cloud.

[0138] For example, the weighted average method is used to calculate the three-dimensional center of the effective cluster. The weight is the point cloud reflection intensity I. Distance-weighted compensation is introduced:

[0139] d is the target distance.

[0140] The obtained target 3D coordinates (global coordinate system), ID, timestamp and other information are packaged into the location information of the entity point cloud, and sent and displayed on the device interface.

[0141] This application proposes a downsampling method based on polar coordinate layering. Compared with existing commonly used sampling methods (such as fixed-coefficient voxel filtering and random downsampling), it improves processing efficiency in the following three aspects while simultaneously taking into account point cloud density and positioning accuracy: First, following the distribution characteristics of "dense near and sparse far" of lidar point clouds, the radial distance is used as the basis for layering, and the interlayer spacing is adaptively set (smaller interlayer spacing near and larger interlayer spacing far) to avoid the problems of redundant near point clouds and loss of feature of distant point clouds caused by fixed interlayer spacing.

[0142] Second, the downsampling coefficient is dynamically adjusted. The downsampling coefficient is positively correlated with the point cloud density of each layer. The high-density point cloud in the near area is downsampled with a larger intensity to remove redundant data and reduce the amount of computation. The low-density point cloud in the far area is downsampled with a weak intensity to retain key feature points. Compared with the fixed coefficient downsampling method, the amount of point cloud data is reduced by 40%-60%, while avoiding the loss of key features.

[0143] Third, the downsampling calculation logic has been optimized. By combining the spatial characteristics of polar coordinate layering, invalid calculations are reduced, and the point cloud processing efficiency is improved by more than 30% compared with the existing commonly used sampling methods. While reducing the hardware computing pressure, the real-time performance of point cloud segmentation and target positioning is ensured, which is suitable for the dynamic movement and real-time perception scenarios in the process of aerial vehicle rescue, and achieves a dual balance between efficiency and accuracy.

[0144] Example 5 Figure 6 A schematic diagram of the target search and association device according to Embodiment 5 of this application is shown. Figure 6 As shown in the embodiments of this application, the apparatus includes: a memory configured to store instructions; and a processor configured to retrieve the instructions from the memory and, when executing the instructions, to implement the methods of any of the above embodiments.

[0145] The device may be an integrated controller within the operating machinery or a remote controller; this application does not limit this. When the device executes the target detection method of Embodiment 1 above, it is a target detection device based on dual-light fusion.

[0146] For more details on the implementation and effects of the target detection device, please refer to the other embodiments mentioned above, which will not be repeated here.

[0147] Example 6 Figure 7 The schematic diagram illustrates the structure of a target search system according to Embodiment Six of this application, which is based on the same inventive concept as the target search method described in Embodiment Two. Figure 7As shown, the target search system includes: a perception module for acquiring infrared images, visible light images, and point cloud data of the target; a target detection device, such as the "memory + processor" architecture of Embodiment 5, for acquiring the infrared image and the visible light image of the target from the perception module, and outputting target detection results based on the infrared image and the visible light image; a target positioning device, such as the "memory + processor" architecture of Embodiment 5, for acquiring the point cloud data from the perception module and the target detection results from the target detection device, and performing target positioning based on the point cloud data and the target detection results to obtain corresponding target location information; and a location determination module for determining the optimal location point associated with the target search task based on the target location information from the target positioning device and preset search rules.

[0148] The perception module includes a dual-light camera and a lidar. The former simultaneously acquires visible light images and infrared thermal images of the target area, while the latter is used to acquire real-time three-dimensional point cloud data of the target area.

[0149] The target detection device and the target positioning device can be referred to in the other embodiments described above, and will not be repeated here.

[0150] The location determination module, which is, for example, a controller or a functional module in the controller, can also adopt the "memory + processor" architecture of Embodiment 5.

[0151] Furthermore, the target detection device, the target localization device, and the position determination module can collectively constitute an intelligent processing and control module to cooperate with the perception module, thereby achieving target search. The intelligent processing and control module can also be, for example, a controller with a "memory + processor" architecture, with the target detection device, the target localization device, and the position determination module serving as functional units within the controller.

[0152] For more details on the implementation of the target search system in Embodiment Six, please refer to Embodiments One to Five above, which will not be repeated here.

[0153] When the target search system of Embodiment Six is ​​applied to high-altitude rescue, it can be used as a rescue system mounted on rescue equipment (such as fire ladder trucks or high-altitude rescue drones). It integrates the advantages of Embodiments One to Five and has at least the following advantages: First, addressing the issue of poor reliability of existing high-altitude rescue equipment in complex environments such as smoke and low light, this project achieves all-weather, highly robust automatic identification and precise positioning of trapped personnel by fusing visible light and infrared dual-light images with lidar point cloud data. Specifically, based on dual-light cameras and lidar, through deep feature layer fusion and innovative sparse point cloud positioning technology, the project significantly improves the success rate of personnel target detection and the accuracy of 3D positioning in complex scenarios such as rain, fog, smoke, obstructions, and nighttime.

[0154] Secondly, addressing the challenges of complex environments and numerous obstacles at high-altitude rescue sites, and the risk of collisions on the approach paths of rescue equipment, a real-time 3D environmental map constructed using lidar is used to achieve automatic planning of collision-free and optimized safe approach paths for aerial work platform booms. Existing technologies employ 2D image-based perception schemes, which cannot provide the centimeter-level 3D spatial coordinates required for precise interactive rescue.

[0155] Third, based on the identified three-dimensional location and environmental information of the trapped personnel, the rescue location (safety bucket of the aerial ladder truck) can be autonomously confirmed and planned. Existing positioning solutions often cannot understand the intentions of the trapped personnel, such as the inability to recognize their specific postures (e.g., waving or extending their arms for help). Therefore, they cannot intelligently adjust the rescue strategy according to the personnel's proactive behavior and cannot meet high-level interactive rescue needs such as "precisely parking the rescue platform in a position that the person with outstretched arms can safely reach".

[0156] Fourth, in response to the problem that existing rescue systems are isolated in their various stages and do not form a closed loop, this application's embodiment deeply integrates multiple stages such as perception, identification, decision-making, planning, and control execution to construct an intelligent rescue system that can autonomously complete the entire process of "search-identification-understanding-decision-execution," achieving full autonomy and intelligence from target discovery to completion of rescue preparation.

[0157] Other embodiments of this application also provide a working machine, including the target search system of Embodiment Six. This working machine is, for example, intelligent emergency rescue equipment, such as fire ladder trucks and high-altitude rescue drones. The working machine can also be other machinery requiring target search, such as exploration robots and reconnaissance vehicles used in ruins, deep seas, fields, and space.

[0158] Other embodiments of this application also provide a machine-readable storage medium storing instructions that cause a machine to perform the various methods involved in embodiments one to four above.

[0159] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0160] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0161] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0162] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0163] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0164] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0165] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0166] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0167] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A target detection method based on dual-light fusion, characterized in that, include: Acquire infrared and visible light images of the target being detected; as well as The infrared image and the visible light image are input into a pre-trained Twin-YOLOv8 network model to output target detection results; The Twin-YOLOv8 network model is an improved YOLOv8 network model with a dual-branch detection head, and the dual-branch detection head includes: a main branch for predicting the target bounding box and class confidence; and an occlusion perception branch for predicting the occlusion rate.

2. The target detection method according to claim 1, characterized in that, The Twin-YOLOv8 network model includes: A backbone network is used to extract features from the infrared image and the visible light image to obtain corresponding infrared features and visible light features; An attention enhancement module is used to perform attention-weighted feature fusion on the infrared features and the visible light features to obtain corresponding fused features; A neck network is used to perform multi-scale feature enhancement on the fused features to obtain corresponding enhanced features; and The dual-branch detection head, wherein the main branch and the occlusion-aware branch are configured for parallel inference, for outputting the target detection result including the target bounding box, the class confidence, and the occlusion rate based on the enhanced features.

3. The target detection method according to claim 2, characterized in that, The Twin-YOLOv8 network model also includes the following modules positioned before the backbone network: The input module is used to receive the infrared image and the visible light image; as well as The preprocessing module is used to perform image preprocessing on the infrared image and the visible light image received by the input module, and to provide the infrared image and the visible light image after image preprocessing to the backbone network.

4. The target detection method according to claim 2, characterized in that, The neck network adopts a Feature Pyramid Network (FPN)-Path Aggregation Network (PAN) structure, and adds cross-scale attention gating to the FPN to enhance the features of small targets, and configures scale-adaptive convolution in the PAN to enhance the feature details of targets at the corresponding scale.

5. The target detection method according to claim 2, characterized in that, The main branch uses the CIoU loss function to optimize the bounding box regression loss, and / or the occlusion perception branch uses several convolutional layers and labeled occlusion level tags to identify the occlusion rate.

6. The target detection method according to claim 2, characterized in that, The dual-branch detection head is further configured to: correct the category confidence score by combining the occlusion rate before outputting the category confidence score.

7. The target detection method according to claim 1, characterized in that, The target detection method further includes: The Twin-YOLOv8 network model is optimized and trained based on environmental datasets associated with complex scenes and / or target image datasets associated with various target occlusion scenes.

8. A target search method, characterized in that, include: Acquire infrared images, visible light images, and point cloud data of the target; The infrared image and the visible light image are processed using the target detection method according to any one of claims 1 to 7 to output image data characterizing the target detection result; Target localization is performed based on the point cloud data and the image data to obtain the corresponding target location information; as well as Based on the target location information and preset search rules, determine the optimal location point associated with the target search task.

9. A dual-light fusion target detection device, characterized in that, include: The memory is configured to store instructions; as well as The processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the target detection method according to any one of claims 1 to 7.

10. A target search system, characterized in that, include: The sensing module is used to acquire infrared images, visible light images, and point cloud data of the target; The target detection device of claim 9 is configured to acquire the infrared image and the visible light image of the target from the sensing module, and output image data characterizing the target detection result based on the infrared image and the visible light image; A target localization device is used to perform target localization based on the point cloud data and the image data to obtain the corresponding target location information; as well as The location determination module is used to determine the best location point associated with the target search task based on the target location information and preset search rules.

11. A type of operating machinery, characterized in that, Includes the target search system as described in claim 10.

12. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions for causing the machine to perform the target detection method according to any one of claims 1 to 7 or the target search method according to claim 8.

Images