Target detection and positioning method based on laser radar point cloud and infrared image fusion

Abstract

The invention relates to the technical field of multi-sensor fusion perception, and discloses a target detection and positioning method based on laser radar point cloud and infrared image fusion. The method comprises the following steps: constructing a time window based on a historical track by using a space-time heterogeneous association controller to trigger synchronous acquisition; feature conflicts of infrared and laser are detected, and a penetrability geometric depth vector is extracted through waveform hierarchical analysis; a depth gradient is calculated to determine a physical truth value edge point, a geometric confidence attenuation field is constructed to execute morphological contraction on the infrared thermal image, and a precise semantic contour matrix is generated; according to the method, through physical complementation of photo-thermal heterogeneous features, high-precision target reconstruction under strong scattering aerosol interference is realized, and the problem of heterogeneous feature physical decoupling caused by a strong scattering medium is solved.

Description

Technical Field

[0001] This invention relates to the field of multi-sensor fusion sensing technology, and in particular to a target detection and localization method that fuses lidar point clouds and infrared images. Background Technology

[0002] In airspace monitoring applications targeting periodically moving targets such as wind turbine blades, the sensing system needs to maintain high-precision three-dimensional spatial measurement capabilities in all-weather environments. However, in environments with strong scattering aerosols such as dense fog, water vapor clouds, or rain and snow, the optical transmission channel undergoes physical degradation, leading to nonlinear decoupling of the spatiotemporal and semantic dimensions of the feature data acquired by a single sensor. Specifically, the Mie scattering effect caused by suspended particles results in high-intensity near-field echo noise at the lidar receiver. This noise floor masks the weak echo signal of far-field targets, causing blockage and distortion of the geometric ranging channel. Simultaneously, the thermal radiation diffusion effect of the atmospheric medium causes nonlinear halo expansion of the physical boundary of the target in infrared imaging, broadening the point spread function of the infrared image and failing to reflect the precise geometric contour of the target. Existing fusion sensing technologies are usually based on the assumption of clean atmosphere and rely on the independent confidence of a single sensor for weighting. They cannot establish effective three-dimensional spatial constraints when both sensors are simultaneously constrained by physical laws, resulting in the monitoring system failing to output accurate target location results under extreme weather conditions. Summary of the Invention

[0003] This invention provides a target detection and localization method that fuses lidar point clouds and infrared images, aiming to solve the technical problem that strong scattering medium interference causes physical decoupling of heterogeneous sensor features in the spatiotemporal and semantic dimensions, thus making it impossible to achieve high-precision localization.

[0004] In view of the above problems, the present invention provides a target detection and localization method by fusing lidar point clouds and infrared images. This method is applied to a sensing system configured with a spatiotemporal heterogeneous correlation controller. The sensing system includes a full-waveform lidar module, an infrared thermal imaging camera module, and a historical trajectory storage module. The method includes the following steps: Step S1: The spatiotemporal heterogeneous correlation controller calculates the target motion cycle based on the data in the historical trajectory storage module and constructs the target's expected arrival time window; only when the system clock is within the target's expected arrival time window, the full-waveform lidar module and the infrared thermal imaging camera module are activated to perform synchronous acquisition, obtaining the original full-waveform echo data and the original infrared thermal image data; Step S2: The spatiotemporal heterogeneous correlation controller performs feature conflict detection on the original infrared thermal image data and the original full waveform echo data; when a logical conflict is detected between the infrared thermal radiation features and the near-field echo intensity of the lidar, it is determined that there is interference from the scattering medium, and the original full waveform echo data is subjected to layered analysis to extract the secondary echo signal in the far field region to generate a penetrating geometric depth vector. Step S3: The spatiotemporal heterogeneous correlation controller calculates the temporal gradient of the penetration geometric depth vector in the scanning direction to determine the physical truth edge points; uses the physical truth edge points to construct a geometric confidence attenuation field, and performs a morphological shrinkage operation on the original infrared thermal image data based on the geometric confidence attenuation field to remove the thermal diffusion halo region and generate an accurate semantic contour matrix. Step S4: Orthogonally fuse the penetrating geometric depth vector with the precise semantic contour matrix to calculate the three-dimensional spatial coordinates of the target.

[0005] Furthermore, the specific conditions for determining the presence of scattering medium interference in step S2 are as follows: The spatiotemporal heterogeneous correlation controller identifies a connected region in the original infrared thermal image data with a temperature higher than the background temperature; Simultaneously, the lidar beam corresponding to the connected region is retrieved. If the peak intensity of the first echo of the beam within the preset near-field safe distance is greater than the preset scattering threshold, the infrared feature is determined to be a transmission signal and the laser feature to be a scattering signal, confirming that the current condition is under interference from the scattering medium.

[0006] Furthermore, the specific process of performing hierarchical parsing in step S2 includes: The spatiotemporal heterogeneous correlation controller generates a temporal shielding mask to filter out the first echo peak value and its temporal neighbor signals. The secondary echo peak with the highest signal-to-noise ratio is searched in the remaining waveform data after filtering, and the flight time of the secondary echo peak is converted into the penetration geometric depth vector.

[0007] Furthermore, the specific process of determining the physical truth edge points in step S3 includes: Calculate the depth difference between adjacent light spots in the penetration geometry depth vector; When the magnitude of the depth difference jumps and exceeds a preset entity edge threshold, the corresponding position is marked as the physical truth edge point. The spatiotemporal heterogeneous correlation controller calculates the minimum Euclidean distance from each pixel in the original infrared thermal image data to the edge trajectory formed by fitting the physical ground truth edge points.

[0008] Furthermore, the morphological contraction operation performed in step S3 includes constructing a geometric confidence decay field and calculating the thermal radiation overflow entropy; The formula for calculating the geometric confidence decay field is as follows: in, Represents the pixel coordinates of an infrared image. This represents the geometric confidence level of the pixel. Represents pixel coordinates The minimum Euclidean distance to the edge trajectory formed by fitting the edge points of the physical true values. The physical radius parameter of the laser spot. For edge steepness factor.

[0009] Furthermore, the specific process of generating the accurate semantic contour matrix in step S3 also includes: Calculate the local gradient magnitude of the original infrared thermal image data, and construct a thermal radiation spillover entropy model by combining it with the geometric confidence attenuation field: in, Represents pixel coordinates The heat radiation overflow entropy at that location, Represents the original infrared thermal image data in pixel coordinates Local gradient magnitude at [location] It is the numerical stability constant; Based on the thermal radiation overflow entropy, the precise semantic contour matrix is ​​generated through a nonlinear trimming function. : in, Raw pixel intensity of raw infrared thermal image data Geometric weighting balancing factor To suppress the intensity coefficient, This is a numerical truncation function. This represents the upper limit of the overflow entropy.

[0010] Further, in step S1, the logic for generating the target's expected arrival time window is as follows: A time series regression prediction algorithm is applied to the historical target motion time series in the historical trajectory storage module to predict the entry and exit times of the current cycle; Outside the target's expected arrival time window, the spatiotemporal heterogeneous correlation controller enters a low-power standby mode and logically blocks all sensor signal inputs.

[0011] Furthermore, the feature is that the target is the rotating blade of a wind turbine generator set, and the environmental obstacle is the tower of the wind turbine generator set; The scattering medium interference refers to optical scattering interference caused by dense fog, water vapor clouds, or rain and snow aerosols.

[0012] Furthermore, the present invention also provides a perception system, including a full-waveform lidar module, an infrared thermal imaging camera module, a historical trajectory storage module, and a spatiotemporal heterogeneous correlation controller, wherein the spatiotemporal heterogeneous correlation controller is configured to execute the above-described target detection and localization method.

[0013] The technical solution provided in this application has at least the following technical effects: A multi-level orthogonal physical constraint system encompassing time, spectrum, and geometry is constructed through a spatiotemporal heterogeneous correlation controller, enabling the reconstruction of the spatiotemporal consistency of targets under strong scattering environments. This method utilizes historical dynamics priors to lock a high signal-to-noise ratio detection window in the time dimension, avoiding interference from pure background noise; it uses the medium transmission characteristics of infrared spectroscopy as a priori guidance, forcing the lidar signal processing logic to skip near-field scattering echoes and lock far-field secondary echoes, restoring the geometric ranging capability in the Z-axis direction; and it uses the temporal gradient of laser ranging as a physical truth benchmark to perform nonlinear morphological trimming on infrared images undergoing thermal diffusion, restoring the semantic contour accuracy in the XY plane dimension. These processes physically correct the feature decoupling caused by aerosol scattering and thermal radiation diffusion, ensuring that the system can output high-precision three-dimensional positioning coordinates that meet safety monitoring requirements under extreme weather conditions. Attached Figure Description

[0014] Figure 1 A flowchart of a target detection and localization method that fuses lidar point clouds and infrared images, provided in an embodiment of the present invention. Detailed Implementation

[0015] The above technical solutions will now be described in detail with reference to the accompanying drawings and specific embodiments to provide a better understanding of them. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. It should be understood that the present invention is not limited to the exemplary embodiments used only to explain the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. Furthermore, it should be noted that, for ease of description, only the parts related to the present invention are shown in the drawings, not all of them.

[0016] For examples, please refer to Figure 1 A target detection and localization method based on the fusion of lidar point cloud and infrared image, applied to a sensing system configured with a spatiotemporal heterogeneous correlation controller, the sensing system comprising a full-waveform lidar module, an infrared thermal imaging camera module, and a historical trajectory storage module; the method includes the following steps: Step S1: The spatiotemporal heterogeneous correlation controller calculates the target motion cycle based on the data in the historical trajectory storage module and constructs the target's expected arrival time window; only when the system clock is within the target's expected arrival time window, the full-waveform lidar module and the infrared thermal imaging camera module are activated to perform synchronous acquisition, obtaining the original full-waveform echo data and the original infrared thermal image data; Step S2: The spatiotemporal heterogeneous correlation controller performs feature conflict detection on the original infrared thermal image data and the original full waveform echo data; when a logical conflict is detected between the infrared thermal radiation features and the near-field echo intensity of the lidar, it is determined that there is interference from the scattering medium, and the original full waveform echo data is subjected to layered analysis to extract the secondary echo signal in the far field region to generate a penetrating geometric depth vector. Step S3: The spatiotemporal heterogeneous correlation controller calculates the temporal gradient of the penetration geometric depth vector in the scanning direction to determine the physical truth edge points; uses the physical truth edge points to construct a geometric confidence attenuation field, and performs a morphological shrinkage operation on the original infrared thermal image data based on the geometric confidence attenuation field to remove the thermal diffusion halo region and generate an accurate semantic contour matrix. Step S4: Orthogonally fuse the penetrating geometric depth vector with the precise semantic contour matrix to calculate the three-dimensional spatial coordinates of the target.

[0017] In response to a call request from the spatiotemporal heterogeneous correlation controller, the historical trajectory storage module transmits the historical target motion time series stored internally to the spatiotemporal heterogeneous correlation controller. The historical target motion time series received by the spatiotemporal heterogeneous correlation controller includes the entry and exit times of the target through the detection area in multiple past cycles.

[0018] The spatiotemporal heterogeneous correlation controller performs a sliding window averaging operation on the input time series data to extract the target's dynamic motion cycle in the current environment. The specific calculation process is as follows: The spatiotemporal heterogeneous correlation controller selects the duration values ​​of several recent historical cycles, calculates the arithmetic mean of the duration values, and obtains the smoothed dynamic motion cycle parameters. The spatiotemporal heterogeneous correlation controller superimposes the dynamic motion cycle parameters onto the actual departure time of the previous cycle to calculate the theoretical entry time and theoretical departure time of the target in the current cycle. The spatiotemporal heterogeneous correlation controller subtracts a preset pre-buffer time before the theoretical entry time and adds a preset post-buffer time after the theoretical departure time, thereby constructing a target's expected arrival time window covering the entire transit process. The spatiotemporal heterogeneous correlation controller can be an embedded computing platform containing a central processing unit (CPU), a field-programmable gate array (FPGA), or a microcontroller unit (MCU), which integrates a clock management unit, a signal processing unit, and a data fusion unit.

[0019] The system's real-time clock signal is continuously input to the spatiotemporal heterogeneous correlation controller for window comparison. When the system's real-time clock signal falls outside the target's expected arrival time window, it triggers the spatiotemporal heterogeneous correlation controller to generate a sleep command. This sleep command is sent to the full-waveform lidar module and the infrared thermal imaging camera module, logically blocking the sensor's data transmission channels, keeping the system in a low-power standby state, and filtering out static aerosol scattering noise from the background environment.

[0020] The system's real-time clock signal, entering within the target's expected arrival time window, triggers the spatiotemporal heterogeneous correlation controller to generate a high-priority gating command. The full-waveform LiDAR module and the infrared thermal imaging camera module respond to the high-priority gating command and perform synchronous acquisition operations. Hardware trigger signals are output in parallel by the spatiotemporal heterogeneous correlation controller, ensuring synchronization between the laser emission time of the full-waveform LiDAR module and the exposure start time of the infrared thermal imaging camera module. The full-waveform LiDAR module outputs raw full-waveform echo data containing echo intensity and time-of-flight information. The infrared thermal imaging camera module outputs raw infrared thermal image data containing thermal radiation distribution information. The raw full-waveform echo data and raw infrared thermal image data are simultaneously transmitted to the spatiotemporal heterogeneous correlation controller, which establishes a one-to-one correspondence index relationship between data frames, completing the spatiotemporal alignment of the multimodal data.

[0021] The raw infrared thermal image data is transmitted to a spatiotemporal heterogeneous correlation controller to perform thermal radiation feature extraction. The controller first calculates the average grayscale value of all pixels in the raw infrared thermal image data and uses this average grayscale value as the background temperature reference. Pixels with grayscale values ​​exceeding the background temperature reference by a preset temperature difference threshold are marked as high-thermal-foreground points by the controller. Adjacent high-thermal-foreground points are processed by a connected component labeling algorithm and aggregated to form temperature difference connected regions. These temperature difference connected regions characterize the projection of physical targets with significant thermal radiation transmission characteristics in the infrared band.

[0022] The laser scanning channel, spatially corresponding to the temperature difference-connected region, is locked by a spatiotemporal heterogeneous correlation controller. The controller extracts a time segment from the raw full-waveform echo data corresponding to a preset near-field safe distance range. This preset near-field safe distance range corresponds to the physical space from zero to twenty meters in front of the laser emitter. Within this time segment, the controller performs a peak search operation, identifying the waveform peak with the strongest signal intensity as the first echo peak. The first echo peak characterizes the echo signal formed when the laser beam encounters an obstacle or is reflected by a medium in the near-field region.

[0023] The existence status of the temperature difference connectivity region and the intensity value of the first echo peak are simultaneously input to the logic decision unit of the spatiotemporal heterogeneous correlation controller. The spatiotemporal heterogeneous correlation controller compares the intensity value of the first echo peak with a preset scattering intensity threshold. The scattering intensity threshold is set to the typical Mie scattering echo intensity value generated by dense fog or water vapor clouds.

[0024] When a temperature difference connectivity region is detected, and the intensity of the corresponding first echo peak exceeds the scattering intensity threshold, the spatiotemporal heterogeneous correlation controller outputs a scattering medium interference determination result. This result confirms that the current detection path is in a physical condition where the infrared beam can penetrate, but the laser beam is blocked by high-density aerosols in the near field. In response to the scattering medium interference determination result, the spatiotemporal heterogeneous correlation controller activates a layered analysis process for the original full-waveform echo data to extract far-field target information obscured by near-field scattering signals.

[0025] In response to the scattering medium interference determination result, the spatiotemporal heterogeneous correlation controller constructs a temporal shielding mask with a preset time width, centered on the time point corresponding to the first echo peak. The temporal shielding mask is configured as a set of temporal weight sequences, wherein the weight values ​​corresponding to the first echo peak and its adjacent time windows are set to zero, while the weight values ​​corresponding to the far-field region are set to one.

[0026] A point-to-point multiplication operation is performed between the time-domain mask and the original full-waveform echo data on the time axis. This multiplication process forces the waveform amplitude corresponding to the near-field strong scattering signal in the original full-waveform echo data to zero, thus physically eliminating the first echo peak and its resulting trailing interference at the signal level. The data after masking is defined as the residual waveform data. The residual waveform data retains only the far-field echo signal characteristics located beyond the near-field safe distance, eliminating the masking effect of high-intensity near-field scattering noise on the detection of subsequent weak signals.

[0027] The remaining waveform data is input to the constant false alarm rate (CFAR) detection module to perform secondary echo search. The CFAR detection module dynamically calculates a detection threshold curve based on the background noise level in the remaining waveform data. During the time interval when the amplitude of the remaining waveform data exceeds the detection threshold curve, the spatiotemporal heterogeneous correlation controller searches for waveform vertices with local maxima and marks these vertices as secondary echo peaks. Secondary echo peaks characterize the weak echo signal formed by the reflection of the laser beam from the physical target after penetrating the scattering medium.

[0028] The position coordinates of the secondary echo peak on the time axis are extracted and recorded as time-of-flight data by the spatiotemporal heterogeneous correlation controller. The spatiotemporal heterogeneous correlation controller multiplies the time-of-flight data by the speed of light constant and divides it by two to calculate the physical distance value corresponding to the physical target. This physical distance value is combined to form a penetrating geometric depth vector. The penetrating geometric depth vector represents the true spatial depth distribution of the target surface under strong scattering medium interference, and serves as the geometric truth input for subsequent planar dimension edge correction.

[0029] The penetrating geometric depth vector is transmitted to a spatiotemporal heterogeneous correlation controller to perform temporal gradient analysis. The spatiotemporal heterogeneous correlation controller calculates the numerical difference between adjacent sampling points in the penetrating geometric depth vector sequentially along the scanning sequence of the laser beam. The numerical difference reflects the degree of spatial abrupt change in the depth direction of the measured object's surface. The spatiotemporal heterogeneous correlation controller defines the absolute value of the numerical difference as the depth gradient magnitude.

[0030] The depth gradient modulus is compared in real time with a preset entity edge threshold. The entity edge threshold is set to a distance greater than the thickness of the wind turbine blade to distinguish between gradual changes on the blade surface and spatial jumps at the blade edge. When the depth gradient modulus exceeds the entity edge threshold, the corresponding laser scanning time point is marked as a depth step moment by the spatiotemporal heterogeneous correlation controller. The position coordinates of the depth step moment in the spatial coordinate system are recorded as physical ground truth edge points. These physical ground truth edge points constitute the precise geometric contour skeleton of the target in three-dimensional space, without any positional deviation caused by thermal radiation diffusion effects.

[0031] Using the physical ground truth edge points as a spatial reference, a spatiotemporal heterogeneous correlation controller constructs a geometric confidence attenuation field for trimming infrared thermal halo. The geometric confidence attenuation field is generated by calculating the spatial confidence level of each pixel in the original infrared thermal image data relative to the physical ground truth edge points. The specific calculation process is performed according to the following formula: In the formula, Represents the pixel coordinates in the original infrared thermal image data. Represents pixel coordinates The Euclidean distance to the nearest physical truth edge point. The physical radius parameter represents the laser spot, which reflects the spatial uncertainty range of the laser measurement point itself. Representing the edge steepness factor, this factor adjusts the rate at which geometric confidence decays from the target interior towards the background region. The ground truth edge points are processed through interpolation or curve fitting to form continuous edge trajectories. The calculation measures the distance from a pixel to the edge trajectory, not just the distance to isolated points.

[0032] When pixel coordinates Located inside the edge of the physical truth value and Less than When the exponent term in the denominator of the formula approaches zero, the geometric confidence level becomes... The value approaches one, indicating that the region is strongly supported by the true value of laser geometry. When the pixel coordinates Crossing the physical truth edge and Greater than At that time, the value of the exponential term increases rapidly, leading to a decrease in the geometric confidence level. It exhibits a non-linear, sharp decline approaching zero. This non-linear decline characteristic forms a spatial gating, strictly limiting the effective information of the infrared image to the physical geometric range determined by the laser.

[0033] The spatiotemporal heterogeneous correlation controller utilizes a geometric confidence attenuation field to quantify and evaluate thermal diffusion noise in raw infrared thermal image data, constructing a thermal radiation spillover entropy model. This model aims to identify spurious edge regions with high thermal radiation intensity but lacking geometric truth support. The specific value of the thermal radiation spillover entropy is calculated using the following formula: In the formula, Represents pixel coordinates The value of thermal radiation overflow entropy at that location. Represents the original infrared thermal image data in pixel coordinates The local gradient magnitude reflects the degree of drastic change in the grayscale value of the infrared image. The geometric confidence score generated by the preceding steps. The numerical stability constant is set to a very small positive number to prevent calculation errors with zero denominators in background regions where the geometric confidence level approaches zero.

[0034] According to this formula, when a pixel is outside the physical target (i.e., geometric confidence level) (Extremely low) but the infrared image shows a dramatic gradient change (i.e., local gradient magnitude). When the value is relatively large, the calculated thermal radiation overflow entropy is... This will result in an extremely large value. This extremely large value physically indicates that the thermal characteristics of this pixel are thermal halo noise caused by atmospheric scattering, rather than the actual object edge.

[0035] Based on the calculated thermal radiation overflow entropy, the spatiotemporal heterogeneous correlation controller performs a nonlinear morphological trimming operation on the raw infrared thermal image data to generate an accurate semantic contour matrix. This trimming operation uses an inverse exponential function to physically remove pixel intensities in high overflow entropy regions. The specific values ​​of the accurate semantic contour matrix are generated according to the following formula: In the formula, This represents the corrected pixel intensity value. The raw pixel intensity represents the original infrared thermal image data. This represents the geometric weight balancing factor, used to adjust the weight ratio between direct geometric constraints and entropy suppression constraints. This represents the suppression strength coefficient, a tuning parameter used to map the overflow entropy to the exponential decay domain. This represents a numerical truncation function that limits the thermal radiation overflow entropy to between zero and the upper limit of the overflow entropy. Within a certain range, to prevent numerical underflow in exponential operations.

[0036] This formula achieves targeted signal modulation through an exponential decay term: in hazy regions with high thermal radiation spillover entropy, the modulation coefficient rapidly approaches zero, thus forcibly suppressing the original infrared intensity to the background level; in solid regions with low thermal radiation spillover entropy, the modulation coefficient remains at a high level, preserving the texture information of the original infrared thermal image. After full-image pixel traversal calculation, the output precise semantic contour matrix is ​​the high-precision target contour data after removing thermal haze errors.

[0037] The penetrating geometric depth vector and the precise semantic contour matrix are simultaneously transmitted to the orthogonal fusion unit of the spatiotemporal heterogeneous correlation controller. The spatiotemporal heterogeneous correlation controller calls the pre-stored sensor extrinsic calibration matrix to map the two-dimensional infrared image coordinate system containing the precise semantic contour matrix to the three-dimensional Cartesian coordinate system of the lidar. The specific reconstruction process is as follows: the spatiotemporal heterogeneous correlation controller uses the depth value in the penetrating geometric depth vector as the Z-axis coordinate constraint and the highlight pixel coordinates in the precise semantic contour matrix as the XY plane boundary constraint, and performs a back projection operation.

[0038] Back projection transforms each pixel on the 2D infrared contour, along with its corresponding penetrating geometric depth value, into a point cloud data point in 3D space. The set of all transformed point cloud data points constitutes the target's 3D point cloud model. This 3D point cloud model combines the penetrating detection advantages of lidar in the depth direction with the high-resolution semantic advantages of infrared thermal imaging in the planar direction, correcting spatial positional deviations caused by aerosol scattering and thermal radiation diffusion. A spatiotemporal heterogeneous correlation controller further performs bounding box fitting calculations on the 3D point cloud model, generating an oriented 3D bounding box that tightly encloses the target's physical entity.

[0039] The spatiotemporal heterogeneous correlation controller loads a 3D digital model of an environmental obstacle, which corresponds to the tower structure of a wind turbine generator. The controller performs distance detection calculations in 3D space, calculating the Euclidean distance between all points on the surface of the oriented 3D bounding box and the surface of the 3D digital model of the environmental obstacle. The smallest calculated Euclidean distance is extracted and defined as the minimum Euclidean distance.

[0040] The minimum Euclidean distance is compared in real time between the spatiotemporal heterogeneous correlation controller and a preset safety threshold. The preset safety threshold is set according to the safety operation specifications of wind turbine generators. When the minimum Euclidean distance is less than the preset safety threshold, the spatiotemporal heterogeneous correlation controller immediately generates an alarm signal. The alarm signal is sent to the main control system of the wind turbine generator through the industrial communication interface, triggering emergency shutdown or pitch avoidance protection actions to prevent blade sweeping accidents.

[0041] After completing the detection and localization of the target, the spatiotemporal heterogeneous correlation controller extracts the timestamp data of the detected target and marks this timestamp data as the actual arrival time of the target. The actual arrival time of the target is written back to the historical trajectory storage module via the data bus. The historical trajectory storage module appends the newly written actual arrival time of the target to the end of the original historical target motion time series and removes the oldest set of data to maintain a constant sequence length.

[0042] The updated historical target motion time series will be retrieved again by the spatiotemporal heterogeneous correlation controller before the start of the next detection cycle. Based on the updated data, the spatiotemporal heterogeneous correlation controller re-runs the time series regression prediction algorithm, correcting the calculation parameters of the dynamic motion cycle. This closed-loop feedback mechanism ensures that the system can adaptively follow changes in the wind turbine generator speed, continuously calibrate the accuracy of the target's expected arrival time window, and maintain long-term stability with consistent spatiotemporal behavior throughout the entire process.

[0043] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target detection and localization method based on the fusion of lidar point cloud and infrared image, characterized in that, This method is applied to a sensing system configured with a spatiotemporal heterogeneous correlation controller, the sensing system including a full-waveform lidar module, an infrared thermal imaging camera module, and a historical trajectory storage module; the method includes the following steps: Step S1: The spatiotemporal heterogeneous correlation controller calculates the target motion cycle based on the data in the historical trajectory storage module and constructs the target's expected arrival time window; only when the system clock is within the target's expected arrival time window, the full-waveform lidar module and the infrared thermal imaging camera module are activated to perform synchronous acquisition, obtaining the original full-waveform echo data and the original infrared thermal image data; Step S2: The spatiotemporal heterogeneous correlation controller performs feature conflict detection on the original infrared thermal image data and the original full waveform echo data; when a logical conflict is detected between the infrared thermal radiation features and the near-field echo intensity of the lidar, it is determined that there is interference from the scattering medium, and the original full waveform echo data is subjected to layered analysis to extract the secondary echo signal in the far field region to generate a penetrating geometric depth vector. Step S3: The spatiotemporal heterogeneous correlation controller calculates the temporal gradient of the penetration geometric depth vector in the scanning direction to determine the physical truth edge points; uses the physical truth edge points to construct a geometric confidence attenuation field, and performs a morphological shrinkage operation on the original infrared thermal image data based on the geometric confidence attenuation field to remove the thermal diffusion halo region and generate an accurate semantic contour matrix. Step S4: Orthogonally fuse the penetrating geometric depth vector with the precise semantic contour matrix to calculate the three-dimensional spatial coordinates of the target.

2. The method according to claim 1, characterized in that, The specific conditions for determining the presence of scattering medium interference in step S2 are as follows: The spatiotemporal heterogeneous correlation controller identifies a connected region in the original infrared thermal image data with a temperature higher than the background temperature; Simultaneously, the lidar beam corresponding to the connected region is retrieved. If the peak intensity of the first echo of the beam within the preset near-field safe distance is greater than the preset scattering threshold, the infrared feature is determined to be a transmission signal and the laser feature to be a scattering signal, confirming that the current condition is under interference from the scattering medium.

3. The method according to claim 2, characterized in that, The specific process of performing hierarchical parsing in step S2 includes: The spatiotemporal heterogeneous correlation controller generates a temporal shielding mask to filter out the first echo peak value and its temporal neighbor signals. The secondary echo peak with the highest signal-to-noise ratio is searched in the remaining waveform data after filtering, and the flight time of the secondary echo peak is converted into the penetration geometric depth vector.

4. The method according to claim 1, characterized in that, The specific process for determining the physical truth edge points in step S3 includes: Calculate the depth difference between adjacent light spots in the penetration geometry depth vector; When the magnitude of the depth difference jumps and exceeds the preset entity edge threshold, the corresponding position is marked as the physical truth edge point; The spatiotemporal heterogeneous correlation controller calculates the minimum Euclidean distance from each pixel in the original infrared thermal image data to the edge trajectory formed by fitting the physical ground truth edge points.

5. The method according to claim 4, characterized in that, The morphological contraction operation performed in step S3 includes constructing a geometric confidence decay field and calculating thermal radiation overflow entropy. The formula for calculating the geometric confidence decay field is as follows: in, Represents the pixel coordinates of an infrared image. This represents the geometric confidence level of the pixel. Represents pixel coordinates The minimum Euclidean distance to the edge trajectory formed by fitting the edge points of the physical true values. The physical radius parameter of the laser spot. For edge steepness factor.

6. The method according to claim 5, characterized in that, The specific process of generating the accurate semantic contour matrix in step S3 also includes: Calculate the local gradient magnitude of the original infrared thermal image data, and construct a thermal radiation spillover entropy model by combining it with the geometric confidence attenuation field: in, Represents pixel coordinates The heat radiation overflow entropy at that location, Represents the original infrared thermal image data in pixel coordinates Local gradient magnitude at [location] It is the numerical stability constant; Based on the thermal radiation overflow entropy, the precise semantic contour matrix is ​​generated through a nonlinear trimming function. : in, Raw pixel intensity of raw infrared thermal image data Geometric weighting balancing factor To suppress the intensity coefficient, This is a numerical truncation function. This represents the upper limit of the overflow entropy.

7. The method according to claim 1, characterized in that, In step S1, the logic for generating the target's expected arrival time window is as follows: A time series regression prediction algorithm is applied to the historical target motion time series in the historical trajectory storage module to predict the entry and exit times of the current cycle; Outside the target's expected arrival time window, the spatiotemporal heterogeneous correlation controller enters a low-power standby mode and logically blocks all sensor signal inputs.

8. The method according to any one of claims 1 to 7, characterized in that, The target is the rotating blade of a wind turbine generator set, and the environmental obstacle is the tower of the wind turbine generator set; The scattering medium interference refers to optical scattering interference caused by dense fog, water vapor clouds, or rain and snow aerosols.

9. A sensing system, characterized in that, include: A full-waveform lidar module is used to emit laser beams and receive full-waveform echo data; Infrared thermal imaging camera module, used to acquire infrared thermal image data; The historical trajectory storage module is used to store the target's motion time series data; And a spatiotemporal heterogeneous correlation controller, which is communicatively connected to the full-waveform lidar module, the infrared thermal imaging camera module and the historical trajectory storage module, for executing the target detection and localization method of lidar point cloud and infrared image fusion as described in any one of claims 1 to 8.

Images