Eagle eye multi-modal data pre-processing system

Abstract

The present application relates to the technical field of multi-sensor fusion and intelligent perception data processing, in particular to an eagle-eye multi-modal data preprocessing system; comprising: collecting a first visible light image, a thermal imaging image, a laser radar point cloud, a second visible light image sequence and environmental sensing data at a reference time and establishing a unified time reference; calculating a risk score according to the average motion amplitude, the average gray difference absolute value and the thermal region proportion of the second visible light image sequence and controlling the compensation start-stop; generating a thermal mask and a working mask according to the environmental background temperature baseline; establishing a two-dimensional cumulative displacement field within the working mask, combining the point-level scanning time offset to perform projection compensation on the point cloud, and outputting a unified data package containing the original point cloud or the compensated point cloud, the corresponding image and the reliability information; which can reduce invalid compensation calculation and improve the consistency of the point cloud and the spatial state at the reference time.

Description

Technical Field

[0001] This invention relates to the field of multi-sensor fusion and intelligent sensing data processing technology, specifically the Eagle Eye multimodal data preprocessing system. Background Technology

[0002] Currently, multimodal perception processing in open-pit mines, bulk cargo terminals, and stockpile yards is usually achieved by aligning the whole-frame timestamps of visible light images, thermal images, and lidar point clouds. When there are dynamic semi-transparent targets such as dust diffusion, material spillage, or equipment exhaust in the application scenario, the existing solutions cannot accurately compensate for the time difference formed by lidar point-by-point scanning. Furthermore, the trigger judgment, compensation range constraint, and output quality labeling for dynamic areas are insufficient, which can easily lead to misalignment between point clouds and images, waste of computing resources, and decreased efficiency in subsequent processing. Summary of the Invention

[0003] The purpose of this invention is to provide an Eagle Eye multimodal data preprocessing system to solve the following technical problems: Existing multimodal sensing and processing technologies for mining operations have significant shortcomings in accurately compensating for the time difference of point-by-point scanning of lidar and determining regional triggering when facing dynamic semi-transparent targets. There is an urgent need to propose a system that can achieve point-by-point displacement compensation based on a unified time reference and dynamically process data as needed. The purpose of this invention can be achieved through the following technical solutions:

[0004] The Eagle Eye multimodal data preprocessing system includes: a multimodal acquisition module, which acquires a first visible light image, a thermal imaging image, a lidar point cloud, a second visible light image sequence, and environmental sensing data at a reference time, wherein the lidar point cloud contains multiple lidar points;

[0005] The risk assessment module calculates a risk score based on the second visible light image sequence and the thermal imaging image, and outputs the compensation enable status.

[0006] The thermal mask generation module generates a thermal mask based on the environmental sensing data and the thermal imaging image, and determines the working mask based on the thermal mask.

[0007] The displacement field generation module generates a two-dimensional cumulative displacement field within the working mask, based on the second visible light image sequence and the time difference between the reference time and the sampling time of each of the plurality of lidar points.

[0008] The time offset acquisition module acquires the scanning time offset of each lidar point in the lidar point cloud relative to the reference time.

[0009] The projection compensation module projects the lidar point cloud onto the image plane. For lidar points falling within the working mask, it performs spatiotemporal displacement compensation and back projection by combining the corresponding scanning time offset with the two-dimensional cumulative displacement field to obtain the compensated point cloud.

[0010] The data output module selects to output the lidar point cloud or the compensated point cloud according to the compensation enable state, and outputs a unified data packet containing the corresponding reference time and timestamp.

[0011] Preferably, the system further includes a credibility generation module, which is used to generate motion consistency values, thermal region consistency values ​​and boundary stability values ​​respectively, and to determine the credibility of each compensation point based on the motion consistency values, thermal region consistency values ​​and boundary stability values;

[0012] The motion consistency value is generated based on the local smoothness of the two-dimensional cumulative displacement field, the thermal region consistency value is generated based on the spatial relationship between the compensation point and the thermal mask, and the boundary stability value is generated based on the distance of the compensation point from the boundary information extracted from the thermal imaging image or the second visible light image sequence, or the boundary crossing situation.

[0013] The motion consistency value, thermal region consistency value, boundary stability value, and confidence level are all normalized values.

[0014] The credibility is generated by fusing the motion consistency value, the thermal region consistency value, and the boundary stability value.

[0015] When the compensation point is located within the thermal mask, the thermal region consistency value is a first preset value; when the compensation point is located within the working mask and outside the thermal mask, the thermal region consistency value is a second preset value; wherein, the first preset value is greater than the second preset value.

[0016] The data output module is also used to mark compensation points with a confidence level lower than a preset confidence level threshold as low confidence points and output them along with the unified data packet.

[0017] Preferably, the system further includes a feedback adjustment module, which is used to receive the processing time, low confidence point ratio, compensation point ratio and anomaly marker of the current cycle, and send the feedback results to the risk judgment module, the thermal mask generation module and the displacement field generation module, and update at least one parameter for the next cycle. The parameters include the start threshold and stop threshold for the risk judgment module, the temperature difference threshold and morphological expansion parameter for the thermal mask generation module, and the smoothing weight for the displacement field generation module.

[0018] The anomaly markers are generated by the data output module or the confidence generation module based on the proportion of low confidence points, the proportion of out-of-bounds compensation points, or processing timeouts. The proportion of out-of-bounds compensation points is the percentage of compensation points that exceed the effective range of the image, the range of the working mask, or the range of the target boundary after compensation, out of all compensation points.

[0019] Preferably, the risk assessment module is further used for:

[0020] Candidate dynamic regions are extracted from the second visible light image sequence, and a brightness fluctuation factor is obtained based on the absolute value of the average gray level difference between the candidate dynamic regions and adjacent images.

[0021] After normalizing the average motion amplitude of the candidate dynamic region in the second visible light image sequence, the proportion of the hot region in the thermal imaging image, and the brightness fluctuation factor, a risk score is obtained by fusing and calculating based on the normalized average motion amplitude, the proportion of the hot region, and the brightness fluctuation factor.

[0022] Wherein, the start threshold is greater than the stop threshold; when the risk score is not less than the start threshold, the start compensation enable state is output; when the risk score is not greater than the stop threshold, the stop compensation enable state is output; when the risk score is between the start threshold and the stop threshold, the compensation enable state of the previous cycle is maintained.

[0023] Preferably, the thermal mask generation module is used to determine the environmental background temperature baseline based on environmental sensing data, or based on background area statistics in the thermal imaging image, or based on a combination of environmental sensing data and background area statistics; the background area is the area in the thermal imaging image located within a preset non-target area and / or with a temperature lower than a preset upper limit of background temperature.

[0024] Preferably, the displacement field generation module is specifically used for:

[0025] Calculate the two-dimensional displacement of adjacent images in the working mask-defined area of ​​the second visible light image sequence; accumulate the two-dimensional displacement of the adjacent images according to the order of the sampling time of each lidar point in the lidar point cloud from the reference time;

[0026] For each lidar point, displacement interpolation is performed by combining the time offset between adjacent image sampling times of each lidar point to obtain the cumulative displacement field at the corresponding time; and the cumulative displacement field is locally smoothed within the working mask-defined area to obtain the smoothed cumulative displacement field.

[0027] Preferably, the time offset acquisition module is used to acquire the scanning time offset in one of the following ways: when the original LiDAR data contains point-level sampling times, the scanning time offset is determined based on the difference between the sampling times of each point and the reference time; when the original LiDAR data does not contain point-level sampling times but contains scanning angle information, the scanning time offset is determined based on the scanning angle, starting scanning angle and scanning angular velocity of each point in the original LiDAR data or the preset parameters of the device, and the scanning time offset is limited to the range of a single frame scanning cycle.

[0028] Preferably, the projection compensation module is specifically used for:

[0029] Transform the lidar points to the camera coordinate system based on the lidar-to-camera extrinsic parameters;

[0030] The converted LiDAR points are projected onto the image plane based on the camera's intrinsic parameters.

[0031] For lidar points that do not fall into the working mask, retain their original coordinates. For lidar points that fall into the working mask, read the displacement in the two-dimensional cumulative displacement field at the corresponding time according to the corresponding scanning time offset. Then, translate the pixel coordinates of the lidar points that fall into the working mask in the image plane in the opposite direction of the read displacement to compensate to the corresponding position at the reference time.

[0032] The compensated pixel coordinates are back-projected while keeping the original depth unchanged to obtain the compensated 3D point coordinates; the compensated 3D point coordinates are then transformed back into the lidar coordinate system to form a compensated point cloud.

[0033] Preferably, the projection compensation module is further used for:

[0034] When the radial velocity of the target is greater than a preset velocity threshold or the projected displacement of the same target in two consecutive frames of the second visible light image sequence is greater than a preset displacement threshold, the depth before compensation is corrected according to the depth correction amount before backprojection.

[0035] The radial velocity of the target is determined by lidar velocity measurement values, external velocity measurement data, or by the change in spatial distance between adjacent time points of the centroid of the same cluster extracted after performing clustering processing on two consecutive frames of point clouds.

[0036] Preferably, the multimodal acquisition module acquires the first frame of the first visible light image and the first frame of the thermal imaging image at a reference time as an alignment reference, and continuously acquires multiple frames of images within a single frame scanning cycle of the lidar point cloud to form the second visible light image sequence.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. This system outputs compensation enable status based on risk score to achieve on-demand startup, avoiding redundant data computation load caused by global indiscriminate compensation; at the same time, it uses the generated working mask to limit the displacement field calculation range, and combines the scanning time offset of each lidar point to read the displacement for reverse translation, solving the dynamic target misalignment problem caused by traditional whole frame time alignment.

[0039] 2. This system combines the two-dimensional cumulative displacement field, thermal mask, and boundary information extracted from the image to generate the confidence level of compensation points. Low confidence points are marked according to the set value and then output with a unified data packet. This method provides an accurate basis for distinguishing point-level quality for subsequent processing, avoids mixing high and low reliability data, and solves the problem of reduced post-processing efficiency caused by the lack of quality labeling in existing solutions. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0041] Figure 1 This is an architecture diagram of the Eagle Eye multimodal data preprocessing system of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0043] like Figure 1 As shown, this implementation is applicable to edge processing nodes such as open-pit mines, bulk cargo terminals, and stockpile and reclaim yards that need to simultaneously receive visible light, thermal imaging, and lidar data. The system uses a unified reference time within a processing cycle as a benchmark to write high-resolution first visible light image, thermal imaging image, lidar point cloud, second visible light image sequence, and environmental sensing data into the same cycle buffer.

[0044] The periodic buffer is preferably set as a shared memory area of ​​the edge processing node. All types of data are written with timestamps and source identifiers and are released after the current period processing ends. The system first determines whether displacement compensation of the lidar point cloud needs to be started in this period based on the second visible light image sequence and thermal imaging image.

[0045] A hot mask and a working mask are generated only when the risk score meets the activation conditions. Then, a two-dimensional cumulative displacement field is established under the constraint of the working mask. The point within the working mask is then projected to compensate for the scanning time offset of each lidar point. The output is a unified data packet containing the original point cloud or compensated point cloud, the corresponding image, and confidence information. This method avoids performing compensation calculations indiscriminately for all cases, reduces invalid processing of static areas, and makes the output point cloud closer to the spatial state at the unified reference time.

[0046] The Eagle Eye multimodal data preprocessing system includes:

[0047] The multimodal acquisition module acquires a first visible light image, a thermal imaging image, a lidar point cloud, a second visible light image sequence, and environmental sensing data at a reference time. The lidar point cloud contains multiple lidar points.

[0048] The risk assessment module calculates a risk score based on the second visible light image sequence and thermal imaging image, and outputs the compensation enable status.

[0049] The thermal mask generation module generates a thermal mask based on environmental sensing data and thermal imaging images, and determines the working mask based on the thermal mask.

[0050] The displacement field generation module generates a two-dimensional cumulative displacement field within the working mask, based on the second visible light image sequence and the time difference between the reference time and the sampling time of each of the multiple lidar points.

[0051] The time offset acquisition module acquires the scanning time offset of each lidar point in the lidar point cloud relative to the reference time.

[0052] The projection compensation module projects the LiDAR point cloud onto the image plane. For LiDAR points that fall within the working mask, it performs spatiotemporal displacement compensation and back projection by combining the corresponding scanning time offset and the two-dimensional cumulative displacement field to obtain the compensated point cloud.

[0053] The data output module selects to output either lidar point cloud or compensated point cloud based on the compensation enable status, and outputs a unified data packet containing the corresponding reference time and timestamp.

[0054] Based on the above embodiments, the main steps in the system of the embodiments are described as follows;

[0055] Acquire a first visible light image, a thermal imaging image, a lidar point cloud, a second visible light image sequence, and environmental sensing data. Based on a unified time reference, perform time correlation on various types of data to generate a periodic input dataset. Here, the unified time reference means that the data from each sensor within the same processing cycle are aligned with the same reference image sampling time as the alignment base point. For example, the first visible light image and the thermal imaging image are used as snapshot references, and the lidar point cloud and the second visible light image sequence are unfolded around this reference.

[0056] The first visible light image is mainly used for subsequent fusion display and high-precision alignment reference; the second visible light image sequence is mainly used to estimate the two-dimensional motion of dynamic targets within the scanning window; and environmental sensing data is used to help determine the environmental background temperature baseline.

[0057] Conventional solutions often only perform frame-by-frame timestamp alignment, treating the entire frame of LiDAR point cloud as data at the same moment, which does not truly reflect the time difference caused by point-by-point scanning. This implementation method incorporates the entire frame image, the sequence image, and the point-level time information into the same processing link by using a unified time reference and a periodic buffer, so that each subsequent LiDAR point can be mapped to its actual scanning offset.

[0058] This embodiment can achieve consistent organization of multimodal inputs based on a unified time reference, avoid timing mismatch caused by different sensors processing independently, and ultimately solve the core problem of misalignment between dynamic images and point clouds caused by the lack of point-by-point time reference in the existing technology.

[0059] A second visible light image sequence is acquired. Based on the candidate dynamic regions between adjacent images, the motion amplitude and grayscale changes of the candidate dynamic regions are calculated to generate a risk score and determine the compensation enable state. The candidate dynamic region refers to the region in the second visible light image sequence where there is a pixel displacement or brightness change that exceeds a preset threshold, such as the edge area of ​​spilled material, the dust plume diffusion area, and the area near the equipment exhaust.

[0060] The average motion amplitude is used to characterize the strength of the displacement of pixels in the region between adjacent images, and the average gray level difference absolute value is used to characterize the brightness instability of the region between adjacent images. The brightness fluctuation factor is obtained by normalizing the average gray level difference absolute value according to the preset gray level range. Specifically, during normalization, the difference between the current average gray level difference absolute value and the preset gray level lower limit is calculated, and then it is divided by the difference between the preset gray level upper limit and the preset gray level lower limit. The system here forces the preset gray level upper limit to be strictly greater than the preset gray level lower limit to avoid the denominator being zero and causing program abnormalities. If the calculation result is less than 0, it is taken as 0, and if it is greater than 1, it is taken as 1, thereby limiting the brightness fluctuation factor to the range of 0 to 1.

[0061] In practice, the average motion amplitude, hot area proportion, and brightness fluctuation factor are normalized respectively. For example, the average motion amplitude is normalized by dividing the preset maximum reasonable motion displacement by that maximum displacement. The hot area proportion is directly mapped to a decimal between 0 and 1 with 100% as the full scale. Then, they are weighted and summed according to preset weights to obtain the risk score. The risk score The formula used to determine whether there are dynamically semi-transparent targets requiring compensation in the current period is:

[0062]

[0063] In the formula, This is the normalized average motion amplitude, used to characterize the motion intensity of candidate dynamic regions in the image sequence. This represents the percentage of the hot region, used to characterize the extent to which thermal anomalies are covered in the image. This is a brightness fluctuation factor used to characterize the time-domain brightness fluctuations caused by targets such as dust and spilled materials. Preset weights are used to balance the contribution of motion information under the current operating conditions. The preset weights for the contribution of thermal information under the current operating conditions. The preset weights for the contribution of brightness fluctuation information under the current operating conditions. , , All are non-negative values ​​and the sum of their weights is 1;

[0064] Risk assessment employs a dual-threshold approach: the activation threshold is greater than the deactivation threshold; when the risk score is not less than the activation threshold, the activation compensation enable state is output; when the risk score is not greater than the deactivation threshold, the deactivation compensation enable state is output; when the risk score is between the two, the compensation enable state of the previous cycle is maintained; the activation and deactivation thresholds can be set to different value ranges based on field samples, for example, the activation threshold can be between 0.55 and 0.70, and the deactivation threshold can be between 0.35 and 0.50; single-threshold gating is not sensitive enough to boundary conditions and is prone to frequent switching in situations such as intermittent dust occurrence and instantaneous material spillage;

[0065] This implementation method uses the coordinated judgment of motion, heat ratio, and brightness fluctuation, combined with a dual threshold holding mechanism, to accurately activate when the risk increases and maintain the previous cycle state when the risk decreases but has not yet stabilized, reducing switching jitter. This implementation method can realize the on-demand activation of the compensation link based on multi-source risk scoring, avoiding indiscriminate reprocessing of all cycles, and solving the problem of excessive processing resource consumption and compensation state jitter caused by the lack of a stable triggering mechanism in the prior art.

[0066] Acquire thermal imaging images, perform temperature difference segmentation on the thermal imaging images based on the ambient background temperature baseline, generate thermal masks, and perform morphological expansion on the thermal masks to determine the working mask; the ambient background temperature baseline refers to the reference temperature used to characterize the background temperature level of the current situation, which can be determined by environmental sensing data, statistical analysis of the background area in the thermal imaging images, or a combination of both.

[0067] Background area refers to the area in a thermal imaging image that is located within a preset non-target area and / or has a temperature lower than the preset background temperature upper limit, such as ground areas far from work vehicles and material flow, and low-temperature background wall areas; thermal mask refers to the set of pixels whose temperature is higher than the combination of the background temperature baseline and the temperature difference threshold; working mask is the compensation area obtained by morphological expansion based on thermal mask, used to cover the semi-transparent target neighborhood with discontinuous thermal boundaries.

[0068] In practical implementation, a hot mask can be constructed according to the following formula: when pixel Temperature values ​​in thermal imaging images Compared with the ambient background temperature baseline used to reflect the current cycle background heat level The difference is not less than the temperature difference threshold used to distinguish dynamic thermal regions from ordinary background thermal disturbances. At that time, the pixel belongs to the hot mask, i.e. ;otherwise Temperature difference threshold The value can be set to a preset range according to the working conditions, such as the field calibration range corresponding to 2.5 degrees Celsius to 6 degrees Celsius;

[0069] Obtain the binary hot mask Then, the shape is expanded using the expansion operation function in mathematical morphology to generate a working mask. :

[0070]

[0071] in, The expansion parameter defines the neighborhood range that extends outward from the thermal boundary. Examples of values ​​for this parameter are 3 to 11 pixels of the second visible light image. This parameter is used to reserve compensation margin for areas where the thermal boundary is discontinuous, dust edges are sparse, or thermal distribution remains incomplete.

[0072] Traditional fixed temperature threshold segmentation methods are easily affected by solar reflected hot spots, localized equipment heating, and ambient temperature fluctuations. This implementation method uses ambient temperature and background area statistics to jointly constrain the background baseline, and then uses an expanded working mask to limit the scope of action to the real dynamic thermal area and its neighborhood, thereby suppressing background noise and avoiding miscompensation of the static background area.

[0073] This embodiment can achieve directional filtering of dynamic regions based on background temperature baseline and working mask, avoiding distortion of compensation range caused by thermal noise and boundary breaks, and ultimately solving the problem of mixed compensation objects caused by lack of regional constraints in the existing technology.

[0074] The system acquires risk scores, working masks, LiDAR point clouds, and second visible light image sequences. Based on the compensation enable state, it performs displacement compensation on the points within the working mask and generates a unified data packet. The unified data packet refers to a data structure that includes the image corresponding to the reference time, point cloud, timestamp, and compensation-related flag information.

[0075] In this embodiment, the data packet is preferably organized in the form of a structured memory object within the edge processing node and packaged into data units with a fixed field order before output, so that subsequent identification, calculation and tracing programs can directly read it; when the compensation enable signal is off, the data output module directly selects the original point cloud to write into the unified data packet; when the compensation enable signal is on, the data output module selects the compensated point cloud to write into the unified data packet, and attaches the confidence result at the same time.

[0076] The unified data packet contains the following fields: first visible light image, thermal imaging image, working mask, point cloud coordinate set, point-level confidence set, environmental sensing data, risk score, compensation enable status, and unified timestamp; after writing, the data packet can be directly read by the subsequent identification program through object reference, or it can be encapsulated by the edge node into a standard message and sent to the upper-layer analysis program.

[0077] This embodiment can select between the original point cloud and the compensated point cloud for output based on the compensation enable state, avoiding the need for subsequent modules to repeatedly adapt to different data formats at the interface level, and ultimately solving the problem of complex post-processing links caused by inconsistent output structures in existing technologies.

[0078] The system also includes a feedback adjustment module, which receives the processing time, low confidence point ratio, compensation point ratio and anomaly markers of the current cycle, and sends the feedback results to the risk judgment module, the thermal mask generation module and the displacement field generation module, and updates at least one parameter for the next cycle. The parameters include the start threshold and stop threshold for the risk judgment module, the temperature difference threshold and morphological expansion parameter for the thermal mask generation module, and the smoothing weight for the displacement field generation module.

[0079] Anomaly markers are generated by the data output module or the confidence generation module based on the proportion of low confidence points, the proportion of out-of-bounds compensation points, or processing timeouts. The proportion of out-of-bounds compensation points is the percentage of compensation points that exceed the effective range of the image, the range of the working mask, or the range of the target boundary after compensation, out of all compensation points.

[0080] Based on the above embodiments, the feedback adjustment module of the embodiments is described as follows;

[0081] Obtain the processing time, low confidence point ratio, compensation point ratio, and anomaly flags for the current cycle, and update the parameters for the next cycle based on the feedback results; the processing time mentioned above refers to the processing time from when the data enters the cycle buffer to when the unified data packet is encapsulated in the current cycle.

[0082] The percentage of low-confidence points refers to the proportion of compensation points with confidence levels below a set value among all compensation points; the percentage of compensation points refers to the proportion of points that perform compensation among all point cloud points; the anomaly marker refers to the status indicator triggered by an excessively high percentage of low-confidence points, an excessively high percentage of out-of-bounds compensation points, or a processing timeout; while the percentage of out-of-bounds compensation points is used to indicate the proportion of compensation points that fall outside the effective range of the image, the range of the working mask, or the range of the target boundary after compensation among all compensation points.

[0083] In practical implementation, the feedback adjustment module can directly read the risk score, compensation enable status and statistical results from the unified data packet, read the proportion of low confidence points from the confidence generation module, and read the processing time and out-of-bounds statistics from the data output module, and write these results into the parameter area of ​​the next cycle; the parameter area is preferably set as a small-capacity parameter block in shared memory, which is updated after reading and written back for direct use in the next cycle;

[0084] Fixed thresholds and fixed smoothing weights are prone to failure when there are changes in day and night temperature difference, dust concentration, or equipment operating conditions. This implementation method can make targeted adjustments to the start threshold, stop threshold, temperature difference threshold, morphological expansion parameter, and smoothing weight by quantitatively evaluating the processing time, reliability distribution, and out-of-bounds situations.

[0085] To achieve clear closed-loop control of the above adjustments, the system follows the following quantization rules: if the processing time of the current cycle continuously exceeds the preset time threshold, the system automatically increases the absolute magnitude of the start or stop threshold by 0.02 to 0.05 according to the preset quantization step size, such as increasing the start or stop threshold by 0.02 to 0.05, or by decreasing the second visible light image pixels by 1 to 2 pixels according to the fixed pixel step size, thereby reducing the morphological expansion parameter to narrow the range of the working mask, thereby reducing the computational load of the edge processing nodes;

[0086] When the proportion of low-confidence points is higher than the set proportion, it indicates that the current thermal mask or displacement field extraction is severely affected by background interference. The system improves the extraction conditions of the thermal mask by increasing the temperature difference threshold by a preset step size, or increases the smoothing weight to suppress local noise jumps in the displacement field. When the proportion of out-of-bounds compensation points is greater than the preset out-of-bounds proportion threshold, it indicates that the calculation amplitude of the two-dimensional cumulative displacement may have a positive error or drift. At this time, the system reduces the smoothing weight or shrinks the morphological expansion parameter accordingly. Through the above-mentioned structured conditional branch processing rules, the dynamic adaptive balance of the displacement compensation degree is achieved.

[0087] This embodiment can update parameters based on periodic statistical results, avoiding performance degradation caused by long-term use of static parameters, and ultimately solving the problem that existing technologies are difficult to adapt to changes in operating conditions due to the lack of runtime result callbacks.

[0088] The risk assessment module is also used to: calculate pixel changes in the second visible light image sequence by using the frame difference method or optical flow method between adjacent images, extract regions whose changes exceed a preset change threshold as candidate dynamic regions, and obtain a brightness fluctuation factor based on the absolute value of the average gray level difference between the candidate dynamic regions and adjacent images.

[0089] After normalizing the average motion amplitude of the candidate dynamic region in the second visible light image sequence, the proportion of the hot region in the thermal imaging image, and the brightness fluctuation factor, the risk score is obtained by fusing and calculating based on the normalized average motion amplitude, the proportion of the hot region, and the brightness fluctuation factor.

[0090] Among them, the start threshold is greater than the stop threshold; when the risk score is not less than the start threshold, the start compensation enable state is output; when the risk score is not greater than the stop threshold, the stop compensation enable state is output; when the risk score is between the start threshold and the stop threshold, the compensation enable state of the previous cycle is maintained.

[0091] The thermal mask generation module is used to determine the environmental background temperature baseline based on environmental sensing data, or based on the statistics of the background area in the thermal imaging image, or based on the environmental sensing data and the statistics of the background area. The background area is the area in the thermal imaging image that is located in the preset non-target area and / or has a temperature lower than the preset upper limit of the background temperature.

[0092] Based on the above embodiments, the supplementary features in the embodiments are explained as follows:

[0093] The absolute value of the average gray level difference between candidate dynamic regions and adjacent images is obtained. The absolute value of the average gray level difference is normalized according to the preset gray level range to determine the brightness fluctuation factor. The preset gray level range refers to the normalization scale set for the brightness change of the second visible light image sequence, which can be obtained from the brightness statistics under normal operating conditions.

[0094] The brightness fluctuation factor mainly reflects the local brightness fluctuations caused by the boundaries of dust plumes and spilled materials, such as the continuous brightening and darkening of local areas in the image, and the appearance of drifting textures. In the case of dust and semi-transparent targets, relying solely on motion displacement can easily lead to misjudgment of the edges of stable mobile devices and drifting dust. The brightness fluctuation factor can further distinguish the unstable areas in the time domain, thereby improving the sensitivity of risk assessment to semi-transparent dynamic targets.

[0095] This embodiment can supplement the information sources of risk scoring based on grayscale changes, avoid the confusion of target types caused by relying solely on exercise volume, and ultimately improve the pertinence of dynamic risk assessment;

[0096] Acquire background area statistics from environmental sensor data and thermal imaging images, and determine the environmental background temperature baseline based on single or combined sources. Environmental sensor data may include at least ambient temperature, and may also include auxiliary information such as humidity, wind speed, wind direction, and particulate matter concentration. Background area statistics may be the average temperature, median temperature, or other statistical temperatures of the background area. The environmental background temperature baseline serves as a reference line for distinguishing between normal background thermal levels and target thermal anomalies under current operating conditions. When the ambient temperature changes rapidly or the background is affected by sunlight beyond the preset fluctuation range, using only the global temperature of the thermal image as a threshold may easily lead to deviations.

[0097] This implementation allows the background baseline to be determined using environmental sensing data, background area statistics, or both. When determined by both, the data collected by the environmental temperature sensor can be used as the reference center, and the statistical average temperature of the background area can be used as the correction term. The two are weighted and fused by setting corresponding weights. In the specific calculation, the weight of the environmental temperature is set to a preset high confidence weight, for example, 0.7 to 0.9, and the weight of the statistical average temperature of the background area is set to a preset low confidence weight, for example, 0.1 to 0.3. The sum of the two weights is 1. After multiplying them by the corresponding temperature values ​​respectively, the fused baseline is obtained. Alternatively, the deviation of the background area in the thermal image relative to the conventional background can be superimposed on the environmental sensing temperature.

[0098] The specific process involves extracting the statistical temperature of the background region from several historical periods, calculating the time variance of the statistical temperature for each period, selecting periodic data with a time variance less than a preset variance threshold as the interference-free background, and using its average temperature as the normal background temperature. The difference between the statistical temperature of the background region in the current period and the normal background temperature is calculated and added to the current environmental sensing temperature as a deviation to complete the correction. A reference line that dynamically floats with the environment is established, thereby improving the stability of the thermal mask under different environments. This embodiment can achieve adaptive adjustment of thermal region extraction based on a multi-source determination method of the background temperature baseline, avoiding the sensitivity of the temperature threshold to environmental changes, and ultimately improving the reliability of dynamic region screening.

[0099] In engineering applications, this implementation can be deployed in edge servers or industrial processing equipment; when a new processing cycle begins, the multimodal acquisition module writes various types of data into the shared memory cycle buffer; the risk judgment module first reads the second visible light image sequence and thermal imaging image to form a compensation enable state; only when the enable state is enabled will subsequent mask generation, displacement calculation and point cloud compensation be invoked.

[0100] After the cycle processing is completed, the data output module writes the unified data packet into the output buffer for subsequent volume measurement, target identification or pollution source tracing programs to call; the feedback adjustment module updates the parameters for the next cycle based on the statistical results of this cycle; for situations where dynamic material spillage and dust diffusion coexist, this method can make the correspondence between the original point cloud and the reference image more stable, making it easier for subsequent programs to directly use the unified format data for processing.

[0101] In this embodiment, the two-dimensional cumulative displacement field generation, point-level scanning time acquisition, and projection compensation process are further explained. This part of the processing is preferably completed in the same process of the edge processing node: the second visible light image sequence is stored in the image array in memory in chronological order, the working mask is stored in a binary array of the same size, and the lidar point cloud is stored in a point structure array. Each point contains at least spatial coordinates and original accompanying information.

[0102] The displacement field generation module calculates the two-dimensional displacement between adjacent images within the working mask range, and accumulates the displacements at each time moment to form a cumulative displacement field corresponding to several time moments; the time offset acquisition module writes the relative scan time for each point; the projection compensation module reads the scan time offset of each point, retrieves the displacement from the cumulative displacement field at the corresponding time moment, and completes two-dimensional reverse translation and depth-preserving back projection.

[0103] The displacement field generation module is specifically used for:

[0104] Calculate the two-dimensional displacement of adjacent images in the working mask-defined area of ​​the second visible light image sequence; accumulate the two-dimensional displacement of adjacent images according to the order of sampling time of each lidar point in the lidar point cloud from the reference time;

[0105] For each lidar point, displacement interpolation is performed by combining the time offset between adjacent image sampling times of each lidar point to obtain the cumulative displacement field at the corresponding time; and the cumulative displacement field is locally smoothed within the working mask-defined area to obtain the smoothed cumulative displacement field.

[0106] Based on the above embodiments, the displacement field generation module of the embodiments is described as follows;

[0107] The second visible light image sequence and working mask are obtained. Based on the region defined by the working mask, two-dimensional displacement calculation is performed on adjacent images to generate displacement fields at adjacent time points. The region defined by the working mask refers to the region where displacement calculation is allowed, such as a thermal region and its expanded neighborhood. Its function is to focus the calculation on the location where dynamic targets may exist and avoid invalid calculations on a large area of ​​static background.

[0108] Two-dimensional displacement refers to the planar displacement of the same image pixel between two consecutive frames, typically consisting of horizontal and vertical components. In practical implementation, the second visible light image sequence is divided into two parts at the sampling time. and the next sampling time Acquired adjacent images and The corresponding pixel within the working mask From time arrive The two-dimensional displacement between them can be expressed as:

[0109]

[0110] Among them, superscript This represents the matrix transpose, where the two-dimensional displacement is determined by the image's lateral displacement component. and image longitudinal displacement components constitute;

[0111] The displacement is calculated based on the movement and drifting trend of the dynamic target contour within the working mask area. Calculating the entire image increases the amount of invalid processing and is subject to interference from static background textures. This implementation uses working mask constraints to concentrate the displacement estimation around the thermal dynamic area, thereby improving the effective response to weakly textured dynamic targets such as dust and material edges. This embodiment can achieve range reduction of displacement estimation based on the working mask, avoiding interference from static backgrounds on dynamic image analysis, and ultimately improving the targeting of the displacement field.

[0112] The displacement fields at adjacent time points are obtained. Based on the order of sampling times from the reference time to each lidar point, the adjacent displacements are accumulated, and the target time is interpolated to generate a two-dimensional cumulative displacement field for the corresponding time. The cumulative displacement field refers to the total displacement result from the reference time to a certain target time. The target time refers to the time corresponding to the scanning time offset of a certain lidar point. Interpolation is used to solve the problem that the sampling time of the lidar point may not completely coincide with the actual sampling time of the second visible light image.

[0113] In practical implementation, if the target time lies between several adjacent image sampling times, the adjacent displacements are first accumulated sequentially to obtain the distance from the reference time to the image sampling time. The cumulative displacement is calculated using the following formula:

[0114]

[0115] Among them, the accumulation operation Accumulate from the initial image sequence index corresponding to the reference time up to the sampling time. The previous index, For the first step in the accumulation process Two-dimensional displacement of adjacent images;

[0116] If the scan time offset of a certain point Falling into adjacent sampling time and Between these points, interpolation can be used to obtain the corresponding cumulative displacement. The specific formula is as follows:

[0117]

[0118] in, Indicates immediately following the sampling time The two-dimensional displacement between subsequent adjacent image pairs, and the relative position coefficient of the target time between two sampling times. The value is between 0 and 1, which represents the proportion of the sampling time of that point in the time interval of the image segment. The specific calculation method is to subtract the difference between the sampling time of the previous image from the target time, and divide by the time span between the sampling time of the next image and the sampling time of the previous image. Through this coefficient, the discrete image displacement can be converted into a point-level time-readable continuous cumulative displacement.

[0119] To suppress displacement jumps caused by local texture loss or noise disturbance, the cumulative displacement field can be locally smoothed within the working mask. Specifically, when smoothing, a neighborhood window of a preset size, such as 3×3 or 5×5 pixels, is used. For the target center pixel, the effective pixels in its neighborhood that belong to the working mask are traversed, their cumulative displacement values ​​are read, and they are multiplied by the Gaussian spatial distribution coefficient, which increases with the distance from the center pixel. The sum of all products is divided by the sum of the Gaussian spatial distribution coefficients of the effective pixels to obtain the basic displacement of the spatial neighborhood.

[0120] The Gaussian spatial distribution coefficient G(x,y) between the target center pixel (x0,y0) and its neighboring effective pixels (x,y) is calculated using the following formula:

[0121]

[0122] Where σ is a preset standard deviation parameter used to control the range of spatial smoothing;

[0123] A smoothing weight is introduced, which is multiplied by the basic displacement of the spatial neighborhood. The product of the difference between the smoothing weight and 1 and the original displacement of the center pixel is added, and the output is the smoothed cumulative displacement value of the center pixel. The smoothing weight can be set to a preset value, such as a value between 0.5 and 0.9. The larger the weight, the more emphasis is placed on neighborhood consistency, and the smaller the weight, the more the original local changes are preserved. In the case of dust and spilled materials, if only the instantaneous displacement of two adjacent frames is used, unstable displacement calculation values ​​are easily generated due to local texture breaks.

[0124] This implementation method uses cumulative interpolation to enable each lidar point to read a displacement value closer to its sampling time. Combined with the above-mentioned rigorous local smoothing logic, it reduces neighborhood abrupt changes. This embodiment can generate a two-dimensional cumulative displacement field that can be called point by point based on the cumulative, interpolated and smoothing processing, avoiding compensation errors caused by the inconsistency between discrete image sampling and point-level scanning time, and ultimately improving the accuracy of compensation timing matching.

[0125] The time offset acquisition module is used to obtain the scanning time offset in one of the following ways: when the raw LiDAR data contains point-level sampling times, the scanning time offset is determined based on the difference between the sampling times of each point and the reference time; when the raw LiDAR data does not contain point-level sampling times but contains scanning angle information, the scanning time offset is determined based on the scanning angle, starting scanning angle and scanning angular velocity of each point in the raw LiDAR data or the preset parameters of the device, and the scanning time offset is limited to the range of a single frame scanning cycle.

[0126] Based on the above embodiments, the time offset acquisition module of the embodiments is described as follows;

[0127] The raw data from the lidar is acquired. Based on whether the raw data contains point-level sampling time, the scanning time offset is calculated for each point to generate a scanning time offset. The scanning time offset refers to the time difference of a lidar point relative to the reference time, which is used to determine which target time corresponding to the cumulative displacement field should be read for that point. The scanning time offset is preferably written into the same memory structure as the point cloud array in a one-to-one correspondence, so that the projection compensation module can directly access it point by point.

[0128] When the raw LiDAR data includes point-level sampling times, the actual sampling times can be used. Subtract reference time Calculate the method of the first Scan time offset of each point This difference represents the temporal order in which the point was scanned relative to the reference image.

[0129] When the raw data does not contain point-level sampling time but contains scan angle information, the first point can be estimated based on the scan angle relationship. Scan time offset of each point The formula is:

[0130]

[0131] in, Indicates the first The scanning angle of each point Indicates the starting scan angle of this frame. This represents the scanning angular velocity; this ratio reflects the scanning time taken to rotate from the starting position to the corresponding angle at the current point; after calculation, it needs to be... Limit it to a single frame scan cycle to avoid time overflow caused by angle crossing cycles;

[0132] Compared to processing methods that treat a frame of point cloud as data at the same moment, this implementation explicitly writes out the scanning sequence of each point, so that subsequent compensation is no longer based on the average of the whole frame, but is implemented to read the corresponding displacement point by point. This embodiment can realize point-by-point temporal positioning of the point cloud based on the scanning time offset, avoiding the dynamic target stretching error caused by the assumption of the whole frame being at the same moment, and ultimately solving the problem of insufficient utilization of the point-by-point scanning characteristics of lidar in the existing technology.

[0133] The projection compensation module is specifically used for:

[0134] Transform the lidar points to the camera coordinate system based on the lidar-to-camera extrinsic parameters;

[0135] The converted LiDAR points are projected onto the image plane based on the camera's intrinsic parameters.

[0136] For lidar points that do not fall into the working mask, retain their original coordinates. For lidar points that fall into the working mask, read the displacement in the two-dimensional cumulative displacement field at the corresponding time according to the corresponding scanning time offset. Then, translate the pixel coordinates of the lidar points that fall into the working mask in the image plane in the opposite direction of the read displacement to compensate to the corresponding position at the reference time.

[0137] The compensated pixel coordinates are back-projected while keeping the original depth unchanged to obtain the compensated 3D point coordinates; the compensated 3D point coordinates are then transformed back into the lidar coordinate system to form a compensated point cloud.

[0138] The projection compensation module is also used for:

[0139] When the radial velocity of the target is greater than a preset velocity threshold or the projected displacement of the same target in two consecutive frames of the second visible light image sequence is greater than a preset displacement threshold, the depth before compensation is corrected according to the depth correction amount before backprojection.

[0140] The radial velocity of the target is determined by lidar velocity measurement values, external velocity measurement data, or by the change in spatial distance between the centroids of the same cluster of targets extracted after performing clustering processing on two consecutive frames of point clouds at adjacent time points.

[0141] Based on the above embodiments, the projection compensation module of the embodiments is described as follows;

[0142] Acquire the lidar point cloud, lidar to camera extrinsic parameters, camera intrinsic parameters, working mask, and scanning time offset. Based on the projection relationship and the two-dimensional cumulative displacement field at the corresponding time, perform reverse spatiotemporal translation on the points falling within the working mask to generate a compensated point cloud.

[0143] The aforementioned extrinsic parameters of the lidar to the camera are used to complete coordinate system transformation, while the intrinsic parameters of the camera are used to complete pixel projection and back projection; time inverse compensation refers to translating the target pixel from its actual scanning time along the opposite direction of displacement to the image plane position corresponding to the reference time; depth-preserving back projection refers to keeping the camera depth unchanged before compensation during back projection, and only correcting the image plane position;

[0144] In the specific implementation, a rotation matrix representing the attitude relationship between the laser radar and the camera is first used. and the translation vector characterizing the spatial offset of the laser radar to the camera. Points in the lidar coordinate system Transform to camera coordinate system to obtain coordinates The coordinates in this camera coordinate system are represented as follows:

[0145]

[0146] in, , , These are the coordinate components along the X, Y, and Z axes of the camera coordinate system, respectively.

[0147] Then use the camera intrinsic parameter matrix Projecting it onto the image plane yields the pixel position of that point in the image plane. , here For points that do not fall within the working mask, retain their original coordinates.

[0148] For points falling within the working mask, their scan time offset is used as the basis for calculation. Read the two-dimensional cumulative displacement at the corresponding time point This displacement consists of a lateral displacement component. and longitudinal displacement components The composition; the physical meaning of this displacement is the cumulative movement of the target on the image plane from the reference time to the actual sampling time at that point; after translating this point in the opposite direction of the read displacement to the reference time, the corresponding pixel coordinates are:

[0149]

[0150] This represents the pixel coordinates of the image plane translated in the opposite direction of the read displacement; subsequently, while maintaining the depth before compensation... Under the condition of invariance, the compensated camera coordinates are obtained by back projection:

[0151]

[0152] in, Camera intrinsic parameter matrix The inverse matrix;

[0153] By rotating from the camera coordinate system back to the lidar coordinate system Peace and migration relationship Convert them back to the lidar coordinate system to obtain the compensated lidar point coordinates. After processing all points, a compensated point cloud is formed. In this embodiment, the three-dimensional displacement estimation is converted into a two-dimensional displacement reading operation within the working mask by inverse translation of the image plane and depth-preserving back projection, and the corresponding dynamic target compensated point cloud is output. This embodiment can achieve point-by-point compensation based on projection, inverse translation and back projection, avoiding the computational burden caused by complex three-dimensional estimation, and ultimately improving the correspondence between the point cloud and the reference image.

[0154] The radial velocity of the target or the projected displacement of two consecutive frames of the same target is obtained. The depth before compensation is corrected according to a preset velocity threshold or a preset displacement threshold before backprojection. The radial velocity of the target can be obtained from the laser radar velocity value, the change in point cloud distance at adjacent time points, or external velocity data. The preset velocity threshold is used to identify targets that are approaching or moving away at high speed, and the preset displacement threshold is used to identify targets that are moving fast in the image. The depth correction amount is determined by the radial correction parameter and the magnitude of the read displacement.

[0155] In practical implementation, when the absolute value of the target's radial velocity is greater than a preset velocity threshold, or when the projected displacement of the same target in two consecutive frames is greater than a preset displacement threshold, the depth can be corrected by combining the radial motion direction of the target relative to the sensor, resulting in the corrected depth. The specific formula is as follows:

[0156]

[0157] in, To obtain the target's true radial velocity using lidar or external velocimetry equipment, and to set the target to approach the system in a positive direction. The value is negative when it is far away, and positive when it is far away; This represents the scan time offset corresponding to that point. Since the compensation logic reverses the sampling coordinates from later times to earlier times, when the target approaches the system in the forward direction, it is closer at the sampling time than at the reference time. If positive, the system performs compensation with increased depth;

[0158] Conversely, when the target is far from the system, the absolute distance of the target in three-dimensional space at the reference time is less than the distance at the sampling time. If the value is negative, compensation for depth reduction is performed; by introducing a real physical time step, the absolute position of the target in three-dimensional space at the historical reference time is rigorously restored, avoiding projection view distortion errors caused by using two-dimensional pixel scalars;

[0159] When the target has radial motion, simply keeping the original depth unchanged cannot fully describe the real position change. By introducing a motion direction constraint factor, a depth adjustment that is strictly linked to the displacement magnitude and motion direction is added before back projection.

[0160] Conventional depth-preserving methods can lead to depth lag or incorrect compensation direction when dealing with high-speed radial targets. This implementation method sets trigger conditions and motion direction factors to enable depth correction and match the true direction only when the velocity or displacement increases abnormally, thereby avoiding over-processing of general working conditions and erroneous physical attenuation. This embodiment can enhance compensation for special high-speed targets based on depth correction under velocity or displacement conditions, avoiding the insufficient applicability of the simple depth-preserving strategy in the case of rapid radial motion, and ultimately improving the robustness of the compensation results.

[0161] During engineering deployment, the image array, displacement field array, and point cloud array in this embodiment can all be stored in the memory container of the same processing cycle; after the displacement field is generated, it is written into a two-dimensional floating-point array; the time offset result is written into the additional field of the point cloud point structure; the projection compensation module reads the displacement and time offset point by point, and the compensated point cloud is written into a new point cloud buffer; after processing, the compensated point cloud buffer can be directly referenced by a unified data package for subsequent volume measurement or recognition program calls; for working conditions with dynamic material throwing, dust dispersion, and equipment exhaust, this method is beneficial to reduce point cloud contour stretching and tilting phenomena;

[0162] In this embodiment, the generation of compensation point reliability and the setting of acquisition rhythm are further explained. Through the above settings, a quantitative basis for the reliability of compensation points is provided for subsequent programs, and the number of images in the second visible light image sequence within one lidar scanning cycle is guaranteed to meet the generation conditions of the two-dimensional cumulative displacement field, so that the two-dimensional cumulative displacement field has availability. The reliability calculation is preferably performed immediately after the compensation point cloud is generated, and the relevant intermediate results are still stored in the current cycle memory area. Points with a reliability lower than the set value are marked with a low reliability bit in the point structure and output together with the unified data packet.

[0163] The system also includes a credibility generation module, which generates motion consistency values, thermal region consistency values, and boundary stability values ​​respectively, and determines the credibility of each compensation point based on the motion consistency values, thermal region consistency values, and boundary stability values.

[0164] Among them, the motion consistency value is generated based on the local smoothness of the two-dimensional cumulative displacement field, the thermal region consistency value is generated based on the spatial relationship between the compensation point and the thermal mask, and the boundary stability value is generated based on the distance of the compensation point from the boundary information extracted from the thermal imaging image or the second visible light image sequence or the boundary crossing situation; the boundary information is obtained by Canny edge detection operator or temperature gradient threshold segmentation method.

[0165] The motion consistency value, thermal region consistency value, boundary stability value, and confidence level are all normalized values.

[0166] The reliability score is generated by fusing motion consistency values, thermal region consistency values, and boundary stability values.

[0167] When the compensation point is located within the thermal mask, the thermal region consistency value is a first preset value; when the compensation point is located within the working mask but outside the thermal mask, the thermal region consistency value is a second preset value; wherein, the first preset value is greater than the second preset value;

[0168] The data output module is also used to mark compensation points with a confidence level lower than a preset confidence level threshold as low confidence points and output them along with the unified data packet.

[0169] Based on the above embodiments, the credibility generation module of the embodiments is described as follows;

[0170] The two-dimensional cumulative displacement field, thermal mask and target boundary information are obtained. Based on the local smoothness, spatial attribution relationship and boundary distance relationship, motion consistency value, thermal region consistency value and boundary stability value are generated respectively, and the credibility of each compensation point is determined accordingly.

[0171] The motion consistency value is used to reflect whether the displacement field near the location of the compensation point is stable and continuous, such as the displacement directions of adjacent pixels being close and the magnitude changes being gradual; the thermal region consistency value is used to reflect whether the compensation point is located within the thermal mask coverage area; the boundary stability value is used to reflect the spatial relationship between the compensation point and the target boundary extracted from the thermal imaging image or the second visible light image sequence, such as being located inside the boundary, close to the boundary, or beyond the boundary; these three values ​​and the final confidence level are all represented by normalized numerical values ​​to facilitate unified comparison and threshold judgment;

[0172] In specific implementation, the first Final credibility of each compensation point It can be obtained by weighted summation according to preset weights, and the specific formula is as follows:

[0173]

[0174] in, The motion consistency value is used to characterize the first... The degree of local smoothness of the displacement field in the neighborhood of each compensation point This is the thermal region consistency value, used to characterize the first... The degree of spatial matching between each compensation point and the hot mask. This is the boundary stability value, used to characterize the first... The distance between each compensation point and the target boundary, or any boundary crossing; and , , The preset weights, which are used to balance the impact of the three types of credibility factors on the final credibility, are all non-negative and sum to 1;

[0175] In practical implementation, the motion consistency value is generated based on the intensity of local changes in the cumulative displacement field through quantization rules. The system calculates the variances of the lateral and longitudinal displacement components of each displacement vector within the current compensation point and its preset neighborhood, such as a 3×3 pixel window. These variances are used to replace the local direction and amplitude variances in polar coordinates. The normalized weighted sum of these two variances is mapped to a confidence interval of 0 to 1 using a negative exponential function. When calculating the normalized weighted sum, the normalized weighted sum, which is the independent variable of the negative exponential function, is calculated using the following formula. :

[0176]

[0177] in, and These are the variances of the lateral and longitudinal displacement components within the preset neighborhood, respectively. and These are the preset maximum allowable lateral variance and maximum longitudinal variance of the system. and The empirical weights are used to obtain the normalized proportional terms of the two, which are then combined and used as independent variables input into the negative exponential function.

[0178] To avoid uncertainties in the mapping process, the motion consistency value is calculated using the following formula:

[0179]

[0180] in, A preset attenuation coefficient that controls the steepness of the mapping curve, for example, The value range is 2.0 to 5.0;

[0181] The smoother the changes of each displacement component within this neighborhood, i.e., the smaller the variance, the better the normalized weighted sum. The closer a value is to 0, the higher the calculated motion consistency value. The closer it is to 1, the better; if there is a significant jump, that is, the larger the variance, the value will decay exponentially to a lower motion consistency value.

[0182] Realistic dynamic targets exhibit continuity in local areas, while displacement results caused by noise, occlusion, or texture loss present localized scattered characteristics; the thermal region consistency value can be directly assigned based on the spatial relationship between the compensation point and the thermal mask: when the compensation point is located within the thermal mask, a preset high confidence value is set;

[0183] When the compensation point is located inside the working mask but outside the hot mask, a preset low confidence value is set, and the preset high confidence value is greater than the preset low confidence value. Both the high confidence value and the low confidence value can be set to preset values ​​between 0 and 1 based on field experience. For example, the high confidence value can be set to around 0.8 and the low confidence value can be set to around 0.3.

[0184] Points within the thermal mask correspond to dynamic thermal targets, while points located within the working mask but outside the thermal mask correspond to boundary transition zones or noise regions. Boundary stability values ​​can be determined in a stepped manner based on the distance from the compensation point to the target boundary or whether the boundary has been crossed.

[0185] When the compensation point is located inside the target boundary and the spatial distance to the nearest boundary is greater than the preset safe pixel distance, the system directly assigns the boundary stability value to 1; when the compensation point is also located inside the boundary but its distance to the boundary is within the safe pixel distance, the value will be set to a decrease value between 0 and 1 according to the linear decay rule that is proportional to the boundary distance, that is, the decrease value is strictly equal to the quotient obtained by dividing the actual pixel distance from the current compensation point to the nearest boundary by the safe pixel distance; once it is detected that the compensation point crosses the boundary or exceeds the effective image contour range after compensation, the boundary stability value is forcibly blocked and assigned a value of 0.

[0186] By using distance-based segmented assignment, the constraints on out-of-bounds compensation points are enhanced; conventional non-discriminatory processing methods will use all compensation points equally in subsequent fusion and calculation, which can easily lead to erroneous compensation points being directly included in subsequent calculations.

[0187] This implementation method uses three aspects—motion, thermal region, and boundary—to perform quantitative evaluation, enabling subsequent procedures to reduce the weight of low-confidence points or remove them directly. This implementation method can achieve quality labeling of compensation results based on multi-factor confidence assessment, avoiding the mixing of erroneous compensation points and high-reliability compensation points, and ultimately solving the problem of decreased post-processing accuracy caused by the lack of point-level quality differentiation in existing technologies.

[0188] The credibility of each compensation point is obtained. Based on a set value, a threshold judgment is made on the credibility. Compensation points below the set value are marked as low-credibility points and output along with the unified data packet. The set value refers to the judgment threshold used to distinguish between high-credibility and low-credibility compensation points, which can be determined by on-site verification, for example, it can be set between 0.35 and 0.55. The low-credibility point marking is preferably written into the point cloud data structure in the form of point-level status bits so that subsequent identification, volume measurement and pollution source tracing programs can directly use it when reading the unified data packet. This embodiment can realize the hierarchical output of compensation points based on the credibility threshold, avoid the subsequent program to repeatedly calculate the compensation reliability, and ultimately improve the availability of the unified data packet.

[0189] The multimodal acquisition module acquires the first frame of the first visible light image and the first frame of the thermal imaging image at the reference time as an alignment reference, and continuously acquires multiple frames of images within a single frame scanning cycle of the lidar point cloud to form a second visible light image sequence.

[0190] Based on the above embodiments, the multimodal acquisition module of the embodiments is described as follows;

[0191] Acquire a first visible light image, a thermal imaging image, and a second visible light image sequence. Based on the lidar scanning frequency, set the sampling frame rate of the second visible light image sequence to generate a periodic image sequence that meets the compensation requirements. When the lidar scanning frequency is 10 Hz, it means that the single frame scanning period is a fixed scanning window.

[0192] The second visible light image sequence is acquired within the scanning window at a frame rate of 50 Hz to 90 Hz, so that the number of images obtained in a single frame of lidar scanning cycle is no less than 5 frames. The significance of having no less than 5 images is that it can at least cover the vicinity of the reference time and multiple intermediate times within the scanning window, thereby providing sufficient time sampling points for subsequent cumulative displacement and interpolation.

[0193] In the specific implementation, one frame each of the first visible light image and the thermal imaging image is acquired at the reference time and written into the periodic buffer as a unified alignment reference; the second visible light image sequence is acquired continuously within the same scanning window to form a time-ordered image array; if the image frame rate is lower than the preset low-frequency threshold, the rapid scattering trajectory and dust diffusion edge will have pixel crossings exceeding the preset tolerance between adjacent frames, resulting in an increase in displacement estimation error; if the image frame rate is higher than the preset high-frequency threshold, it will increase the data processing volume of the edge processing nodes;

[0194] Using a sampling range of 50 Hz to 90 Hz achieves a suitable balance between temporal resolution and system computational resource consumption. This embodiment specifically sets up a second visible light image sequence as a dynamic observation channel for compensation, so that point-by-point time rollback has continuous image basis, rather than relying on single-frame snapshots for coarse estimation. This embodiment can provide sufficient temporal information for the two-dimensional cumulative displacement field based on the acquisition method of combining reference images and sequence images, avoiding the discontinuity of the displacement field caused by excessively sparse image sampling, and ultimately ensuring that the compensation processing has stable input conditions.

[0195] In actual deployment, this embodiment can connect the multimodal acquisition module to the edge processing device, which receives a high-resolution first visible light image, a thermal imaging image, and multiple second visible light images within the same period at the beginning of each scanning cycle.

[0196] After compensation is completed, the credibility generation module writes the credibility and low credibility marker of each compensation point into the point cloud structure. The data output module encapsulates the point cloud, image, working mask, risk score, and credibility information into a unified data packet and writes it into the output buffer. Subsequent identification programs, volume calculation programs, or pollution source tracing programs can read data from this buffer and directly decide whether to use the corresponding compensation point with reduced weight based on the low credibility marker. In this way, a complete processing process from data collection and compensation to quality labeling and subsequent calls can be formed in the actual system.

[0197] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. The Eagle Eye Multimodal Data Preprocessing System, characterized in that, include: The multimodal acquisition module acquires a first visible light image, a thermal imaging image, a lidar point cloud, a second visible light image sequence, and environmental sensing data at a reference time, wherein the lidar point cloud contains multiple lidar points. The risk assessment module calculates a risk score based on the second visible light image sequence and the thermal imaging image, and outputs the compensation enable status. The risk assessment module is also used for: Candidate dynamic regions are extracted from the second visible light image sequence, and a brightness fluctuation factor is obtained based on the absolute value of the average gray level difference between the candidate dynamic regions and adjacent images. After normalizing the average motion amplitude of the candidate dynamic region in the second visible light image sequence, the proportion of the hot region in the thermal imaging image, and the brightness fluctuation factor, a risk score is obtained by fusing and calculating based on the normalized average motion amplitude, the proportion of the hot region, and the brightness fluctuation factor. Wherein, the start threshold is greater than the stop threshold; when the risk score is not less than the start threshold, the start compensation enable state is output; when the risk score is not greater than the stop threshold, the stop compensation enable state is output; when the risk score is between the start threshold and the stop threshold, the compensation enable state of the previous cycle is maintained. The thermal mask generation module generates a thermal mask based on the environmental sensing data and the thermal imaging image, and determines the working mask based on the thermal mask. The displacement field generation module generates a two-dimensional cumulative displacement field within the working mask, based on the second visible light image sequence and the time difference between the reference time and the sampling time of each of the plurality of lidar points. The time offset acquisition module acquires the scanning time offset of each lidar point in the lidar point cloud relative to the reference time. The projection compensation module projects the lidar point cloud onto the image plane. For lidar points falling within the working mask, it performs spatiotemporal displacement compensation and back projection by combining the corresponding scanning time offset with the two-dimensional cumulative displacement field to obtain the compensated point cloud. The data output module selects to output the lidar point cloud or the compensated point cloud according to the compensation enable state, and outputs a unified data packet containing the corresponding reference time and timestamp. It also includes a credibility generation module, which is used to generate motion consistency value, thermal region consistency value and boundary stability value respectively, and determine the credibility of each compensation point based on the motion consistency value, thermal region consistency value and boundary stability value; The motion consistency value is generated based on the local smoothness of the two-dimensional cumulative displacement field, the thermal region consistency value is generated based on the spatial relationship between the compensation point and the thermal mask, and the boundary stability value is generated based on the distance of the compensation point from the boundary information extracted from the thermal imaging image or the second visible light image sequence, or the boundary crossing situation. The motion consistency value, thermal region consistency value, boundary stability value, and confidence level are all normalized values. The credibility is generated by fusing the motion consistency value, the thermal region consistency value, and the boundary stability value. When the compensation point is located within the thermal mask, the thermal region consistency value is a first preset value; when the compensation point is located within the working mask and outside the thermal mask, the thermal region consistency value is a second preset value; wherein, the first preset value is greater than the second preset value. The data output module is also used to mark compensation points with a confidence level lower than a preset confidence level threshold as low confidence points and output them along with the unified data packet.

2. The Eagle Eye multimodal data preprocessing system according to claim 1, characterized in that, It also includes a feedback adjustment module, which is used to receive the processing time, low confidence point ratio, compensation point ratio and anomaly marker of the current cycle, and send the feedback results to the risk judgment module, the thermal mask generation module and the displacement field generation module, and update at least one parameter for the next cycle. The parameters include the start threshold and stop threshold for the risk judgment module, the temperature difference threshold and morphological expansion parameter for the thermal mask generation module, and the smoothing weight for the displacement field generation module. The anomaly markers are generated by the data output module or the confidence generation module based on the proportion of low confidence points, the proportion of out-of-bounds compensation points, or processing timeouts. The proportion of out-of-bounds compensation points is the percentage of compensation points that exceed the effective range of the image, the range of the working mask, or the range of the target boundary after compensation, out of all compensation points.

3. The Eagle Eye multimodal data preprocessing system according to claim 1, characterized in that, The thermal mask generation module is used to determine the environmental background temperature baseline based on environmental sensing data, or based on the statistics of the background area in the thermal imaging image, or based on the environmental sensing data and the statistics of the background area. The background area is the area in the thermal imaging image that is located within a preset non-target area and / or has a temperature lower than a preset upper limit of background temperature.

4. The Eagle Eye multimodal data preprocessing system according to claim 1, characterized in that, The displacement field generation module is specifically used for: Two-dimensional displacements are calculated for adjacent images of the second visible light image sequence within the working mask-defined region; the two-dimensional displacements of the adjacent images are accumulated according to the order from the reference time to the sampling time of each lidar point in the lidar point cloud; for each lidar point, displacement interpolation is performed based on the time offset between the sampling times of each lidar point in adjacent images to obtain the cumulative displacement field at the corresponding time; and the cumulative displacement field is locally smoothed within the working mask-defined region to obtain the smoothed cumulative displacement field.

5. The Eagle Eye multimodal data preprocessing system according to claim 1, characterized in that, The time offset acquisition module is used to acquire the scanning time offset in one of the following ways: when the original LiDAR data contains point-level sampling times, the scanning time offset is determined based on the difference between the sampling times of each point and the reference time; when the original LiDAR data does not contain point-level sampling times but contains scanning angle information, the scanning time offset is determined based on the scanning angle, starting scanning angle and scanning angular velocity of each point in the original LiDAR data or the preset parameters of the device, and the scanning time offset is limited to the range of a single frame scanning cycle.

6. The Eagle Eye multimodal data preprocessing system according to claim 1, characterized in that, The projection compensation module is specifically used for: Transform the lidar points to the camera coordinate system based on the lidar-to-camera extrinsic parameters; The converted LiDAR points are projected onto the image plane based on the camera's intrinsic parameters. For lidar points that do not fall into the working mask, retain their original coordinates. For lidar points that fall into the working mask, read the displacement in the two-dimensional cumulative displacement field at the corresponding time according to the corresponding scanning time offset. Then, translate the pixel coordinates of the lidar points that fall into the working mask in the image plane in the opposite direction of the read displacement to compensate to the corresponding position at the reference time. The compensated pixel coordinates are back-projected while keeping the original depth unchanged to obtain the compensated 3D point coordinates. The compensated 3D point coordinates are converted back to the lidar coordinate system to form a compensated point cloud.

7. The Eagle Eye multimodal data preprocessing system according to claim 6, characterized in that, The projection compensation module is also used for: When the radial velocity of the target is greater than a preset velocity threshold or the projected displacement of the same target in two consecutive frames of the second visible light image sequence is greater than a preset displacement threshold, the depth before compensation is corrected according to the depth correction amount before backprojection. The radial velocity of the target is determined by lidar velocity measurement values, external velocity measurement data, or by the change in spatial distance between adjacent time points of the centroid of the same cluster extracted after performing clustering processing on two consecutive frames of point clouds.

8. The Eagle Eye multimodal data preprocessing system according to claim 1, characterized in that, The multimodal acquisition module acquires the first frame of the first visible light image and the first frame of the thermal imaging image at a reference time as an alignment reference, and continuously acquires multiple frames of images within a single frame scanning cycle of the lidar point cloud to form the second visible light image sequence.

Images