Laser target precision recognition method based on multi-modal fusion perception

Abstract

The application relates to the technical field of target identification, in particular to a laser destruction target accurate identification method based on multi-modal fusion perception, which comprises the following steps: collecting visible light images and infrared images of a target area in real time; receiving a target tracking frame framed by an upper computer, and defining the geometric center of the target tracking frame as an initial target position; obtaining joint saliency of each pixel point under a laser non-firing state; screening peripheral edge points to extract displacement vectors of the peripheral edge points pointing to the initial target position; obtaining a laser contact high-heat point under a laser firing state; deducing cumulative propagation costs of each pixel point; determining a mask truncation threshold, screening available edge points from the peripheral edge points, and real-time inferring an actual target position to control a laser optical axis to be aligned and perform laser destruction. The application effectively overcomes the technical defects of target tracking failure under the double interference of smoke shielding and thermal deformation, and improves the operation accuracy and reliability of high-energy laser point destruction on a target.

Description

Technical Field

[0001] This application relates to the field of target recognition technology, specifically to a method for accurate identification of laser destruction targets based on multimodal fusion perception. Background Technology

[0002] In automated laser destruction of electronic components, the continuous injection of extremely high heat by high-energy lasers can cause thermal expansion, warping, or relative displacement of the circuit board substrate. This necessitates a vision system that tracks the target in real time and guides the laser axis for dynamic correction, ensuring the laser axis is aligned with the deforming target center. However, the instant the high-energy laser contacts the target surface, the target rapidly vaporizes and burns, producing extremely bright flares and a large amount of outward-spreading dense smoke. This completely blinds the vision sensor in the target center area. In this situation, it is necessary to rely on the unobstructed peripheral edge features to inversely deduce the target center displacement. Therefore, accurately defining the masking layer obstructing the smoke spread becomes crucial.

[0003] When vaporized smoke spreads disorderly on the surface of a circuit board, it encounters the obstruction of chips or metal pins with physical height, resulting in an extremely irregular and asymmetrical shape of the actual outward diffusion boundary. In addition, two-dimensional text silkscreen or ink on the surface of the circuit board can easily interfere with the extraction of the boundary of the real physical obstruction. Existing technologies often cannot eliminate the interference of two-dimensional silkscreen or ink, and generally set a static digital mask with a fixed radius in the center of the overexposed area in an attempt to block and eliminate smoke interference. This method cannot fit the irregular diffusion boundary generated after the fluid encounters obstruction. This inevitably leads to the mask being set too large and incorrectly cutting off the originally usable edge features, or the mask being set too small and leaving severely distorted smoke interference features in the tracking field of view, resulting in the extraction of incorrect outer edge features. This causes the retrieved target coordinates to jump and deviate drastically, leading to serious consequences such as laser landing point deviation and incomplete destruction of the core chip. Summary of the Invention

[0004] To address the aforementioned technical challenges, a precise identification method for laser destruction targets based on multimodal fusion perception is provided.

[0005] The solution to the technical problem presented in this application is to provide a method for accurate identification of laser destruction targets based on multimodal fusion sensing, comprising the following steps: The system acquires multimodal images of the target area in real time, including visible light images and infrared images, and establishes a projection mapping relationship between the two modal images; it receives the target tracking box selected by the host computer and defines its geometric center as the initial position of the target. In the state where the laser is not fired, the gradient features of pixels in the two modal images are fused to obtain the joint saliency of each pixel; and based on the tracking box, the peripheral edge points in the visible light image are screened to extract the displacement vector of each peripheral edge point pointing to the initial position of the target. In the laser firing state, the overexposed area is extracted based on the pixel value in the infrared image, and its geometric centroid is projected onto the visible light image through the projection mapping relationship. The projection position is defined as the laser contact hot spot. Taking the laser contact hot spot as the starting wavefront node and the joint saliency as the spatial constraint for wavefront expansion, the cumulative propagation cost of each pixel is derived using the wavefront propagation algorithm. The truncation search area is defined by the span of the boundary of the tracking box from the laser contact hot spot. The mask truncation threshold is determined by the cumulative propagation cost within the truncation search area. Based on this, usable edge points are selected from the outer edge points. The actual position of the target is inferred in real time based on the position coordinates and displacement vectors of all usable edge points in the visible light image, so as to control the laser optical axis alignment and perform laser destruction.

[0006] Preferably, obtaining the joint saliency of each pixel includes: extracting the gradient magnitude of each pixel in the visible light image and the infrared image respectively, and performing normalization processing; using the projection mapping relationship, calculating the product of the normalized gradient magnitude between each pixel in the infrared image and the pixel at its projection position in the visible light image, as the joint saliency of each pixel.

[0007] Preferably, the step of filtering peripheral edge points in a visible light image based on the tracking box includes: in the state where the laser is not fired, defining the area outside the tracking box in the visible light image as the peripheral area, and filtering pixels in the peripheral area whose normalized gradient magnitude is greater than a preset threshold, defining them as peripheral edge points.

[0008] Preferably, the process of obtaining the overexposed area is as follows: under laser firing state, all pixels in the infrared image are traversed in real time, pixels with pixel values ​​greater than a preset thermal saturation threshold are marked, and connected component analysis algorithm is used to aggregate and extract connected components from all adjacent marked pixels, and the connected component with the largest area is selected as the overexposed area.

[0009] Preferably, the derivation process of the cumulative propagation cost is as follows: The cumulative propagation cost of the pixel corresponding to the laser contact hotspot in the visible light image is set to 0; the cumulative propagation cost of the remaining pixels is set to infinity; taking the laser contact hotspot as the starting wavefront node, the remaining pixels in the 8-neighborhood of the current wavefront node, excluding those already selected as wavefront nodes, are defined as extension nodes. Based on the joint saliency of each extension node and its distance from the current wavefront node, calculate the incremental expansion cost required for single-step wavefront propagation from the current wavefront node to each extension node. Calculate the sum of the cumulative propagation cost and the extension cost increment of the current wavefront node. If the value is less than the cumulative propagation cost currently recorded by the extension node, then the cumulative propagation cost of the extension node is overwritten and updated to the sum; otherwise, it remains unchanged. Among all pixels that have had their cumulative propagation cost updated and have not been selected as wavefront nodes, the pixel with the smallest cumulative propagation cost is selected as the new current wavefront node. The above update operation is repeated until the traversal is complete, and the cumulative propagation cost of each pixel in the visible light image is obtained.

[0010] Preferably, the calculation process of the expansion cost increment is as follows: negatively map the joint saliency of each extension node as the expansion step size weight, calculate the pixel distance between the current wavefront node and each extension node, and use the ratio of this distance to the expansion step size weight of the extension node as the expansion cost increment.

[0011] Preferably, the process of obtaining the truncation search region is as follows: extract all vertices of the tracking box, calculate the maximum pixel distance between all vertices and the laser contact hotspot, and multiply it by a preset boundary shrinkage coefficient as the truncation search radius; take the laser contact hotspot in the visible light image as the center, and take the circular area covered by the truncation search radius as the truncation search region.

[0012] Preferably, the mask truncation threshold is the maximum cumulative propagation cost of all pixels located within the truncation search area.

[0013] Preferably, the screening process for available edge points is as follows: for all peripheral edge points in the visible light image, select peripheral edge points whose cumulative propagation cost is greater than the mask truncation threshold, and define them as available edge points.

[0014] Preferably, the real-time inference of the actual target position includes: extracting the position coordinates of each available edge point in the visible light image in real time, superimposing them with the corresponding displacement vector, and using the result as the target inference coordinate independently inferred from the available edge point; and taking the average coordinate of the target inference coordinates independently inferred from all available edge points as the actual target position.

[0015] This application has at least the following beneficial effects: This application acquires dual-modal images in real time and performs gradient fusion to obtain joint saliency. Its beneficial effect lies in utilizing the cross-modal complementary characteristics of visible light and infrared light to forcibly filter out false features of two-dimensional screen-printed ink on the circuit board surface that lack physical height. It accurately extracts the contours of solid components with three-dimensional convex height that can physically block smoke, providing an objective physical resistance data foundation for subsequent high-fidelity deduction of interference boundaries. Furthermore, by filtering peripheral edge points to extract the displacement vectors of each peripheral edge point pointing to the initial position of the target, its beneficial effect is that, under interference-free conditions where the laser has not yet been fired and the camera's field of view is clear, pixels with significant edge responses can be pre-selected outside the tracking frame, and their fixed geometry with the target center can be calculated. Displacement vectors, these pre-stored displacement vectors, are independent of whether the target center is visible, providing an independent geometric constraint for the subsequent reverse deduction of the target center position using surviving peripheral points under completely blind conditions. Extracting the centroid of the overexposed area in the infrared image during laser firing smooths out random errors caused by irregular fluctuations at the edges of the overexposed area, accurately and stably establishing the true physical origin of interference propagation. Then, through projection mapping, this centroid is transformed onto the visible light image, thus precisely locating the actual point of interaction between the laser and the target under completely blind conditions, providing a unique and reliable physical basis for the starting point of the subsequent wavefront propagation algorithm. The cumulative propagation cost of each pixel is deduced using the wavefront propagation algorithm, its beneficial effect being that the algorithm... Strictly adhering to the laws of fluid mechanics, the method perfectly recreates, at the digital level, the asymmetric and irregular diffusion pattern of vaporized smoke rapidly spreading on a flat substrate and then being impeded and trapped by the steps of three-dimensional components, achieving precise digital representation of the obstructed area. A truncation search region is defined, and the extreme value of the accumulated propagation cost within this region is extracted as the mask truncation threshold. Its advantage lies in its ability to adapt to differences in smoke diffusion range under different laser powers and target sizes. Simultaneously, the tracking box range is adjusted by a boundary contraction coefficient to eliminate noise interference from the outermost edge, balancing complete shielding of the smoke-covered area with effective filtering of edge disturbances. Usable edge points are selected from the outermost edge points, based on the visible light image of all usable edge points. The system uses the position coordinates and displacement vectors of the target to infer its actual position in real time, thereby controlling the laser optical axis to align and execute laser destruction. Its beneficial effect is that it enables the determination of pre-stored peripheral edge points before firing, accurately removes failure points shrouded in dense smoke, and extracts usable edge points that have not yet been affected by the smoke. This screening mechanism ensures that subsequent target center calculations rely only on currently visible and undisturbed usable edge points, avoiding the introduction of incorrect constraints by obscured points. This enables accurate inversion and stable calculation of the true target center position even in blind zone conditions where the target center cannot be directly viewed. It effectively overcomes the technical defects of target tracking failure under the dual interference of smoke obscuration and thermal deformation, and improves the operational accuracy and reliability of high-energy laser target point destruction. Attached Figure Description

[0016] The following section provides a more detailed description of the laser destruction target identification method based on multimodal fusion perception, as described in this application, with reference to the accompanying drawings.

[0017] Figure 1 A flowchart illustrating the steps of the laser destruction target precision identification method based on multimodal fusion perception provided in this application embodiment. Detailed Implementation

[0018] The following description, in conjunction with the accompanying drawings and embodiments, provides a more detailed explanation of the laser destruction target identification method based on multimodal fusion perception proposed in this application.

[0019] Please see Figure 1 The diagram illustrates a flowchart of a method for accurate identification of laser destruction targets based on multimodal fusion sensing, according to an embodiment of this application. The method includes the following steps: Step 1: Real-time acquisition of multimodal images of the target area, including visible light images and infrared images, and establishment of projection mapping relationship between the two modal images; receiving the target tracking box selected by the host computer and defining its geometric center as the initial position of the target; Step 2: In the state where the laser is not fired, the gradient features of the pixels in the two modal images are fused to obtain the joint saliency of each pixel; and based on the tracking box, the peripheral edge points in the visible light image are screened to extract the displacement vector of each peripheral edge point pointing to the initial position of the target. Step 3: Under laser firing state, extract the overexposed area based on the pixel values ​​in the infrared image, project its geometric centroid onto the visible light image through a projection mapping relationship, and define the projection position as the laser contact hot spot; take the laser contact hot spot as the starting wavefront node, use the joint saliency as the spatial constraint for wavefront expansion, and use the wavefront propagation algorithm to deduce the cumulative propagation cost of each pixel; delineate the truncation search area by the span of the tracking box boundary from the laser contact hot spot, and use the cumulative propagation cost within the truncation search area to determine the mask truncation threshold; Step 4: Based on the mask truncation threshold, select usable edge points from the outer edge points. Based on the position coordinates and displacement vectors of all usable edge points in the visible light image, infer the actual position of the target in real time, so as to control the laser optical axis alignment and perform laser destruction.

[0020] This embodiment uses electronic components on discarded circuit boards as the target for destruction. Some chips containing confidential data (such as memory chips and encrypted MCUs) on these discarded circuit boards have a rigid epoxy resin encapsulation shell, an internal silicon wafer, and a metal lead frame. To completely destroy the internal silicon wafer, continuous focused bombardment with a high-energy laser is required. During continuous laser bombardment, the extremely high heat causes thermal stress warping or deformation of the circuit board substrate, causing the actual target center to move in physical space. At the same time, the intense light and dense smoke generated by the firing instantly and completely obscure the target's location, causing the camera to be "blinded" in the core area. Since the laser equipment cannot see the moved target center, it cannot generate a correction command, and the emitted laser will miss its target. Therefore, it is necessary to deduce the actual target center, hidden beneath the smoke, in real time.

[0021] When the laser-operated equipment is powered on but not emitting destructive high-energy laser light, the weak heat dissipation residual from the electronic components ensures that the infrared camera, in a state of thermal equilibrium, can capture the weak radiation boundaries caused by the objective differences in infrared emissivity between different materials (e.g., between a resin substrate and metal pins). In step 1, the specific process is as follows: Step 101: Using a visible light camera and an infrared camera deployed on a gimbal structure, simultaneously and in real time acquire visible light and infrared images of the target area, perform cross-modal pixel coordinate alignment, and obtain the projection mapping relationship from the infrared image to the visible light image. In this setup, the visible light camera and the infrared camera are mounted side by side on the gimbal structure. The two cameras objectively have parallel parallax, which inevitably causes the pixel position coordinates of the same target object to be translated and scaled out in the visible light image and the infrared image. Therefore, it is necessary to align the coordinate systems of the acquired images.

[0022] Specifically, corner detection algorithms are called separately to extract feature corners from the visible light image and the infrared image, forming feature corner sets. For the two extracted feature corner sets, distance comparison is performed using feature descriptors to filter and match feature corner matching pairs representing the same physical location in space. These pairs are then substituted into the affine transformation matrix solution formula to calculate the affine transformation matrix that projects pixels from the infrared image onto the visible light image. The solved affine transformation matrix is ​​then used to perform a spatial transformation on the infrared image to align its pixel coordinates with those of the visible light image. The corner detection algorithm, feature descriptors, and affine transformation matrix solution are all well-known techniques.

[0023] Step 102: Receive the target tracking frame selected by the host computer, and define the geometric center of the tracking frame as the initial position of the target; The target tracking frame is received via a serial interface from an external host computer operator who has pre-selected and sent the target tracking frame. For the target tracking frame, the coordinates of the four vertices surrounding the tracking frame are read. The geometric center of the tracking frame is obtained by calculating the intersection of the diagonals of the rectangle and is defined as the initial position of the target. This position represents the initial target center that the laser optical axis needs to be precisely aligned with and destroyed when there is no interference from transient strong light and smoke.

[0024] The surface of discarded circuit boards is usually coated with large areas of text silkscreen or coating ink. Because these silkscreen inks are extremely thin and lack physical height, they produce drastic pixel color jumps in visible light images. However, in infrared images, due to their extremely low heat capacity and close adhesion to the substrate, they are unlikely to generate a sufficiently identifiable thermal radiation gradient. If the two images are directly fused, these false two-dimensional features without physical height will severely interfere with the subsequent deduction of smoke propagation resistance. Therefore, by extracting pixel gradients and introducing a product operation, one-sided false features are forcibly eliminated. The specific process in step 2 is as follows: Step 201: Extract the gradient magnitude of each pixel in the visible light image and the infrared image respectively, and normalize it. Using the projection mapping relationship, calculate the product of the normalized gradient magnitude between each pixel in the infrared image and the pixel at the projection position in the visible light image, and use it as the joint saliency of each pixel. Specifically, the gradient is calculated using the Sobel operator, and the magnitude of the gradient is also calculated. Both the Sobel operator and the magnitude calculation are well-known techniques and will not be elaborated upon here. Furthermore, the maximum normalization method is used to normalize the gradient magnitudes of all pixels in the image. The maximum normalization method is also a well-known technique and will not be elaborated upon here. The projection mapping relationship represents the affine transformation matrix. Further, for missing pixels in the visible light image that failed to receive direct mapping data during the projection process, the joint saliency of its adjacent and already assigned pixels is obtained, neighborhood interpolation is performed, and the interpolation result is assigned to the missing pixel, ultimately generating the joint saliency of continuous pixels across the entire visible light image.

[0025] It should be noted that, to avoid computational crashes due to a zero denominator when performing regular normalization in absolutely flat and featureless regions, or to avoid mistakenly amplifying weak camera white noise into significant edge features, a basic noise floor suppression logic is introduced before performing pixel value scaling. Specifically, if the maximum gradient magnitude of all pixels in the image is less than a pre-set basic noise floor constant, then the gradient magnitude of all pixels in the image is forcibly set to 0. The basic noise floor constant can be set by collecting the average gradient magnitude of an empty background without a target. Therefore, only when there is a visual contour jump in the visible light image and a material change in the infrared image at a certain location on the circuit board surface will the noise floor be suppressed. When there is a jump in emissivity thermal radiation, both gradients are significantly greater than zero, so the joint significance obtained from the product term can retain a significant positive value. Conversely, for screen printing ink with a large area, although the gradient modulus in the visible light image is large, the gradient modulus in the infrared image is close to zero, so the joint significance at that point must also be zero. The larger the joint significance, the more it indicates that there is a strong visual contour jump and a sudden change in material thermal radiation at that location on the circuit board surface, effectively eliminating the interference of two-dimensional flat screen printing, and truly reflecting the existence of a three-dimensional solid component boundary protruding from the circuit board surface. This solid boundary has a strong physical blocking effect on the transient vaporization smoke generated by laser firing.

[0026] Furthermore, to address the dilemma that the initial position of the target will inevitably become invalid after the target is bombarded by a high-energy laser, the undisturbed edge pixels are extracted in the outer safe area of ​​the visible light image before the laser bombardment, and the position of these outer points relative to the initial target center is quantified, so as to serve as the key constraint basis for subsequent multi-point voting inversion compensation under the condition of complete blindness.

[0027] Step 202: In the state where the laser is not fired, the area outside the tracking box in the visible light image is defined as the outer region. Pixels with a normalized gradient magnitude greater than a preset threshold in the outer region are selected and defined as outer edge points. The displacement vector from each outer edge point to the initial position of the target is extracted. The process of obtaining the preset threshold is as follows: calculate the mean value of the normalized gradient magnitude of all pixels in the visible light image, and use it as the preset threshold.

[0028] It should be noted that the displacement vector quantifies the static spatial relationship between the initial position of the target and its surrounding contour frame.

[0029] Furthermore, at the instant the laser-operated equipment performs the ignition and bombardment action, the high-energy laser contact with the target material causes a rapid local temperature rise, inevitably resulting in a large area of ​​bright, saturated white spots on the target surface within the infrared image. If only the pixel with the highest global temperature is extracted, the extreme value of this single pixel will exhibit irregular and violent fluctuations within the flat overexposure noise area, leading to positioning failure. Therefore, the specific process in step 3 is as follows: Step 301: In the laser firing state, all pixels in the infrared image are traversed in real time. Pixels with pixel values ​​greater than the preset thermal saturation threshold are marked. A connected component analysis algorithm is used to aggregate and extract connected components from all adjacent marked pixels. The connected component with the largest area is selected as the overexposed area. Using the projection mapping relationship, the projection position of the geometric centroid of the overexposed area in the visible light image is used as the laser contact hot spot. The method for obtaining the preset thermal saturation threshold is as follows: extract the maximum pixel value and the average pixel value of all pixels in the infrared image, calculate the average value of the two, and select the maximum value between the average value and the preset thermal saturation baseline value as the thermal saturation threshold. Secondly, the preset thermal saturation baseline value can be obtained by calibrating a specific type of infrared camera with a blackbody radiation source in advance. In this embodiment, it is usually set to 85% of the extreme value of the image bit depth grayscale. For example, for an 8-bit image, the constant value is 216. As for other implementation methods, the implementer can set it according to the actual situation. In addition, the geometric centroid is calculated using image moments. Image moments and connected component analysis algorithms are well-known technologies and will not be described in detail here.

[0030] It should be noted that during conventional continuous laser bombardment, the overall brightness of the infrared image will drift due to changes in ambient temperature or fluctuations in laser power. By extracting the maximum and average pixel values ​​in the current infrared image and calculating their average as a baseline, the system can dynamically adapt to changes in image brightness and accurately segment abnormally bright areas that are significantly detached from background noise, thus avoiding oversegmentation or undersegmentation problems that are easily caused by fixed thresholds. Secondly, in extreme conditions such as when the laser has just ignited or when the target is completely obscured by extremely thick smoke, the infrared image may not be significantly overexposed. In this case, the maximum pixel value of the image is abnormally low, resulting in an extremely small average value calculated above. If this extremely small average value is used directly as the segmentation threshold, large areas of normal background environment will be incorrectly extracted as high-heat areas. To prevent this failure, a preset thermal saturation baseline value reflecting the camera's physical limits is introduced, and the larger of the two values ​​is taken to forcibly set an insurmountable physical isolation baseline. This ensures that only pixels that have actually experienced overexposure due to high heat are extracted, effectively preventing background misjudgment under low threshold conditions. By accurately positioning the geometric centroid of the overexposed area within the visible light field of view, the origin of interference from transient strong light and smoke is established, serving as the initial wavefront center for deriving the hindrance and spread boundary in the subsequent topological evolution calculation model.

[0031] When vaporized smoke diffuses disorderly outwards on a flat surface of a circuit board, its diffusion wavefront is significantly physically impeded when it encounters a solid chip step or metal pin with a significant height difference, resulting in irregular asymmetric boundaries. This interference propagation pattern is digitally represented using a two-dimensional image morphology algorithm. Specifically, a laser contact hotspot is used as the divergence source, and the joint saliency extracted in the previous fusion is used as the propagation resistance. A Fast Marching Model (FMM) algorithm is introduced to deduce the topological cost distribution matrix that objectively conforms to the three-dimensional contour of the component.

[0032] Step 302: Use the wavefront propagation algorithm to deduce and update the cumulative propagation cost of each pixel in the visible light image, specifically as follows: In the visible light image, the cumulative propagation cost of the pixel corresponding to the high hot spot of the laser contact is set to 0; the cumulative propagation cost of the remaining pixels is set to infinity. Taking the laser contact hot spot as the starting wavefront node, the remaining pixels in the 8-neighborhood of the current wavefront node, excluding those already selected as wavefront nodes, are defined as extension nodes. The joint saliency of each extension node is negatively mapped and used as the expansion step size weight. The pixel distance between the current wavefront node and each extension node is calculated, and the ratio of this distance to the expansion step size weight of that extension node is used as the expansion cost increment. Calculate the sum of the cumulative propagation cost and the extension cost increment of the current wavefront node. If the value is less than the cumulative propagation cost currently recorded by the extension node, then the cumulative propagation cost of the extension node is overwritten and updated to the sum; otherwise, it remains unchanged. Among all pixels that have had their cumulative propagation cost updated and have not been selected as wavefront nodes, the pixel with the smallest cumulative propagation cost is selected as the new current wavefront node. The above update operation is repeated until the traversal is complete, and the cumulative propagation cost of each pixel in the visible light image is obtained. Among them, the wavefront propagation algorithm adopts the fast-moving algorithm, which is a well-known technique and will not be elaborated here; secondly, the specific calculation formula for the extended step size weight is as follows: In the formula, For the first The expansion step size weights of each extension node, For the first The joint significance of the extended nodes, This is the parameter tuning factor; in this embodiment, the parameter tuning factor is set to... This is to prevent calculations from crashing when the denominator is zero; Additionally, the pixel distance is 1 when the extended node is adjacent to the current wavefront node in the horizontal or vertical direction; the pixel distance is 1 when the extended node is adjacent to the current wavefront node in the diagonal direction. .

[0033] It should be noted that, The larger the value, the more drastic the physical obstacle change encountered at that location, and the smaller the calculated expansion step weight. At this time, the expansion cost increment increases sharply, which means that in the process of topological evolution, the accumulated cost required for the wavefront to cross the physical obstacle is higher and the expansion speed is slower. Thus, the irregular boundary shape of the smoke spread encountering physical step blockage and the resulting stagnation and flow around is accurately reproduced at the digital level. The larger the accumulated propagation cost, the greater the total resistance experienced by the wavefront from the starting point to the pixel and the longer the time taken, reflecting that the probability of the location being affected by the smoke wave is lower or the impact occurs later.

[0034] In the fast-moving algorithm, the cumulative propagation cost shows a monotonically increasing diffusion trend from the laser contact hot spot to the periphery. This cumulative propagation cost quantitatively characterizes the cumulative difficulty required for the smoke wavefront to spread from the hot spot to that location, that is, the time sequence in which that location is covered by smoke. The smaller the value, the earlier the location is covered by smoke; conversely, the larger the value, the later the location is covered by smoke.

[0035] To transform the aforementioned continuous cumulative propagation cost distribution into a Boolean mask boundary that can be used for image segmentation, a reasonable threshold needs to be determined such that regions with a cumulative propagation cost below the threshold are identified as smoke-covered areas, and regions with a cumulative propagation cost above the threshold are identified as safe areas unaffected by the smoke.

[0036] Because different firing laser powers result in drastically different smoke diffusion ranges, using a fixed threshold might lead to an extracted mask that is too large or too small. Considering that the tracking frame is defined by the operator before laser firing, its geometric range objectively reflects the actual size and spatial distribution of the target circuit board. The four vertices of the tracking frame represent the farthest boundary of the target area on the image plane, with the vertex furthest from the high-heat center defining the maximum spatial range of the target under optimal diffusion conditions. However, since the target area inevitably fails after laser firing, and the target edges may deviate due to thermal stress deformation or turbulent fluctuations at the smoke edges, directly using the full range of the maximum distance would lead to an excessively large cutoff threshold, introducing external environmental noise interference. Therefore, a boundary contraction coefficient is introduced to appropriately shrink the tracking frame range, ensuring coverage of the main target area while reserving sufficient safety margin to resist edge disturbances.

[0037] Step 303: Extract all vertices of the tracking box, calculate the maximum pixel distance between all vertices and the laser contact hotspot, and multiply it by a preset boundary shrinkage coefficient as the truncation search radius; take the laser contact hotspot in the visible light image as the center, and take the circular area covered by the truncation search radius as the truncation search area, and extract the maximum cumulative propagation cost of all pixels located in the truncation search area as the mask truncation threshold. It should be noted that the maximum pixel distance represents the farthest target corner that the tracking box can frame; the preset boundary shrinkage coefficient is used to eliminate the interference of environmental noise outside the tracking box on the extraction of the maximum pixel distance, and to reserve sufficient anti-deformation safety margin inside the tracking box. This value is set to 0.8. As other implementation methods, implementers can set it according to the actual situation.

[0038] It should be noted that by extracting the maximum cumulative propagation cost within the truncated search area, the peak value of the ultimate cost that can be reached when the smoke wavefront encounters the most severe physical obstruction within the currently defined physical diffusion range is accurately identified. This characterizes the highest degree of difficulty in wavefront propagation under extremely hindered conditions. Using the maximum value as the truncated baseline, the absolute physical boundary of the smoke diffusion under the current operating conditions can be established. In the subsequent definition process, abnormal propagation areas exceeding this ultimate peak value can be effectively filtered out, preventing the mask from erroneously overflowing out of control in the direction of strong resistance. This ensures that the defined interference failure area can be strictly controlled within the expected correlation range of the target tracking box, improving the safety and convergence of blind zone bullseye extrapolation.

[0039] Furthermore, based on the mask truncation threshold, straightforward Boolean logic is used to forcibly isolate pixels, ensuring that severely interfering areas are completely eliminated. Then, the surviving pixels are extracted, and a fallback error-proofing mechanism is introduced. By superimposing previously recorded displacement vectors for multi-point addition and calculating the arithmetic mean, local deformation errors are cleverly offset. The actual target position is successfully inferred within the blind zone where the target center is doubly obscured, generating safe and reliable tracking guidance commands. Therefore, the specific process in step 4 is as follows: Step 401: For all peripheral edge points in the visible light image, select peripheral edge points whose cumulative propagation cost is greater than the mask truncation threshold and define them as usable edge points; It should be noted that when the cumulative propagation cost of the peripheral edge points is less than or equal to the mask truncation threshold, it indicates that the location is in a state of overexposure by strong light or severe obstruction by thick smoke, and the pixel features at this location are no longer reliable. Conversely, it indicates that the location is far from the laser contact hotspot or is strongly blocked by the tall physical electronic device in the middle, and the vaporized smoke has not yet reached this location. This indicates that the location is still in a safe and unobstructed usable area, and the pixel features at this location are reliable and can be retained for subsequent target back-calculation.

[0040] Given the rigidity of the circuit board, the geometric distance from the unobstructed outer edge points to the actual target center can be considered constant for a very short time. However, the high thermal shock of the laser can cause local thermal stress warping deformation of the substrate, and even cause slight displacement and fluctuations in some extracted safety points. Therefore, directly relying on single-point displacement vector compensation will cause severe jitter in the inferred target center. By introducing a fault-tolerant mechanism of multi-point independent deduction and averaging, individual small deformation errors are canceled out in the arithmetic mean, ensuring a stable and objective target center of gravity is calculated.

[0041] Step 402: Retrieve the displacement vectors pointing from each available edge point to the initial position of the target, extract the position coordinates of each available edge point in the visible light image in real time, and use the superposition result of the coordinates with the corresponding displacement vector as the target inference coordinates independently derived by the available edge point; calculate the average coordinates of the target inference coordinates independently derived by all available edge points as the actual position of the target. It should be noted that if an accidental explosion causes a large area of ​​dense smoke, resulting in all peripheral edge points of the entire screen being judged as invalid areas, i.e., the number of usable edge points is 0, then subsequent calculations will be immediately stopped, and the actual position of the target output from the previous frame image will be retrieved as the alternative coordinates for output. This will forcibly maintain the short-term position of the servo mechanism under completely blind conditions, preventing the gimbal from deflecting out of control. Secondly, by solving for the mean coordinates, the spatial discrete jump error introduced by the deformation and displacement of individual usable edge points due to heat is neutralized and smoothed out. Then the actual position of the target is the location of the target that cannot be directly seen due to occlusion interference.

[0042] The control gimbal structure realigns its laser optical axis and continuously locks onto the actual position of the target to perform continuous high-energy focused bombardment on the target, ensuring that the destructive high-energy laser always accurately bombards the true target of the core electronic device under the interference of transient dense smoke and thermal stress deformation, until the end of the entire laser destruction task. Specifically, the process involves: reading the coordinates of the current visible light camera's field of view center, calculating the difference between the actual position coordinates of the current target and the field of view center position, obtaining the two-dimensional pixel offset of the target's lateral and longitudinal deviation from the center of the field of view, using this as an error input parameter, and outputting a servo electrical signal command through a PID controller. This servo electrical signal command is then transmitted to the mechanical servo drive component of the gimbal structure to drive the gimbal motor to perform rotation angle correction, thereby offsetting the two-dimensional pixel offset and enabling the laser optical axis to be accurately aligned back to the actual position of the discarded electronic components. The PID (Proportion Integration Differentiation) controller is a well-known technology and will not be described in detail here.

Claims

1. A method for accurate identification of laser-destruction targets based on multimodal fusion sensing, characterized in that, The method includes the following steps: The system acquires multimodal images of the target area in real time, including visible light images and infrared images, and establishes a projection mapping relationship between the two modal images; it receives the target tracking box selected by the host computer and defines its geometric center as the initial position of the target. In the state where the laser is not fired, the gradient features of pixels in the two modal images are fused to obtain the joint saliency of each pixel; and based on the tracking box, the peripheral edge points in the visible light image are screened to extract the displacement vector of each peripheral edge point pointing to the initial position of the target. In the laser firing state, the overexposed area is extracted based on the pixel value in the infrared image, and its geometric centroid is projected onto the visible light image through the projection mapping relationship. The projection position is defined as the laser contact hot spot. Taking the laser contact hot spot as the starting wavefront node and the joint saliency as the spatial constraint for wavefront expansion, the cumulative propagation cost of each pixel is derived using the wavefront propagation algorithm. The truncation search area is defined by the span of the boundary of the tracking box from the laser contact hot spot. The mask truncation threshold is determined by the cumulative propagation cost within the truncation search area. Based on this, usable edge points are selected from the outer edge points. The actual position of the target is inferred in real time based on the position coordinates and displacement vectors of all usable edge points in the visible light image, so as to control the laser optical axis alignment and perform laser destruction.

2. The method for accurate identification of laser destruction targets based on multimodal fusion sensing as described in claim 1, characterized in that, The method for obtaining the joint saliency of each pixel includes: extracting the gradient magnitude of each pixel in the visible light image and the infrared image respectively, and performing normalization processing; using the projection mapping relationship, calculating the product of the normalized gradient magnitude between each pixel in the infrared image and the pixel at its projection position in the visible light image, and using this product as the joint saliency of each pixel.

3. The method for accurate identification of laser destruction targets based on multimodal fusion sensing as described in claim 1, characterized in that, The method of filtering peripheral edge points in a visible light image based on a tracking box includes: in the state where the laser is not fired, defining the area outside the tracking box in the visible light image as the peripheral area, and filtering pixels in the peripheral area whose normalized gradient magnitude is greater than a preset threshold, defining them as peripheral edge points.

4. The method for accurate identification of laser destruction targets based on multimodal fusion sensing as described in claim 1, characterized in that, The process of obtaining the overexposed area is as follows: under laser firing state, all pixels in the infrared image are traversed in real time, and pixels with pixel values ​​greater than the preset thermal saturation threshold are marked. A connected component analysis algorithm is used to aggregate and extract connected components from all adjacent marked pixels, and the connected component with the largest area is selected as the overexposed area.

5. The method for accurate identification of laser destruction targets based on multimodal fusion sensing as described in claim 1, characterized in that, The derivation process of the cumulative propagation cost is as follows: The cumulative propagation cost of the pixel corresponding to the laser contact hotspot in the visible light image is set to 0; the cumulative propagation cost of the remaining pixels is set to infinity; taking the laser contact hotspot as the starting wavefront node, the remaining pixels in the 8-neighborhood of the current wavefront node, excluding those already selected as wavefront nodes, are defined as extension nodes. Based on the joint saliency of each extension node and its distance from the current wavefront node, calculate the incremental expansion cost required for single-step wavefront propagation from the current wavefront node to each extension node. Calculate the sum of the cumulative propagation cost and the extension cost increment of the current wavefront node. If the value is less than the cumulative propagation cost currently recorded by the extension node, then the cumulative propagation cost of the extension node is overwritten and updated to the sum; otherwise, it remains unchanged. Among all pixels that have had their cumulative propagation cost updated and have not been selected as wavefront nodes, the pixel with the smallest cumulative propagation cost is selected as the new current wavefront node. The above update operation is repeated until the traversal is complete, and the cumulative propagation cost of each pixel in the visible light image is obtained.

6. The method for accurate identification of laser destruction targets based on multimodal fusion sensing as described in claim 5, characterized in that, The calculation process of the expansion cost increment is as follows: negatively map the joint saliency of each extension node as the expansion step size weight, calculate the pixel distance between the current wavefront node and each extension node, and use the ratio of this distance to the expansion step size weight of the extension node as the expansion cost increment.

7. The method for accurate identification of laser destruction targets based on multimodal fusion sensing as described in claim 1, characterized in that, The process of obtaining the truncation search region is as follows: extract all vertices of the tracking box, calculate the maximum pixel distance between all vertices and the laser contact hotspot, and multiply it by a preset boundary shrinkage coefficient as the truncation search radius; take the laser contact hotspot in the visible light image as the center, and take the circular area covered by the truncation search radius as the truncation search region.

8. The method for accurate identification of laser destruction targets based on multimodal fusion sensing as described in claim 1, characterized in that, The mask truncation threshold is the maximum cumulative propagation cost of all pixels located within the truncation search area.

9. The method for accurate identification of laser destruction targets based on multimodal fusion sensing as described in claim 1, characterized in that, The process of selecting usable edge points is as follows: for all peripheral edge points in the visible light image, select peripheral edge points whose cumulative propagation cost is greater than the mask truncation threshold and define them as usable edge points.

10. The method for accurate identification of laser destruction targets based on multimodal fusion sensing as described in claim 1, characterized in that, The real-time inference of the target's actual position includes: extracting the position coordinates of each available edge point in the visible light image in real time, superimposing them with the corresponding displacement vector, and using the result as the target inference coordinates independently inferred from the available edge point; and taking the average coordinates of the target inference coordinates independently inferred from all available edge points as the target's actual position.

Images