Chip tray remaining life prediction method and device, equipment and program product

Abstract

The present disclosure relates to a chip tray residual life prediction method and device, equipment and program product, and relates to the technical field of chip post-process testing. The method comprises: obtaining to-be-detected component data, the to-be-detected component data comprising spatial image data and multi-band spectral data of the to-be-detected component; performing feature extraction processing on the multi-band spectral data and the spatial image data to obtain spectral feature extraction results and spatial feature extraction results; performing feature fusion and damage identification processing on the spectral feature extraction results and the spatial feature extraction results to obtain damage identification results and a health state index of the to-be-detected component; and determining a residual life prediction result of the to-be-detected component according to the health state index. The present disclosure jointly uses the spectral feature extraction results and the spatial feature extraction results in the life prediction process, thereby improving the comprehensiveness of feature extraction; the above features can take into account both hidden damage and apparent damage of the to-be-detected component, thereby improving the prediction accuracy of the residual life of the component.

Description

Technical Field

[0001] This disclosure relates to the field of chip back-end testing technology, and more specifically, to a method for predicting the remaining lifetime of a chip tray, a device for predicting the remaining lifetime of a chip tray, an electronic device, and a computer program product. Background Technology

[0002] Semiconductor device manufacturing processes are divided into front-end and back-end processes. Wafer manufacturing and testing are called front-end processes, while chip packaging, testing, and finished product warehousing are called back-end processes. The back-end testing process is responsible for performing electrical performance testing and sorting on the packaged chips. In this process, the chip carrier (such as a tray) serves as the tooling for carrying and fixing the chips; its structural integrity directly affects the safety of the chips during transportation and testing, as well as the reliability of the test results.

[0003] Trays are typically made of polystyrene (PS), polycarbonate (PC), or conductive plastic. They have multiple pockets on their surface, and each pocket is surrounded by multiple retaining walls (also called limiting posts or baffles) to hold the chip edges in place and prevent the chip from shifting during the process flow. The integrity of the retaining walls directly affects the chip transport stability and testing accuracy.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a method, device, electronic device, computer-readable storage medium, and computer program product for predicting the remaining lifetime of a chip tray, thereby overcoming, to at least some extent, the problems of one-sided feature extraction, insufficient detection accuracy, and low detection efficiency in related tray wall life prediction schemes.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] According to a first aspect of this disclosure, a method for predicting the remaining lifetime of a chip tray is provided, comprising: acquiring data of a component to be inspected, the data of the component to be inspected including spatial image data and multi-band spectral data of the component to be inspected; performing feature extraction processing on the multi-band spectral data and the spatial image data to obtain spectral feature extraction results and spatial feature extraction results; performing feature fusion and damage identification processing on the spectral feature extraction results and the spatial feature extraction results to obtain damage identification results and a health status index of the component to be inspected; and determining the remaining lifetime prediction result of the component to be inspected based on the health status index.

[0008] In one exemplary embodiment of this disclosure, acquiring the data of the component to be tested includes: triggering a data acquisition signal in response to the component to be tested arriving at the testing station; performing positioning processing on the component to be tested based on the data acquisition signal, so that the positioning accuracy of the component to be tested is within a specified accuracy threshold range; acquiring spatial images of the component to be tested to obtain spatial image data; and acquiring multi-band spectral images of the component to be tested to obtain multi-band spectral data.

[0009] In one exemplary embodiment of this disclosure, the step of acquiring multi-band spectral images of the component to be tested to obtain the multi-band spectral data includes: determining image acquisition environment parameters and the material type of the component to be tested; determining spectral image acquisition parameters corresponding to different bands based on the image acquisition environment parameters and the material type; acquiring images of the component to be tested in different bands based on the spectral image acquisition parameters of each band to obtain a multi-band panoramic spectral image; and performing image preprocessing on the multi-band panoramic spectral image to obtain the multi-band spectral data.

[0010] In one exemplary embodiment of this disclosure, the step of preprocessing the multi-band panoramic spectral image to obtain the multi-band spectral data includes: performing contour recognition and region segmentation processing on the multi-band panoramic spectral image to obtain a multi-band fixed wall image; performing image denoising processing on the fixed wall image of each band to obtain a multi-band denoised fixed wall image; performing light intensity difference elimination processing and pixel alignment processing on the multi-band denoised fixed wall image to obtain a multi-band aligned fixed wall image; determining the reflectance difference value corresponding to different bands, and performing background separation processing on the multi-band aligned fixed wall image based on the reflectance difference value to obtain the multi-band spectral data, wherein the multi-band spectral data is a standardized fixed wall image after format conversion processing.

[0011] In one exemplary embodiment of this disclosure, the step of performing feature extraction processing on the multi-band spectral data and the spatial image data to obtain spectral feature extraction results and spatial feature extraction results includes: acquiring a pre-trained component life prediction model, the component life prediction model including a spectral feature extraction branch and a spatial feature extraction branch; performing spectral feature extraction processing on the multi-band spectral data based on the spectral feature extraction branch to obtain the spectral feature extraction results; and performing spatial feature extraction processing on the spatial image data based on the spatial feature extraction branch to obtain the spatial feature extraction results.

[0012] In one exemplary embodiment of this disclosure, the multi-band spectral data includes multi-band spectral feature blocks, and the spectral feature extraction branch includes a convolutional layer, an attention module, and an output branch. The step of performing spectral feature extraction processing on the multi-band spectral data based on the spectral feature extraction branch to obtain the spectral feature extraction result includes: performing spectral feature extraction processing on the multi-band spectral feature blocks for different bands using the convolutional layer of the spectral feature extraction branch to obtain spectral features for each band; performing feature enhancement processing on the spectral features of each band based on the material type of the component to be detected using the attention module to obtain spectral enhancement features for each band; performing feature fusion processing on the spectral enhancement features of multiple bands to obtain a latent damage feature map; and outputting a damage cause probability vector and an internal anomaly score of the component based on the latent damage feature map using the output branch, wherein the damage cause probability vector includes the damage type of the component to be detected.

[0013] In one exemplary embodiment of this disclosure, the spatial feature extraction branch includes an encoder, a decoder, and an output head. The step of performing spatial feature extraction processing on the spatial image data based on the spatial feature extraction branch to obtain the spatial feature extraction result includes: performing multi-level encoding processing on the spatial image data using the encoder to obtain an encoded feature map, the encoded feature map including multiple levels of encoded features, the encoded feature map being the spatial features of a fixed wall; determining the deepest level encoded feature from the multiple levels of encoded features; the decoder performing upsampling decoding operations layer by layer starting from the deepest level encoded feature, and performing feature fusion processing with the shallow detail features corresponding to each level to obtain a decoded feature map; and the output head outputting a visible damage segmentation map and a quantization index vector based on the decoded feature map.

[0014] In one exemplary embodiment of this disclosure, the component life prediction model further includes a feature fusion module. The step of performing feature fusion and damage identification processing on the spectral feature extraction results and the spatial feature extraction results to obtain the damage identification result and health status index of the component to be detected includes: determining the material type of the component to be detected; determining the material calculation parameters and feature fusion weights corresponding to the component to be detected based on the material type; performing feature fusion processing on the spectral feature extraction results and the spatial feature extraction results based on the feature fusion weights to obtain a fused feature vector; performing damage identification processing based on the fused feature vector to obtain the damage identification result; and determining the health status index based on the material calculation parameters and the fused feature vector.

[0015] In one exemplary embodiment of this disclosure, the damage identification result includes damage type and damage level. The damage identification processing based on the fused feature vector to obtain the damage identification result includes: performing damage type identification processing on the fused feature vector to obtain the damage type, which includes missing damage, crack damage, and wear damage; determining the spectral feature difference vector and spatial morphological damage degree value of the component to be detected based on the fused feature vector; determining the vector distance between the spectral feature difference vector and a reference difference vector; determining the damage difference between the spatial morphological damage degree value and a preset damage threshold; and determining the damage level based on the vector distance and the damage difference.

[0016] In one exemplary embodiment of this disclosure, determining the remaining lifespan prediction result of the component under test based on the health status index includes: acquiring a health status index sequence of the component under test within a specified time period; determining the first component failure threshold and component auxiliary features corresponding to the component under test; determining the remaining lifespan prediction result based on the first component failure threshold, the component auxiliary features, and the health status index sequence; and determining the warning level of the component under test based on the comparison between the remaining lifespan prediction result and the remaining lifespan threshold, wherein the warning level is used to trigger a specified level of component remaining lifespan warning prompt.

[0017] In one exemplary embodiment of this disclosure, the method further includes: obtaining a pre-constructed component anomaly sample set, the component anomaly sample set including multiple anomaly labeled samples, the anomaly labeled samples being samples whose prediction deviation of the remaining lifetime prediction result reaches a preset deviation threshold; when a model update condition is detected, updating the component lifetime prediction model based on the anomaly labeled samples to obtain an updated component lifetime prediction model.

[0018] According to a second aspect of this disclosure, a device for predicting the remaining lifetime of a chip tray is provided, comprising: a component data acquisition module for acquiring data of a component to be tested, the data of the component to be tested including spatial image data and multi-band spectral data of the component to be tested; a feature extraction module for performing feature extraction processing on the multi-band spectral data and the spatial image data to obtain spectral feature extraction results and spatial feature extraction results; a health status index determination module for performing feature fusion and damage identification processing on the spectral feature extraction results and the spatial feature extraction results to obtain damage identification results and a health status index of the component to be tested; and a remaining lifetime prediction module for determining the remaining lifetime prediction result of the component to be tested based on the health status index.

[0019] In one exemplary embodiment of this disclosure, the component data acquisition module includes a component data acquisition unit, configured to: trigger a data acquisition signal in response to the component to be tested arriving at the testing station; perform positioning processing on the component to be tested based on the data acquisition signal, so that the positioning accuracy of the component to be tested is within a specified accuracy threshold range; acquire spatial images of the component to be tested to obtain spatial image data; and acquire multi-band spectral images of the component to be tested to obtain multi-band spectral data.

[0020] In one exemplary embodiment of this disclosure, the component data acquisition unit includes a spectral data acquisition unit, configured to: determine image acquisition environment parameters and the material type of the component to be tested; determine spectral image acquisition parameters corresponding to different bands based on the image acquisition environment parameters and the material type; acquire images of the component to be tested in different bands based on the spectral image acquisition parameters of each band to obtain a multi-band panoramic spectral image; and perform image preprocessing on the multi-band panoramic spectral image to obtain the multi-band spectral data.

[0021] In one exemplary embodiment of this disclosure, the spectral data acquisition unit includes a data preprocessing subunit, configured to: perform contour recognition and region segmentation processing on the multi-band panoramic spectral image to obtain a multi-band fixed wall image; perform image denoising processing on the fixed wall image of each band to obtain a multi-band denoised fixed wall image; perform light intensity difference elimination processing and pixel alignment processing on the multi-band denoised fixed wall image to obtain a multi-band aligned fixed wall image; determine the reflectance difference value corresponding to different bands, and perform background separation processing on the multi-band aligned fixed wall image based on the reflectance difference value to obtain the multi-band spectral data, wherein the multi-band spectral data is a standardized fixed wall image after format conversion processing.

[0022] In one exemplary embodiment of this disclosure, the feature extraction module includes a feature extraction unit, configured to: acquire a pre-trained component life prediction model, the component life prediction model including a spectral feature extraction branch and a spatial feature extraction branch; perform spectral feature extraction processing on the multi-band spectral data based on the spectral feature extraction branch to obtain the spectral feature extraction result; and perform spatial feature extraction processing on the spatial image data based on the spatial feature extraction branch to obtain the spatial feature extraction result.

[0023] In one exemplary embodiment of this disclosure, the multi-band spectral data includes multi-band spectral feature blocks. The spectral feature extraction branch includes a convolutional layer, an attention module, and an output branch. The feature extraction unit includes a spectral feature extraction subunit, used for: performing spectral feature extraction processing on the multi-band spectral feature blocks for different bands by the convolutional layer of the spectral feature extraction branch to obtain spectral features for each band; performing feature enhancement processing on the spectral features of each band based on the material type of the component to be detected by the attention module to obtain spectral enhancement features for each band; performing feature fusion processing on the spectral enhancement features of multiple bands to obtain a latent damage feature map; and outputting a damage cause probability vector and an internal anomaly score of the component based on the latent damage feature map by the output branch, wherein the damage cause probability vector includes the damage type of the component to be detected.

[0024] In one exemplary embodiment of this disclosure, the spatial feature extraction branch includes an encoder, a decoder, and an output head. The feature extraction unit includes a spatial feature extraction subunit, configured to: perform multi-level encoding processing on the spatial image data through the encoder to obtain an encoded feature map, the encoded feature map including multi-level encoded features, the encoded feature map being the spatial features of a fixed wall; determine the deepest level encoded feature from the multi-level encoded features; perform upsampling decoding operations layer by layer starting from the deepest level encoded feature by the decoder, and perform feature fusion processing with the shallow detail features corresponding to each level to obtain a decoded feature map; and output a visible damage segmentation map and a quantization index vector based on the decoded feature map by the output head.

[0025] In one exemplary embodiment of this disclosure, the component life prediction model further includes a feature fusion module. The health status index determination module includes a health status index determination unit, configured to: determine the material type of the component to be tested; determine the material calculation parameters and feature fusion weights corresponding to the component to be tested based on the material type; perform feature fusion processing on the spectral feature extraction result and the spatial feature extraction result based on the feature fusion weights to obtain a fused feature vector; perform damage identification processing based on the fused feature vector to obtain the damage identification result; and determine the health status index based on the material calculation parameters and the fused feature vector.

[0026] In one exemplary embodiment of this disclosure, the damage identification result includes damage type and damage level. The health status index determination unit includes a damage result determination subunit, configured to: perform damage type identification processing on the fused feature vector to obtain the damage type, wherein the damage type includes missing damage, crack damage, and wear damage; determine the spectral feature difference vector and spatial morphological damage degree value of the component to be detected based on the fused feature vector; determine the vector distance between the spectral feature difference vector and the reference difference vector; determine the damage difference between the spatial morphological damage degree value and a preset damage threshold; and determine the damage level based on the vector distance and the damage difference.

[0027] In one exemplary embodiment of this disclosure, the remaining lifetime prediction module includes a remaining lifetime prediction unit, configured to: acquire a health status index sequence of the component to be tested within a specified time period; determine the first component failure threshold and component auxiliary features corresponding to the component to be tested; determine the remaining lifetime prediction result based on the first component failure threshold, the component auxiliary features, and the health status index sequence; and determine the warning level of the component to be tested based on a comparison between the remaining lifetime prediction result and the remaining lifetime threshold, wherein the warning level is used to trigger a specified level of component remaining lifetime warning prompt.

[0028] In one exemplary embodiment of this disclosure, the remaining lifetime prediction device for the chip tray further includes a model update module, configured to: acquire a pre-built component anomaly sample set, the component anomaly sample set including multiple anomaly labeled samples, the anomaly labeled samples being samples whose prediction deviation of the remaining lifetime prediction result reaches a preset deviation threshold; when a model update condition is detected, update the component lifetime prediction model based on the anomaly labeled samples to obtain an updated component lifetime prediction model.

[0029] According to a third aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory storing computer-readable instructions that, when executed by the processor, implement the chip tray remaining lifetime prediction method according to any one of the preceding claims.

[0030] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method for predicting the remaining lifetime of a chip tray according to any one of the preceding claims.

[0031] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the remaining lifetime prediction method for a chip tray as described in any of the preceding claims.

[0032] The technical solution provided in this disclosure may include the following beneficial effects: The remaining lifetime prediction method for chip trays in the exemplary embodiments of this disclosure, on the one hand, simultaneously acquires spatial image data and multi-band spectral data of the component to be inspected during the data acquisition stage, and uses the obtained spectral feature extraction results and spatial feature extraction results together in the lifetime prediction process, thereby improving the comprehensiveness of feature extraction. On the other hand, based on the spectral feature extraction results and spatial feature extraction results, both latent and overt damage to the component to be inspected can be taken into account. At the same time, by combining the health status index to quantify the health level of the component to be inspected, the prediction accuracy of the remaining lifetime of the component can be improved, and the detection efficiency can be increased.

[0033] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings: Figure 1 A schematic diagram of the cavity recesses on the chip tray is shown; Figure 2 A flowchart illustrating a method for predicting the remaining lifetime of a chip tray according to an exemplary embodiment of the present disclosure is shown schematically. Figure 3 This diagram illustrates the effect of fixed wall surface damage under three wavebands according to an exemplary embodiment of the present disclosure; Figure 4This schematically illustrates an overall architecture diagram of a component under test for predicting remaining lifetime according to an exemplary embodiment of the present disclosure. Figure 5 A schematic diagram of a fixed wall breach identified according to an exemplary embodiment of this disclosure is shown; Figure 6 A block diagram illustrating a chip tray remaining lifetime prediction device according to an exemplary embodiment of the present disclosure is shown. Figure 7 A schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure is shown. Figure 8 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown schematically. Detailed Implementation

[0035] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0036] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of the specific details described, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known structures, methods, apparatuses, implementations, materials, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0037] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, or in one or more software-hardened modules, or in different network and / or processor devices and / or microcontroller devices.

[0038] According to embodiments of this disclosure, a method for predicting the remaining lifetime of a chip tray, a device for predicting the remaining lifetime of a chip tray, a computer-readable storage medium, an electronic device, and a computer program product are provided.

[0039] The principles and spirit of this disclosure will be explained in detail below with reference to several representative embodiments.

[0040] In the back-end testing process of chips, the tray serves as a carrier and transport medium for the chips, containing multiple recesses (i.e., cavities) for placing the chips. (See reference...) Figure 1 , Figure 1 A schematic diagram of the cavity recesses on the chip tray is shown. Figure 1 The tray includes multiple grooves, such as the crystal cavity groove 110, which refers to the equidistant molded crystal cavities (Pockets) located on the plastic / paper carrier tape. These are used to support and fix individual chips in the tape packaging to prevent slippage or electrostatic damage during transportation. They are then sealed with a cover tape for storage on the reel.

[0041] The height of the recessed mounting wall is typically 1.5 to 3.0 mm, with a tolerance range of ±0.1 mm. In actual production, the mounting wall may be damaged after 50 to 200 cycles of use. When the mounting wall is damaged, the chip cannot be effectively secured, leading to problems such as chip protrusion from the cavity, damage during robotic gripper handling, and poor contact of the test socket in subsequent processes. In severe cases, this can result in the scrapping of the entire batch of chips. Therefore, predicting the lifespan of the tray mounting wall is a critical step in chip back-end testing. Trays used in chip back-end testing can typically be made of various engineering plastic materials, and the significant differences in material properties directly affect the lifespan of the mounting wall. The main limitations of existing tray mounting wall lifespan prediction schemes are as follows: Feature extraction is incomplete, resulting in low prediction accuracy. Most existing lifetime prediction schemes rely on single features (such as spectral features or spatial image features alone), failing to simultaneously identify both latent damage (internal cracks, material fatigue) and overt damage (surface cracks, severe wear). Furthermore, they lack a clear health status evaluation system, making it impossible to quantify the health of fixed walls, leading to incomplete damage identification and a lack of explicit interpretive pathways for lifetime prediction. Since the detection accuracy of 2D visible light visual inspection is ±0.05mm, it cannot detect wall height changes or microcracks smaller than 0.05mm; it can only detect grayscale differences in the visible light band, failing to detect microcracks smaller than 0.1mm, making it difficult to meet the stringent requirements of mid-to-high-end chip testing; and it does not fully consider the material characteristics of different engineering plastics, making it unsuitable for lifetime prediction needs of multi-material trays.

[0042] Poor adaptability to small sample scenarios. In industrial scenarios, it is difficult to collect failure samples of tray wall fixings. Related solutions mostly rely on a large number of labeled samples for training. In small sample scenarios, overfitting is prone to occur, and the prediction accuracy drops significantly, making it unable to adapt to the actual industrial production needs. Furthermore, no specific training strategy is designed for the damage characteristics of different engineering plastics, and the material characteristics are not combined with health status assessment, further reducing the prediction accuracy.

[0043] Insufficient adaptability to multiple materials. In chip back-end testing, trays are mainly made of various engineering plastics. The high temperature resistance, wear resistance and dimensional stability of different materials vary significantly, resulting in large differences in damage development speed and life cycle. Most related solutions are designed for trays made of a single material, without considering the differences in spectral characteristics and wear characteristics of different engineering plastics. They use fixed parameter modeling and do not dynamically adjust the health status evaluation criteria according to material characteristics, resulting in inconsistent prediction accuracy in multi-material scenarios.

[0044] There is a lack of a fully closed-loop management mechanism. The existing solutions can only achieve the function of life prediction, but they do not combine the prediction results with damage tracing, maintenance strategies, and data traceability, and cannot form a closed loop of "prediction-early warning-maintenance-traceability". The prediction results are difficult to implement and cannot effectively avoid production losses. At the same time, they do not take into account the maintenance needs of different engineering plastics, cannot provide targeted maintenance suggestions, and do not link the Health Status Index (HSI) with the early warning mechanism. The early warning threshold lacks scientific basis.

[0045] The auxiliary acquisition modules are highly homogenized. Most related solutions use traditional multispectral supplementary lighting sources for image acquisition, which are prone to reflection interference, affecting image quality and feature extraction accuracy. In addition, the light source modules are highly homogenized, have high maintenance costs, and lack innovative designs. They cannot adapt to the spectral acquisition requirements of different engineering plastics (such as transparent PC and antistatic ABS), thus affecting the accuracy of the Health Status Index (HSI) calculation.

[0046] Based on this, in this example embodiment, a method for predicting the remaining lifetime of a chip tray is first provided. The remaining lifetime prediction method of the chip tray disclosed herein can be implemented using a server, or it can be implemented using a terminal device. The terminal described in this disclosure may include mobile terminals such as mobile phones, tablets, laptops, handheld computers, and personal digital assistants (PDAs), as well as fixed terminals such as desktop computers. Figure 2 A schematic diagram illustrating a process flow for predicting the remaining lifetime of a chip tray according to some embodiments of the present disclosure is provided. Reference Figure 2 The method for predicting the remaining lifetime of the chip tray may include the following steps: Step S210: Obtain the data of the component to be tested, which includes the spatial image data and multi-band spectral data of the component to be tested; Step S220: Perform feature extraction processing on multi-band spectral data and spatial image data to obtain spectral feature extraction results and spatial feature extraction results; Step S230: Perform feature fusion and damage identification processing on the spectral feature extraction results and spatial feature extraction results to obtain the damage identification results and health status index of the component to be detected; Step S240: Determine the remaining life prediction result of the component to be tested based on the health status index.

[0047] According to the chip tray remaining lifetime prediction method in this example embodiment, on the one hand, during the data acquisition stage, spatial image data and multi-band spectral data of the component to be tested are acquired simultaneously, and the obtained spectral feature extraction results and spatial feature extraction results are used together in the lifetime prediction process, improving the comprehensiveness of feature extraction. On the other hand, based on the spectral feature extraction results and spatial feature extraction results, both latent and overt damage to the component to be tested can be taken into account. At the same time, by combining the health status index to quantify the health level of the component to be tested, the prediction accuracy of the component's remaining lifetime can be improved, and the detection efficiency can be increased.

[0048] The remaining lifetime prediction method for the chip tray in this example embodiment will be further explained below.

[0049] Before performing the remaining life prediction operation for the component to be tested, hardware setup and debugging can be carried out first. The multispectral acquisition module, supplementary lighting module, positioning module, data processing module, linkage feedback module, and storage module should be installed according to the industrial production line layout to ensure that each module works collaboratively. Specifically, the multispectral acquisition module can include a multispectral industrial camera with a resolution of 25 megapixels or higher, installed directly above the testing station and vertically aligned with the conveyor line; the supplementary lighting module is configured with a corresponding number of independent supplementary lighting units for different wavelengths, installed on both sides of the camera, with the angle adjusted to avoid shadows and reflections; the positioning module can include a visual positioning camera, mechanical limit device, laser calibrator, etc., installed on both sides of the testing station to ensure accurate positioning after the tray is delivered; the data processing module can include a multispectral deep learning processor and is connected to each hardware module via data cables to ensure real-time data transmission; the linkage feedback module can include an audible and visual alarm, industrial display screen, conveyor line linkage controller, etc., installed in a position that is observable and operable by the operator; the storage module is connected to the data processing module, with sufficient storage space reserved.

[0050] In one exemplary embodiment of this disclosure, step S210, acquiring data of the component to be tested, includes: triggering a data acquisition signal in response to the component to be tested arriving at the testing station; performing positioning processing on the component to be tested based on the data acquisition signal, so that the positioning accuracy of the component to be tested is within a specified accuracy threshold range; acquiring spatial images of the component to be tested to obtain spatial image data; and acquiring multi-band spectral images of the component to be tested to obtain multi-band spectral data.

[0051] The component to be inspected can be a chip tray, such as multiple grooves within the chip tray, including the fixing walls or other components around each groove. When the tray to be inspected is conveyed to the inspection station by an industrial conveyor line at a preset speed (adjustable), the position sensor on the conveyor line detects that the tray has reached its position, triggering a data acquisition signal and automatically sending the signal to the data processing module. Upon receiving the signal, the data processing module controls the conveyor line to pause and simultaneously triggers the positioning module to start. The visual positioning camera acquires images of the tray's edge reference points and fixing holes, completing dual visual positioning. A mechanical limit device extends to fix the tray's position, preventing displacement during inspection. Furthermore, a laser calibrator emits a laser to correct slight displacement of the tray, ensuring that the positioning accuracy of the component to be inspected is within a specified accuracy threshold range (e.g., 0~0.01mm). After positioning is completed, the positioning module sends a "positioning qualified" signal to the data processing module, triggering the next data acquisition process.

[0052] During the data acquisition process, image acquisition equipment such as industrial cameras can be used to acquire spatial images of the part to be inspected, thereby obtaining spatial image data. The spatial image data can be a high-resolution single-channel image in the visible light band. Since the visible light band has relatively rich texture information, it is suitable for extracting the surface geometric features of the part to be inspected in this disclosure.

[0053] Because in related solutions, multispectral supplementary lighting sources are prone to specular reflection on transparent / highly reflective trays (such as those made of PC material), leading to the loss of surface crack and scratch features, this embodiment proposes an integrated industrial airflow-assisted cleaning and spectral enhancement device to address the above problems. This device suppresses optical interference through millisecond-level airflow disturbance, achieving contactless cleaning and also amplifying anomalies. Its core components include: an adjustable-speed airflow generator, a high-precision directional nozzle, and a timing controller linked to a multispectral camera. The adjustable-speed airflow generator has parameters such as a power of 500W, a wind speed of 0.5~5m / s, and multiple adjustable settings. The high-precision directional nozzle has an effective outlet orifice diameter of 5mm and an electrically adjustable direction of 0~90°. The synchronization accuracy of the timing controller linked to the multispectral camera is ≤0.1s.

[0054] Furthermore, the airflow-assisted multispectral acquisition device provided in this embodiment is used to acquire spectral data of the component under test. This airflow-assisted multispectral acquisition device can perform differentiated operations based on the material of the component under test. By employing the aforementioned multi-band spectral data acquisition method, the airflow can disturb the air medium, disrupt the specular reflection path, and improve the signal-to-noise ratio of surface defects in materials such as transparent PC.

[0055] In one exemplary embodiment of this disclosure, multi-band spectral image acquisition is performed on the component to be tested to obtain multi-band spectral data, including: determining image acquisition environment parameters and the material type of the component to be tested; determining spectral image acquisition parameters corresponding to different bands based on the image acquisition environment parameters and the material type; acquiring images of the component to be tested in different bands based on the spectral image acquisition parameters of each band to obtain a multi-band panoramic spectral image; and performing image preprocessing on the multi-band panoramic spectral image to obtain multi-band spectral data.

[0056] After receiving the "positioning qualified" signal, the data processing module synchronously triggers the multispectral acquisition module and the supplementary lighting module to start. The supplementary lighting module can determine the spectral image acquisition parameters corresponding to different bands based on the image acquisition environment parameters and the material type of the component under test. The image acquisition environment parameters can be the environmental parameters corresponding to the multi-band spectral data acquisition process, including but not limited to ambient light intensity, air temperature, and relative humidity. The material type can be the specific type of material used in the component under test. The spectral image acquisition parameters can be the parameters used during spectral data acquisition, including but not limited to supplementary lighting brightness, data acquisition angle, exposure time, acquisition speed, and specific preprocessing parameters for different bands.

[0057] In this embodiment, three-band data acquisition is used as an example. Before data acquisition, the three-band acquisition parameters can be configured through software and parameter configuration. For example, the wavelength of the visible light (VIS) band is 400~760nm, the wavelength of the near infrared (NIR) band is 760~1100nm, and the wavelength of the short-wave infrared (SWIR) band is 1100~2500nm. The synchronous exposure time (10~100ms) and acquisition speed (e.g., 3 frames / second) can also be set. In addition, band-specific preprocessing parameters can be configured. For example, the visible light adaptive bilateral filtering window can be configured to 5×5; the near infrared wavelet threshold denoising parameters can be configured to wavelet basis db4 and decomposition layer number 3; the short-wave infrared mean drift denoising bandwidth can be configured to 10; the positioning accuracy threshold can be set to ≤0.01mm, and the spectral calibration parameters and background separation threshold can be configured to reflectance 0.1.

[0058] In other exemplary embodiments of this disclosure, the specific parameters for other bands can also be configured according to specific needs; in addition, more bands of spectral data can be collected according to data acquisition needs. This disclosure does not impose any special limitation on the specific number of bands collected.

[0059] During multispectral data acquisition, the supplementary lighting module automatically adjusts the brightness (0~1000 lux) and angle (0~90°) of the three-band supplementary lighting according to the material of the tray and the ambient light intensity to avoid shadows and reflections on the edges and gaps of the fixed wall. Furthermore, the supplementary lighting module can adjust parameters according to the material of the tray. The parameters can be set manually at the beginning and then automatically recognized by the material adaptive unit.

[0060] Furthermore, the multispectral industrial camera simultaneously acquires panoramic images of the tray through three independent acquisition channels, including visible light, near-infrared, and short-wave infrared images. During acquisition, synchronous exposure and transmission across the three bands are ensured to avoid image misalignment caused by asynchronous acquisition. The three-band data acquisition workflow can include: Step 1, after the tray is in position, the cleaning mode is activated to remove dust; for example, the airflow speed in the cleaning mode can be 3~5 m / s and last for 1 second. For PS material, the airflow speed is limited to 4 m / s to prevent deformation. Step 2, instantly switch to the auxiliary mode to maintain a low-speed airflow layer for data acquisition; the airflow speed in the auxiliary mode can be 1~3 m / s. Step 3, the multispectral camera simultaneously acquires three-band data to obtain multispectral data.

[0061] After acquisition, the multispectral acquisition module transmits the three-band panoramic images to the data processing module in real time for preprocessing. Simultaneously, the supplementary lighting module is shut down, awaiting the next acquisition command. Once acquisition is complete, the acquired three-band panoramic spectral images undergo image preprocessing to obtain multi-band spectral data.

[0062] The mechanisms of action, detection targets, and typical reactions after illumination differ across different wavelength bands. For example, the main mechanism of action in the visible light band is surface reflection / color, and the detection target is surface defects (scratches, stains, color differences). Typical reactions after illumination include significant color / brightness differences in defect areas; scratches produce highlights or shadows; and stains absorb specific wavelengths, causing localized darkening or discoloration. The main mechanism of action in the near-infrared band is scattering + weak absorption, and the detection target is internal latent defects (microcracks, bubbles, delamination). Typical reactions after illumination include internal structural disturbances affecting the scattering path, resulting in blurred bright spots, dark areas, or abnormal textures in the transmission / reflection images; and enhanced contrast in areas with intact surfaces but internal abnormalities. The main mechanism of action in the short-wave infrared band is molecular vibrational absorption, and the detection target is changes in material composition (wear, oxidation, contamination). Typical reactions after illumination include enhanced absorption in worn or oxidized areas due to changes in chemical bonds, appearing as distinct dark areas in SWIR images; and characteristic absorption peaks in oil and moisture residues, creating high contrast.

[0063] This embodiment acquires data in three bands: visible light, near-infrared, and short-wave infrared, corresponding to the feature capture of surface defects, internal hidden cracks, and wear areas, respectively, achieving full-dimensional defect information coverage of "surface + interior" and accurately matching the defect detection scenario of fixed walls. Simultaneously, it integrates independent acquisition channels for the three bands, supporting synchronous exposure and transmission to avoid image misalignment and spectral information deviation caused by asynchronous acquisition, ensuring data correlation. Furthermore, it is equipped with a high-transmittance anti-interference optical lens and a built-in band filtering module to isolate inter-band optical crosstalk at the hardware level, reducing environmental interference such as dust and reflections, and improving the purity of spectral data.

[0064] In one exemplary embodiment of this disclosure, image preprocessing is performed on a multi-band panoramic spectral image to obtain multi-band spectral data, including: performing contour recognition and region segmentation processing on the multi-band panoramic spectral image to obtain a multi-band fixed wall image; performing image denoising processing on the fixed wall image of each band to obtain a multi-band denoised fixed wall image; performing light intensity difference elimination processing and pixel alignment processing on the multi-band denoised fixed wall image to obtain a multi-band aligned fixed wall image; determining the reflectance difference corresponding to different bands, and performing background separation processing on the multi-band aligned fixed wall image based on the reflectance difference to obtain multi-band spectral data, wherein the multi-band spectral data is a standardized fixed wall image after format conversion processing.

[0065] After obtaining a multi-band panoramic spectral image through data acquisition, contour recognition and region segmentation can be performed on the image to obtain a multi-band fixed wall image. Contour recognition involves identifying the contours of multiple chip placement positions on the tray in the panoramic spectral image. Region segmentation involves dividing the region of each chip placement position on the tray in the panoramic spectral image. Using a band-adaptive segmentation algorithm, the contour of each chip placement position on the tray is automatically identified, and four fixed wall regions (around the chip placement position) are extracted for each position. The segmentation accuracy can be configured to be below 0.01 mm. Irrelevant parts such as tray edges and blank areas are removed to obtain the multi-band fixed wall image.

[0066] A multi-band fixed wall image, also known as a multi-band fixed wall region image, can be an image containing relevant information about a fixed wall region in multiple different spectral bands. In this embodiment, the multi-band fixed wall image can also be considered a three-band fixed wall region image. Image denoising processing is performed on the segmented three-band fixed wall region images respectively; the image denoising processing can be an adaptive denoising operation performed on images of different spectral bands. For example, adaptive bilateral filtering can be used in the visible light band to eliminate surface interference such as dust and minor stains; wavelet thresholding can be used in the near-infrared band to retain the characteristic information of internal hidden cracks; and mean-shift denoising can be used in the short-wave infrared band to enhance the spectral difference between worn and intact areas.

[0067] After denoising the fixed wall images in each band, multi-band denoised fixed wall images are obtained. A spectral calibration algorithm is used to eliminate the light intensity differences between the three denoised fixed wall images, ensuring the consistency of the spectral data. Furthermore, this disclosure achieves pixel-level alignment of the three band images through the positioning results during data acquisition, avoiding feature extraction errors caused by image misalignment.

[0068] Specifically, pixel alignment processing can include spatial alignment and spectral alignment. Spatial alignment aims to achieve pixel-level alignment of three-band images, which can be achieved through system calibration, unified positioning reference, pixel-level registration, and alignment quality verification. Because sensor response sensitivities differ across bands, the spectral power distribution of light sources varies in each band, and the transmittance of optical lenses differs across bands, without spectral alignment, the same healthy area might exhibit high reflectivity in the visible light band but low response in the short-wave infrared band. This discrepancy could lead to misclassification by the model as material abnormalities. To eliminate these discrepancies, spectral alignment can be performed, i.e., the aforementioned light intensity difference elimination processing.

[0069] The spectral alignment operation includes the following steps: First, standard whiteboard calibration: Before system deployment, reference images for three bands are acquired using a standard diffuse whiteboard. Second, dark current subtraction: With the light source turned off, dark-field images for the three bands are acquired, and the sensor's noise background is recorded. The corresponding dark-field image is subtracted from each normal image to eliminate dark current interference. Third, gain normalization: For each band, the known reflectivity of the standard whiteboard is divided by the actual response value of the whiteboard image to obtain the gain calibration coefficient for that band. Fourth, dynamic calibration: Because the light source will age and drift over time, the system automatically triggers a recalibration process every preset period (e.g., weekly or every thousand tests) to update the gain coefficient and maintain the consistency of the three band data.

[0070] Because the tray background (usually black polymer or metal), dust particles, and the target mounting wall (such as gold-plated contacts or plastic sealant) exhibit significant differences in reflectance spectral shapes across the visible, near-infrared, and short-wave infrared bands, this embodiment utilizes these differences (e.g., calculating the NIR / VIS ratio, SWIR reflectance threshold, etc.) and sets specific rules to precisely mask the background and dust pixels. These rules could be based on a reflectance value greater than a threshold (e.g., 0.1) for a certain band, or a difference between two bands greater than a threshold (e.g., 0.1). Based on these reflectance differences, background separation processing is performed on the multi-band aligned mounting wall image to remove background, dust, and other interfering factors, outputting a standardized three-band mounting wall area image. Simultaneously, the image data is converted into a model-recognizable format to obtain multi-band spectral data.

[0071] refer to Figure 3 , Figure 3 The diagram illustrates the effect of fixed wall surface damage under three wavelengths according to an exemplary embodiment of the present disclosure. Figure 3 The image shows the effects of surface damage on a fixed wall under three wavelengths: visible light, near-infrared, and short-wave infrared. Figure 3 As can be seen, different wavebands emphasize different image features. For example, the visible light band can show the light and shadow produced by the fixed wall, the near-infrared band can show the sawtooth wear of the groove, and the short-wave infrared band has a relatively clear outline of the fixed wall.

[0072] After the above image preprocessing operations, multi-band spectral data can be obtained. This multi-band spectral data is a standardized fixed-wall image that has undergone format conversion processing. It can then be transmitted to a two-branch deep learning model for remaining lifetime prediction. This disclosure, by preprocessing multi-band panoramic spectral images, obtains input data suitable for model processing and uses it in the subsequent remaining lifetime prediction process.

[0073] In one exemplary embodiment of this disclosure, step S220 involves performing feature extraction processing on multi-band spectral data and spatial image data to obtain spectral feature extraction results and spatial feature extraction results. This includes: acquiring a pre-trained component life prediction model, which includes a spectral feature extraction branch and a spatial feature extraction branch; performing spectral feature extraction processing on the multi-band spectral data based on the spectral feature extraction branch to obtain spectral feature extraction results; and performing spatial feature extraction processing on the spatial image data based on the spatial feature extraction branch to obtain spatial feature extraction results.

[0074] refer to Figure 4 , Figure 4The diagram schematically illustrates the overall architecture for predicting the remaining lifetime of a component under test according to an exemplary embodiment of this disclosure. This disclosure pre-trains a component lifetime prediction model, which is a two-branch deep learning model. This model takes the spatial image data, multi-band spectral data, and component material of the component under test as input data to perform remaining lifetime prediction and obtain the remaining lifetime prediction result for the component under test.

[0075] from Figure 4 As can be seen, the component life prediction model includes a spatial feature extraction branch 410 and a spectral feature extraction branch 420. The spatial feature extraction branch 410 performs feature extraction processing on the spatial image data, as shown in step S410, where the spatial image is processed. The spectral feature extraction branch 420 performs feature extraction processing on the multi-band spectral data, as shown in step S420, where the spectral data is processed. After receiving the standardized image data, the data processing module starts the dual-branch deep learning model, with the two branches operating in parallel to complete feature extraction and deep fusion.

[0076] For the spatial feature extraction branch 410, spatial feature extraction is performed on the spatial image data to obtain the spatial feature extraction result. The spatial image data can be a high-resolution single-channel image in the visible light band. A single-channel image refers to an image where each pixel is represented by only one numerical value indicating brightness (grayscale) information. The spatial image data can be a spatial image tensor in H×W×C format; H is the image height, W is the image width, and C is the number of channels. The spatial feature extraction result can be a spatial feature map corresponding to the spatial image data, typically represented by a multidimensional feature tensor. The spatial feature extraction branch can mine the spatial structure, texture, location, and semantic association information of pixels / regions from the original image data to obtain a spatial feature vector containing the above information.

[0077] Furthermore, this disclosure allows multi-band spectral data to be input into the spectral feature extraction branch 420, which can then perform spectral feature extraction processing to obtain the spectral feature extraction result. The multi-band spectral data can be a multidimensional data block composed of spectral data from multiple different bands. The number of channels in the multi-band spectral data can be determined based on the total number of bands. For example, the multi-band spectral data can be a three-dimensional tensor of size H×W×B; where H is the image height, W is the image width, and B can be the total number of bands. The spectral feature extraction result can be a spectral feature vector obtained by extracting features from the multi-band spectral data corresponding to the component to be detected.

[0078] In the feature extraction process of the two branches, in step S430, feature decoupling constraints are applied to the two types of features. Feature decoupling constraints refer to the technical means of orthogonal constraint loss to achieve the technical goal of feature decoupling, with the aim of allowing the spectral branch and the spatial branch to learn uncorrelated features. Orthogonal constraint loss can penalize the correlation between the output features of the two branches, forcing them to be orthogonal to each other in the vector space. This disclosure uses a two-branch deep learning network of the remaining lifetime prediction model for parallel feature extraction. Subsequently, the features extracted by different branches can be fused for the determination of the damage type and damage level of the component to be detected.

[0079] In one exemplary embodiment of this disclosure, spectral feature extraction processing is performed on multi-band spectral data based on a spectral feature extraction branch to obtain spectral feature extraction results. This includes: the spectral feature extraction branch performs spectral feature extraction processing on the multi-band spectral feature blocks for different bands to obtain spectral features for each band; the attention module performs feature enhancement processing on the spectral features of each band based on the material type of the component to be detected to obtain spectral enhancement features for each band; the spectral enhancement features of multiple bands are fused to obtain a latent damage feature map; and the output branch outputs a damage cause probability vector and an internal anomaly score of the component based on the latent damage feature map, wherein the damage cause probability vector includes the damage type of the component to be detected.

[0080] Multi-band spectral data can be composed of multiple spectral feature blocks of different bands, such as the spectral data blocks of the visible light, near-infrared, and short-wave infrared bands mentioned in this disclosure. This embodiment uses a three-band spectral data block as an example. The multi-band spectral data input to the spectral feature extraction branch can be a spectral cube, i.e., a three-dimensional data block with dimensions H×W×3. H and W are the spatial pixel dimensions of the fixed-wall region, and the three channels of the third dimension correspond to the reflectance values ​​of the visible light, near-infrared, and short-wave infrared bands, respectively. In other words, each pixel in the fixed-wall region simultaneously records the spectral information of all three bands, and the three-band data of all pixels are stacked together to form a spectral cube.

[0081] The spectral feature extraction branch includes convolutional layers, an attention module, and an output branch. For example, in this embodiment, the spectral feature extraction branch may include a convolutional neural network (CNN) with a specified number of layers, multi-scale convolutional layers, squeeze-and-excitation (SE) attention layers, and a band interaction fusion layer. Because different materials have significantly different spectral response characteristics—for example, polyphenylene sulfide (PPS) is heat-resistant but sensitive to thermal fatigue in the short-wave infrared band; PC is transparent but prone to reflection in the visible light band, requiring stronger noise reduction—the attention module can adjust the spectral feature extraction parameters based on the material type identification results of the component to be detected. This allows the attention module to automatically adjust the preprocessing parameters of the spectral branch and the sensitivity of the convolutional kernel according to the material type of the component, ensuring the network focuses its attention on the bands where the material is most vulnerable and prone to problems.

[0082] The specific feature extraction process can be as follows: After receiving the three-band spectral data block, the spectral feature extraction branch first performs basic spectral spatial feature extraction through a CNN with a specified number of layers to obtain a basic feature map carrying preliminary damage information. Then, a multi-scale convolutional layer extracts the core spectral features of each band based on the basic feature map, resulting in the spectral features of each band. These core spectral features may include, but are not limited to, spectral reflectance, absorption coefficient, and peak / valley differences. Then, an attention mechanism (channel attention weight coefficients can be configured from 0.1 to 0.9) is used to enhance the spectral features of each band, strengthening defect features and suppressing irrelevant features, resulting in enhanced spectral features. These enhanced features can be feature data that enhances key features (such as defect features) and suppresses irrelevant features. The spectral feature extraction branch can also dynamically load a preset band sensitivity weight table based on the identified material type of the component to be detected, strengthening key bands.

[0083] After obtaining the spectral enhancement features, a band-interactive fusion layer is used to deeply fuse the enhanced spectral features of the three bands to construct defect-specific spectral features, resulting in a latent damage feature map. The latent damage feature map can be a feature map containing relevant information about latent damage in the fixed wall. Latent damage can include, but is not limited to, internal latent cracks, minor wear, etc. Latent damage can be represented by D... s Then, the spectral anomaly detection unit of the spectral feature extraction branch captures subtle spectral changes of hidden defects such as internal hidden cracks and slight wear based on the hidden damage feature map, and outputs the damage cause probability vector and the internal anomaly score of the component. The damage cause probability vector can be used to determine the damage type of the component to be detected.

[0084] The damage cause probability vector can be obtained by analyzing the abnormal patterns of reflectivity in each band, representing the probability distribution vector of the current damage caused by different factors. For example, the output damage cause probability vector can be [0.85, 0.10, 0.05], which can indicate that the current damage has an 85% probability of being caused by high-temperature fatigue, a 10% probability of being caused by chemical corrosion, and a 5% probability of being caused by mechanical impact. This probability distribution is used to guide the dynamic fine-tuning of weights in HSI calculation, enabling the tracing of damage causes. The component internal anomaly score can be obtained by modeling the spectral feature distribution of healthy materials using a Gaussian mixture model (GMM), calculating the deviation of the spectral features of the component under test from the healthy distribution (Mahathano distance), also known as the Gaussian Mixture Model (GMM) anomaly score. The higher the GMM anomaly score, the more severe the latent damage. This anomaly score is one of the core inputs constituting the latent damage features. The spectral feature extraction results obtained through this branch can be used to analyze the latent damage of the component under test, such as internal fatigue and microcracks.

[0085] In one exemplary embodiment of this disclosure, spatial feature extraction processing is performed on spatial image data based on a spatial feature extraction branch to obtain spatial feature extraction results. This includes: encoding the spatial image data at multiple levels using an encoder to obtain an encoded feature map, where the encoded feature map includes encoded features at multiple levels and represents the spatial features of a fixed wall; determining the deepest encoded feature from the multiple encoded features; performing upsampling decoding operations layer by layer starting from the deepest encoded feature using a decoder, and fusing the decoded features with the corresponding shallow detail features at each level to obtain a decoded feature map; and outputting a visible damage segmentation map and a quantization index vector from the output head based on the decoded feature map.

[0086] Continue to refer to Figure 4 The spatial feature extraction branch includes structures such as an encoder, decoder, and output head. Since the visible light band has the richest texture information and is best suited for extracting surface geometric features, the spatial image data only contains high-resolution single-channel images in the visible light band, while near-infrared and short-wave infrared data are only input to the spectral feature extraction branch. The network structure of the spatial feature extraction branch can be a modified U-Net. A U-Net network can include an encoder with a specified number of layers, each layer having its own corresponding decoder.

[0087] The feature extraction process of the spatial feature extraction branch can include: encoding spatial image data by coding blocks at different levels in the encoder; extracting degenerate features through hierarchical downsampling; and embedding cross-layer residual units (CRUs) between blocks. CRUs can be used to establish direct residual connections between adjacent layers of the encoder, directly transferring shallow detail features across layers to deeper layers, mitigating gradient vanishing, and avoiding the loss of details at damaged edges during downsampling. The encoded feature map is obtained by downsampling through an encoder with multiple coding layers.

[0088] The encoded feature map can include downsampled feature maps with progressively decreasing resolution. The encoded feature map contains spatial features of fixed walls, which may include, but are not limited to, contour integrity, surface smoothness, and dimensional consistency. The deepest features of the encoded feature map can be used as input to the decoder. The encoded feature map can be used to capture global context and high-level semantic information (such as explicit damage information) in spatial images.

[0089] The decoder can start from the deepest encoded features and upsample layer by layer to restore resolution. Each layer fuses shallow detail features from the corresponding encoder layer to compensate for information loss during upsampling. During decoding, the CRU can further enhance residual fusion of high and low layer features, improving the edge segmentation accuracy of cracks, wear, and other damage. After layer-by-layer decoding, a decoded feature map is obtained, with its resolution restored to be completely consistent with the input image of the spatial feature extraction branch. The decoded feature map can accurately locate boundaries by fusing shallow details from the encoder, restoring feature details, mapping semantic information back to pixel-level segmentation results, and outputting a spatial feature vector.

[0090] The feature map output by the decoder can be mapped from the number of channels to the number of damage categories (background, cracks, wear, etc.) using a 1×1 convolution. Then, a Softmax activation is applied to output the probability of each pixel belonging to each damage category, i.e., the explicit damage segmentation map, achieving pixel-level damage localization and classification. Furthermore, the output head can also output a corresponding quantization index vector based on the decoded feature map. The quantization index can refer to explicit damage features, which can be represented by D... p The quantitative indicators can include, but are not limited to: crack length, surface crack propagation length, crack width, maximum crack opening width, wear area ratio, wear area as a percentage of the total area of ​​the fixed wall, edge defect degree, deviation rate of the fixed wall outline from the standard template, surface flatness, and surface height deviation. The feature extraction results from the spatial feature extraction branch can include information such as whether the fixed wall can still hold the chip, whether the crack will rapidly propagate and break, and the current damage level, which, together with the spectral feature extraction results, constitute the input for HSI calculation.

[0091] In one exemplary embodiment of this disclosure, for step S230, the component life prediction model further includes a feature fusion module, which performs feature fusion and damage identification processing on the spectral feature extraction results and spatial feature extraction results to obtain damage identification results and health status index of the component to be detected. This includes: determining the material type of the component to be detected; determining the material calculation parameters and feature fusion weights corresponding to the component to be detected based on the material type; performing feature fusion processing on the spectral feature extraction results and spatial feature extraction results based on the feature fusion weights to obtain a fused feature vector; performing damage identification processing based on the fused feature vector to obtain a damage identification result; and determining the health status index based on the material calculation parameters and the fused feature vector.

[0092] After obtaining the feature extraction results from the two main branches, a feature fusion module can be used to fuse the two results. This feature fusion module can be a spatial-spectral attention fusion module, which receives the feature vectors output from the two main branches and performs feature fusion. (See also...) Figure 4 ,use Figure 4 The feature fusion module (including modules 430 and 440) performs feature fusion. The feature fusion module has structures such as conditional adaptive gating, dynamic weight fusion, and attention fusion modules. The feature fusion module can determine the feature fusion weights according to the material type of the component to be detected, and can also adaptively allocate weights according to the defect type, such as emphasizing spatial features for surface defects and emphasizing spectral features for latent defects.

[0093] The feature fusion module performs deep fusion of spectral and spatial features based on feature fusion weights, outputting a fused feature vector. This vector is then transmitted to the defect sub-classification unit in the remaining lifetime prediction model for defect type classification. The defect sub-classification unit performs damage identification processing based on the fused feature vector, obtaining the damage identification result for the component to be inspected. The damage identification result can include the damage type and damage level of the component to be inspected. After determining the material calculation parameters based on the material type, health status parameters can be determined based on the material calculation parameters and the fused feature vector.

[0094] For example, based on the material type, the corresponding configuration is retrieved from the material parameter library. The configuration parameters include the failure threshold and the spectral latent damage weight w for different materials. s Weight w of spatially manifest damage p Specifically, PPS: failure threshold 0.85, w s equals 0.6, w p Equal to 0.4, focus on spectral latent fatigue; PS: failure threshold 0.65, w s equals 0.3, w p Equal to 0.7, focus on visible spatial damage; PC: failure threshold 0.72, w sequals 0.4, w p The value is 0.6, with a focus on surface scratches. The adjustment coefficients 0.9 and 1.1 are fine-tuning factors relative to the baseline value, used for soft adjustments when the material identification confidence is not 100%, or for interpolation transitions when the material is at the boundary between two classes.

[0095] Furthermore, the attention mechanism within the dual-branch network can be adjusted based on the material to determine the weights used in the HSI calculation. For example, when detecting PPS, the channel attention of the spectral branch will automatically enhance the short-wave infrared channel and suppress the visible light channel. When detecting PS, the attention of the spatial branch will focus on the integrity of the edge contour.

[0096] The adjustment factor represents the degree of deviation of material properties relative to the default baseline. For example, the system default baseline weight is w. s equals 0.5, w p It equals 0.5. If identified as PPS, high-temperature fatigue is the dominant factor, and the spectral weight needs to be adjusted upwards. The adjustment factor is 1.1, and the actual w s This is equivalent to 0.5 multiplied by 1.1, approximately equal to 0.55 or higher. If identified as PS, the spectral characteristics are of little value, and the adjustment factor is 0.9, with the actual w... s That is equivalent to 0.5 multiplied by 0.9, which equals 0.45 or lower.

[0097] When fusing the feature extraction results of the two branches, the feature fusion weights can be determined based on the material type. These weights can be automatically generated by the Self-Space Adaptive Fusion Module (SSAFM). For example, the feature fusion weight can be α. A lightweight multilayer perceptron (MLP) learns the relative importance of the two features, and after Softmax normalization, the fused feature weights for the two features are obtained as [α, 1-α]. That is, α can represent the spectral latent damage weight w. s 1-α can represent the spatially manifest damage weight w p The spectral feature extraction result F output by the spectral feature extraction branch is based on the feature fusion weights. s Spatial feature extraction result F output by the spatial feature extraction branch p The features are concatenated and weighted to obtain the fused feature vector F. fuse =α·F s + (1-α)·F p For example, when latent fatigue is the main cause, α approaches 0.7, while when visible wear is the main cause, α approaches 0.3.

[0098] The fused feature vectors can be directly used for HSI calculation, for example, see [link to relevant documentation]. Figure 4 ,pass Figure 4 The multi-task learning head 450 is mapped to the [0,1] interval, outputs the current HSI value, and is simultaneously fed into the defect fine classification unit and the Long Short-Term Memory (LSTM) temporal prediction network. Since the HSI of the component under test is related to its material and damage as follows: the differences in the properties of different engineering plastics determine the differences in their damage type and development speed, thus determining the w value in the HSI calculation... s w p The weight allocation is as follows: High-temperature resistant materials include, but are not limited to, PPS, polyetheretherketone (PEEK), and liquid crystal polymers (LCP). The core damage in these materials is material fatigue (latent damage) caused by prolonged high temperatures. This damage develops slowly but is irreversible; therefore, w is considered in HSI calculations. s (Spectral latent damage weight) allocation is relatively high (e.g., w in PPS) s =0.6、w p =0.4), focusing on capturing the material fatigue level through spectral characteristics and calculating the HSI decay rate by combining the duration of high-temperature environment.

[0099] High-rigidity materials can include, but are not limited to, LCP, polyethersulfone (PES), and polysulfone (PSU). The core damage in these materials is brittle fracture leading to cracks (both overt and covert), and the damage propagates rapidly (once a microcrack appears, it easily propagates quickly). Therefore, in HSI calculations, w... s w p Weight balancing (such as w in LCP) s =0.5、w p =0.5), focusing on quantifying crack size through spatial characteristics, capturing internal microcracks through spectral characteristics, and calculating the HSI decay rate by combining the stress conditions.

[0100] High abrasion-resistant materials can include, but are not limited to, PEEK and PPS. The core damage of these materials is surface wear (visible damage) caused by long-term friction. The damage development rate is slow, therefore w in HSI calculations... p (Spatial overt damage weight) is allocated relatively high (e.g., PEEK's w) s =0.4、w p =0.6), focusing on quantifying the wear area and flatness through spatial characteristics, and combining the automated grasping frequency to estimate the HSI attenuation rate.

[0101] Low-toughness / easily deformable materials can include, but are not limited to, PS and polypropylene (PP). The core damage of these materials is "breakage and deformation caused by impact and airflow" (obvious damage). The damage develops rapidly, therefore w is used in HSI calculations. p Allocate the highest (e.g., PS's w) s =0.3、w p =0.7), focusing on capturing the size and degree of damage through spatial features, and combining transportation and grasping conditions to estimate the HSI attenuation rate.

[0102] General-purpose materials can include, but are not limited to, polyphenylene ether (PPE), modified polyphenylene ether (MPPE), acrylonitrile butadiene styrene plastic (ABS), and PC. These materials exhibit a wide range of damage types (wear, cracking, minor fatigue), therefore, w in HSI calculations... s w p Weight balancing (e.g., w in PPE / MPPE) s =0.45、w p =0.55), and based on the comprehensive operating conditions, the overall attenuation rate of HSI is estimated.

[0103] Continue to refer to Figure 4 In step S440, HSI health status monitoring is performed. The specific process of determining HSI based on the fused feature vector is as follows: HSI = Material ID × (Spectral latent damage weight × Latent feature + Spatial manifest damage weight × Manifest feature), which can be simplified to HSI = w s ·D s + w p ·D p Wherein, HSI is the health status index of the component under test; w s D represents the weight of spectral latent damage. s These are latent features (extracted from spectral branches, encompassing quantified values ​​of features such as internal cracks and material fatigue); w p D represents the weight of spatially manifest damage. p The features are explicit (extracted from spatial branches, encompassing quantitative values ​​such as surface wear, crack size, and flatness); the material ID is a unique identifier for eight mainstream engineering plastics (e.g., PPS=1.0, PS=0.8). The material ID can be obtained by looking up a table based on the identified material type. The parameter vector corresponding to the material is obtained through the table lookup mapping, and then the parameter vector of the material ID is used to dynamically drive w. s w pThe allocation and failure threshold setting enable HSI calculation with a "one material, one policy" approach.

[0104] The core function of HSI: HSI values ​​range from 0 to 1. The closer the value is to 1, the better the health status and the lower the degree of damage of the tray wall; the closer the value is to 0, the worse the health status and the more severe the damage. When the HSI first reaches the failure threshold of the corresponding material, it is determined that the wall is about to fail and should be taken out of service. The core value of HSI lies in quantifying and integrating latent damage, explicit damage, and material characteristics, providing explicit and interpretable core evidence for life prediction.

[0105] In one exemplary embodiment of this disclosure, damage identification processing based on fused feature vectors is performed to obtain damage identification results, including: performing damage type identification processing on the fused feature vectors to obtain damage types, including missing damage, crack damage, and wear damage; determining the spectral feature difference vector and spatial morphological damage degree value of the component to be detected based on the fused feature vectors; determining the vector distance between the spectral feature difference vector and the reference difference vector, and determining the damage difference between the spatial morphological damage degree value and a preset damage threshold; and determining the damage level based on the vector distance and the damage difference.

[0106] Continue to refer to Figure 4 The obtained fused feature vector is used as the input to the defect sub-classification unit in the multi-task learning head 450. After receiving the fused feature vector, the defect sub-classification unit performs damage type identification and root cause attribution in step S450. Defect judgment and early warning are performed according to the following steps. The defect sub-classification unit performs damage identification processing on the fused feature vector through the multi-label sub-classification head to obtain the damage type. The damage type can be the specific type of damage caused by the fixed wall in the component to be detected. In this embodiment, the damage type can be divided into three major categories and seven sub-categories, specifically including: missing (complete missing, partial missing), crack (surface crack, internal hidden crack), wear (slight wear, moderate wear, severe wear), etc., and the specific type of each damage is clearly marked.

[0107] refer to Figure 5 , Figure 5 A schematic diagram of a fixed wall breach identified by an exemplary embodiment of this disclosure is shown. Figure 5 The diagram shows the identified damage to the fixed wall according to the damage identification steps described above in this disclosure. From... Figure 5 As can be seen, wear appears on the back side of the fixed wall, meaning there is a worn area 510 on the fixed wall.

[0108] Furthermore, based on the fused feature vectors, the spectral feature difference vectors and spatial morphological damage levels can be compared with their respective reference difference vectors and preset damage thresholds. Specifically, the comparison process involves calculating the vector distance between the spectral feature difference vector and the reference difference vector, and the damage difference between the spatial morphological damage level and the preset damage threshold. The damage level is determined based on the comparison results (e.g., vector distance and damage difference), which can be used to measure the degree of damage to the component under inspection. For example, damage levels can include mild damage (spectral reflectance fluctuation 0.05~0.1), moderate damage (spectral reflectance fluctuation 0.1~0.2), and severe damage (spectral reflectance fluctuation >0.2). Further, based on the spectral feature differences, the cause of defects can be preliminarily determined (e.g., wear is caused by friction, cracks are caused by stress), and this information is noted in the inspection results to provide a basis for subsequent maintenance.

[0109] In other exemplary embodiments of this disclosure, damage levels can be further classified according to specific needs. This disclosure does not impose any special limitation on the specific number of damage levels. After determining the damage type and damage level, early warning can be issued based on the damage identification results, such as issuing early warning for latent defects. For latent defects that do not meet the failure criteria (such as internal latent cracks or slight wear), an early warning mechanism is automatically triggered, recording the defect location (accuracy ≤ 0.01 mm), defect type, and development trend. The early warning record is updated every preset time period, and the early warning information is simultaneously transmitted to the linkage feedback module and the storage module.

[0110] In one exemplary embodiment of this disclosure, step S240, determining the remaining life prediction result of the component to be tested based on the health status index, includes: acquiring the health status index sequence of the component to be tested within a specified time period; determining the first component failure threshold and component auxiliary features corresponding to the component to be tested; determining the remaining life prediction result based on the first component failure threshold, component auxiliary features, and health status index sequence; and determining the warning level of the component to be tested based on the comparison between the remaining life prediction result and the remaining life threshold, wherein the warning level is used to trigger a specified level of component remaining life warning prompt.

[0111] This disclosure proposes an implementation scheme for determining the remaining useful life (RUL) prediction result of a component under test based on the Health Status Index (HSI). The core of the RUL prediction is that the material properties determine the HSI parameters, and the HSI sequence determines the remaining useful life (RUL). This embodiment provides a specific implementation method for calculating RUL based on HSI: In the RUL calculation process, the component auxiliary features such as multispectral features, spatial features, and environmental factors of the component under test are combined, and the entire process analysis of material adaptation → damage identification → HSI quantification → RUL prediction is realized based on the HSI-RUL calculation chain.

[0112] The specific principle of this calculation process is as follows: The lifespan of the tray wall is essentially a balance between the "material tolerance limit" and the "external damage effect". HSI is the quantitative manifestation of this balance. The high temperature resistance, wear resistance, rigidity, toughness and other characteristics of different engineering plastics determine their HSI failure threshold (e.g., PEEK has strong tolerance, HSI threshold = 0.88; PS has weak tolerance, HSI threshold = 0.65). The high temperature environment, automated gripping friction, chip collision, dust contamination and other external effects in the chip back-end testing will continuously cause damage (explicit / implicit), causing the HSI to continuously decrease. When the HSI reaches the failure threshold for the first time, the wall fails, and the RUL at this time is the remaining lifespan.

[0113] The core logic of remaining service life analysis is as follows: Spectral and image data of the fixed wall are acquired through a multispectral acquisition module (optimized by an airflow-assisted module for high accuracy). This data is then combined with a material adaptive calibration algorithm to identify the plastic material of the tray, determining its material ID, HSI failure threshold, and w. s w p Weights; latent and overt damage features are extracted using a two-branch algorithm and substituted into the HSI calculation formula to obtain the current HSI value; the HSI sequence is predicted based on the LSTM model to find the number of steps k in which the HSI first reaches the failure threshold; finally, the remaining service life of the component under test is obtained by substituting it into the RUL calculation formula.

[0114] Through the above steps, this disclosure realizes a material-aware HSI-RUL calculation chain, which uses the Health Status Index (HSI) as the core link to connect material identification, damage extraction and life prediction, and realizes an explicit and interpretable path of "material characteristics → HSI quantification → RUL calculation". RUL = (number of steps when HSI first reaches the material failure threshold - 1) × 0.5 hours; where "number of steps" corresponds to the number of cycles of use of the Tray wall (for each cycle of use, data is collected once and HSI is calculated once to form an HSI sequence), and Δt = 0.5 hours / cycle is the actual measured calibration value of the production line, ensuring that the RUL calculation has a clear basis.

[0115] This disclosure presents a Physics-Guided Long Short-Term Memory Network (PG-LSTM). Its inputs are the HSI value of the current cycle, material ID, cumulative usage count, and operating parameters (temperature, frequency). The output is the predicted HSI values ​​for the next N cycles. The LSTM's input vector X... Yes: [HSI] [Material ID, Cumulative usage times t, Current temperature, Average daily usage frequency]; The output sequence Y is: [ , , ,…, And construct a monotonic constraint loss function, as shown in Formula 1: (Formula 1) Where L is the loss function, It can be the actual HSI value of the component under test at the cumulative number of uses t+i; It can be the HSI prediction value of the component to be tested at the cumulative number of uses t+i; γ is the monotonicity penalty coefficient, which is used to balance the weight of prediction accuracy and physical consistency constraints. The non-monotonic decrease that appears in the second penalty sequence can force the prediction result to conform to the physical law of "irreversible damage".

[0116] After obtaining the predicted value of the component to be tested after t+i uses based on Formula 1 above, the RUL can be solved analytically. For example, after obtaining the prediction sequence, the first time the value falls below the material-specific failure threshold can be found analytically. Specifically, as shown in Formula 2: K = min{t + i| < } RUL = (K-1-t)×Δt (Formula 2) Where K can be the number of times the component to be tested can be used repeatedly, or the number of steps; It could be the failure threshold corresponding to the material of the component to be tested; This can be the actual HSI value of the component under test at the cumulative usage count t+i; Δt = 0.5 hours / cycle, where Δt refers to the average time consumed per cycle as measured and statistically analyzed on the production line. In the RUL calculation formula, Δt converts the remaining available cycle count K into the remaining hours RUL. Furthermore, in LSTM time series prediction, mapping the cumulative usage count in the input features to the time dimension helps the model perceive the relationship between the actual aging progress of the tray and the production line cycle time, ensuring consistency in predictions across different production line cycle times.

[0117] Formula 2 can also be expressed as: RUL = (number of steps when HSI first reaches the material failure threshold - 1) × 0.5 hours; where the number of steps corresponds to the number of cycles of use of the Tray fixed wall (each cycle uses 1 time, collects 1 data, calculates 1 HSI, and forms an HSI sequence), and Δt = 0.5 hours / cycle is the actual measured calibration value of the production line.

[0118] This disclosure provides eight commonly used engineering plastics for trays in chip back-end testing. The material properties of different trays directly determine the damage type, development speed, and lifespan of the mounting wall, and are the core basis for HSI calculation, weight allocation, and failure threshold setting. The specific classification and characteristics are as follows: PPS is the preferred material for packaging and testing trays, with core advantages including high temperature resistance (long-term 260℃+, reflow soldering resistant), low moisture absorption, dimensional stability, anti-static properties, chemical resistance, and reusability. Main applications include packaging, high-temperature testing, automated production lines, and JEDEC standard trays. Damage characteristics include wear resistance and minimal deformation; the main damage is material fatigue under long-term high temperatures and slight wear. It has a long lifespan (can be recycled 1000+ times under normal operating conditions). The HSI failure threshold is set at 0.85 (to match its high tolerance characteristics).

[0119] The core advantages of PPE / MPPE are heat resistance (180~200℃), low warpage, high cleanliness, and a lifespan approximately 28% longer than PS. Main applications include cleanrooms, automated packaging, and mid-to-high-end general-purpose trays. Damage characteristics include moderate impact resistance, with primary damage being surface wear and minor cracks. The lifespan is moderate (can be recycled 800~1000 times under normal operating conditions). The HSI failure threshold is set at 0.78.

[0120] PES / PSU is a mainstream material for high-end testing and precision manufacturing processes. Its core advantages are heat resistance (-40℃~180℃), chemical stability, high dimensional accuracy (±0.03mm), and low precipitation. Main applications include high-temperature testing of wafers / chips, high-precision packaging, and high-cleanliness processes. Its damage characteristics include extremely strong dimensional stability, with the main damage being internal material fatigue and latent cracks. It has a long lifespan (900~1100 cycles under normal operating conditions). The HSI failure threshold is set at 0.82.

[0121] LCP is a specialized material for advanced packaging. Its core advantages include ultra-high temperature resistance (280℃~320℃), extremely low warpage, high rigidity, and suitability for ultra-thin / high-density cavities. Main applications include advanced packaging such as chiplets, 2.5D / 3D, and fan-out packaging, as well as ultra-thin chip carriers. Damage characteristics include high rigidity and relatively high brittleness, with the main damage being surface cracks and edge breakage. It has a medium lifespan (700~900 cycles under normal operating conditions) and an HSI failure threshold of 0.75.

[0122] PEEK is a material specifically designed for extreme environments. Its core advantages include ultra-high temperature resistance (long-term 260℃, peak 310℃), chemical resistance, wear resistance, high strength, and low exudation. Main applications include extreme high temperature testing, high-reliability automotive-grade chips, and long-term cyclic use. Its damage characteristics include extremely high wear resistance and strength. The main damage is material aging and latent fatigue under long-term extreme environments. It has the longest lifespan (can be used more than 1200 times under normal operating conditions). The HSI failure threshold is set at 0.88.

[0123] ABS / reinforced ABS is a mid-to-low-end general-purpose material. Its core advantages are impact resistance, moderate cost, good moldability, and controllable electrostatic discharge (ESD). Main applications include consumer electronics, automotive electronics, mid-to-low-end general-purpose trays, and automated gripping. Damage characteristics include strong impact resistance, easy wear, and the main damage is surface wear and scratches. It has a relatively short life cycle (500-700 cycles under normal working conditions). The HSI failure threshold is set at 0.70.

[0124] PS is a low-cost turnover material with core advantages of low cost, easy molding, and lightweight. Its main applications are disposable / low-cost turnover, transportation of mid-to-low-end chips, and non-high-temperature processes. Its damage characteristics are high brittleness and poor heat resistance (not resistant to high-temperature testing). The main damage is cracks and breakage. It has the shortest life cycle (it can be recycled 300-500 times under normal working conditions, mostly for disposable or short-term turnover). The HSI failure threshold is set at 0.65 (to match its low tolerance characteristics).

[0125] PP / PC are general-purpose turnover materials. PP has good toughness, antistatic properties, chemical resistance, and low cost, and is used for general turnover and blister trays. Its damage characteristics include easy deformation and surface wear, and a medium to short lifespan (600-800 cycles under normal working conditions). The HSI failure threshold is set at 0.68. PC has high strength, transparency, and impact resistance, and is used in scenarios requiring visual inspection and high reliability turnover. Its damage characteristics include easy scratching, moderate brittleness, and a medium lifespan (700-900 cycles under normal working conditions). The HSI failure threshold is set at 0.72.

[0126] In one exemplary embodiment of this disclosure, a pre-constructed component anomaly sample set is obtained. The component anomaly sample set includes multiple anomaly labeled samples, and the anomaly labeled samples include multiple samples whose prediction deviations of remaining lifetime prediction results reach a preset deviation threshold. When it is detected that the model update condition is met, the component lifetime prediction model is updated based on the anomaly labeled samples to obtain the updated component lifetime prediction model.

[0127] This disclosure incorporates a small-sample adaptive optimization process during component damage assessment to ensure continuous improvement in model adaptability and recognition accuracy. During damage assessment, the material adaptive adjustment unit automatically identifies the tray material (plastic, ceramic, etc.) and adaptively adjusts the spectral feature extraction parameters and model weights based on the adjustment coefficients corresponding to the material characteristics. For example, the adjustment coefficient for plastic is 0.9, and for ceramic it is 1.1. The adjustment coefficient can represent the degree of deviation of the material characteristics from the default baseline value; that is, the adjustment coefficient is a fine-tuning factor relative to the baseline value. It is used for soft adjustments when the material recognition confidence is not 100%, or for interpolation transitions when the material is at the boundary between two classes.

[0128] During the detection process, samples identified as abnormal (moderate or severe defects) are collected in real time, and the defect type and level are automatically labeled and added to the sample library. The model adopts a self-supervised optimization method, and updates the model when the model update conditions are met. The model update conditions can be that the number of abnormal samples in the abnormal sample set reaches a specified number, or the time since the last model update reaches a specified duration. For example, every time a specified number of abnormal samples are accumulated (e.g., 10), or every preset time (every 24 hours), the model automatically starts the self-supervised optimization process, updates the model parameters based on the newly added samples, without the need for frequent manual labeling and intervention, and continuously improves the model's recognition accuracy and adaptability.

[0129] Based on a few-shot transfer learning algorithm, labeled samples (30-80 images / class) of each defect are collected as an anomaly sample set. Anomaly labeled samples can be those whose prediction deviation from the model's prediction result (i.e., remaining lifetime prediction result) reaches a preset deviation threshold. The anomaly sample set can be a dataset composed of multiple anomaly labeled samples, which can be anomaly samples identified during historical detection processes. When a tray is ultimately replaced or confirmed to be faulty on the production line, this disclosure compares the remaining lifetime prediction result of the component at that time with the actual failure result. If the difference between the predicted remaining lifetime and the actual remaining lifetime exceeds a preset deviation threshold (e.g., 5 hours), the system determines that the data corresponding to that prediction is an anomaly sample.

[0130] Key characteristics of anomaly-labeled samples include: First, they carry true labels. Because anomaly-labeled samples correspond to trays that have completed their full lifespan, the system knows which cycle the sample actually failed in, allowing it to calculate the prediction bias at that time. Second, their value lies in including degradation modes outside the model's current capability boundaries. Normal samples are those the model can accurately predict, while anomaly samples are those the model cannot accurately predict; the latter is the data most needed to improve model performance. Third, they are material-specific. Anomaly samples are labeled with their material type, such as PPS or PS, because different materials have different causes of prediction bias. PPS might indicate changes in high-temperature fatigue patterns, while PS might indicate unexpected mechanical impact cracks.

[0131] An abnormal sample set was input into the dual-branch deep learning model for training, with 100 iterations and a learning rate of 0.001. After training, the data was imported into the inference unit of the data processing module, and the weight allocation rules of the spatial-spectral attention fusion module were configured (surface defect spatial feature weight 0.7, latent defect spectral feature weight 0.8). The defect fine classification criteria, level judgment threshold, and warning parameters were initialized, and the model was debugged to ensure that the model's recognition accuracy was ≥99.8%. The model can simulate trays with different defect types and materials to test the linkage effect of each module, including the coordination of multispectral acquisition and supplementary lighting, positioning accuracy, data transmission speed, linkage feedback response (audio-visual alarm, conveyor line shutdown / diversion), and data storage function, ensuring that there are no abnormalities in each link.

[0132] In the few-sample adaptive optimization scheme, material adaptive adjustment addresses the adaptation problem of known materials. Abnormal sample collection and model self-supervised optimization address the model's degradation over time and its evolution when encountering new damage patterns. These steps work together to ensure that the system can cover the eight known materials and autonomously improve accuracy during operation, forming a complete few-sample adaptive optimization mechanism.

[0133] In summary, the remaining lifetime prediction method for chip trays disclosed herein involves acquiring data of the component to be inspected, including spatial image data and multi-band spectral data of the component; performing feature extraction processing on the multi-band spectral data and spatial image data to obtain spectral feature extraction results and spatial feature extraction results; performing feature fusion and damage identification processing on the spectral feature extraction results and spatial feature extraction results to obtain damage identification results and a health status index of the component to be inspected; and determining the remaining lifetime prediction result of the component to be inspected based on the health status index. On the one hand, during the data acquisition stage, both spatial image data and multi-band spectral data of the component to be inspected are acquired simultaneously, and the obtained spectral feature extraction results and spatial feature extraction results are used together in the lifetime prediction process, improving the comprehensiveness of feature extraction. On the other hand, based on the spectral feature extraction results and spatial feature extraction results, both latent and overt damage to the component to be inspected can be considered. Furthermore, combining the health status index to quantify the health level of the component to be inspected can improve the prediction accuracy of the component's remaining lifetime and increase detection efficiency.

[0134] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0135] Furthermore, in this example embodiment, a device for predicting the remaining lifetime of a chip tray is also provided. (Reference) Figure 6 The remaining life prediction device 600 for the chip tray may include: a component data acquisition module 610, a feature extraction module 620, a health status index determination module 630, and a remaining life prediction module 640.

[0136] Specifically, the component data acquisition module 610 is used to acquire data of the component to be tested, including spatial image data and multi-band spectral data of the component to be tested; the feature extraction module 620 is used to perform feature extraction processing on the multi-band spectral data and spatial image data to obtain spectral feature extraction results and spatial feature extraction results; the health status index determination module 630 is used to perform feature fusion and damage identification processing on the spectral feature extraction results and spatial feature extraction results to obtain damage identification results and health status index of the component to be tested; and the remaining life prediction module 640 is used to determine the remaining life prediction result of the component to be tested based on the health status index.

[0137] In one exemplary embodiment of this disclosure, the component data acquisition module 610 includes a component data acquisition unit, configured to: trigger a data acquisition signal in response to the arrival of the component to be inspected at the inspection station; perform positioning processing on the component to be inspected based on the data acquisition signal, so that the positioning accuracy of the component to be inspected is within a specified accuracy threshold range; acquire spatial images of the component to be inspected to obtain spatial image data; and acquire multi-band spectral images of the component to be inspected to obtain multi-band spectral data.

[0138] In one exemplary embodiment of this disclosure, the component data acquisition unit includes a spectral data acquisition unit, used for: determining image acquisition environment parameters and the material type of the component to be tested; determining spectral image acquisition parameters corresponding to different bands based on the image acquisition environment parameters and the material type; acquiring images of the component to be tested in different bands based on the spectral image acquisition parameters of each band to obtain a multi-band panoramic spectral image; and performing image preprocessing on the multi-band panoramic spectral image to obtain multi-band spectral data.

[0139] In one exemplary embodiment of this disclosure, the spectral data acquisition unit includes a data preprocessing subunit, used for: performing contour recognition and region segmentation processing on the multi-band panoramic spectral image to obtain a multi-band fixed wall image; performing image denoising processing on the fixed wall image of each band to obtain a multi-band denoised fixed wall image; performing light intensity difference elimination processing and pixel alignment processing on the multi-band denoised fixed wall image to obtain a multi-band aligned fixed wall image; determining the reflectance difference value corresponding to different bands, and performing background separation processing on the multi-band aligned fixed wall image based on the reflectance difference value to obtain multi-band spectral data, wherein the multi-band spectral data is a standardized fixed wall image after format conversion processing.

[0140] In one exemplary embodiment of this disclosure, the feature extraction module 620 includes a feature extraction unit, configured to: acquire a pre-trained component life prediction model, the component life prediction model including a spectral feature extraction branch and a spatial feature extraction branch; perform spectral feature extraction processing on multi-band spectral data based on the spectral feature extraction branch to obtain spectral feature extraction results; and perform spatial feature extraction processing on spatial image data based on the spatial feature extraction branch to obtain spatial feature extraction results.

[0141] In one exemplary embodiment of this disclosure, the multi-band spectral data includes multi-band spectral feature blocks. The spectral feature extraction branch includes a convolutional layer, an attention module, and an output branch. The feature extraction unit includes a spectral feature extraction subunit, used for: performing spectral feature extraction processing on the multi-band spectral feature blocks for different bands by the spectral feature extraction branch to obtain spectral features for each band; performing feature enhancement processing on the spectral features of each band by the attention module to obtain spectral enhancement features for each band; performing feature fusion processing on the spectral enhancement features of multiple bands to obtain a latent damage feature map; and outputting a damage cause probability vector and an internal anomaly score of the component based on the latent damage feature map by the output branch. The damage cause probability vector includes the damage type of the component to be detected.

[0142] In one exemplary embodiment of this disclosure, the spatial feature extraction branch includes an encoder, a decoder, and an output head. The feature extraction unit includes a spatial feature extraction subunit, used for: performing multi-level encoding processing on spatial image data by the encoder to obtain an encoded feature map, the encoded feature map including multi-level encoded features, the encoded feature map being the spatial features of a fixed wall; determining the deepest level encoded features from the multi-level encoded features; performing upsampling decoding operations layer by layer by the decoder starting from the deepest level encoded features, and performing feature fusion processing with the shallow detail features corresponding to each level to obtain a decoded feature map; and outputting a visible damage segmentation map and a quantization index vector by the output head based on the decoded feature map.

[0143] In one exemplary embodiment of this disclosure, the component life prediction model further includes a feature fusion module, and the health status index determination module 630 includes a health status index determination unit, used for: determining the material type of the component to be tested; determining the material calculation parameters and feature fusion weights corresponding to the component to be tested based on the material type; performing feature fusion processing on the spectral feature extraction results and spatial feature extraction results based on the feature fusion weights to obtain a fused feature vector; performing damage identification processing based on the fused feature vector to obtain a damage identification result; and determining the health status index based on the material calculation parameters and the fused feature vector.

[0144] In one exemplary embodiment of this disclosure, the damage identification result includes damage type and damage level. The health status index determination unit includes a damage result determination subunit, which is used to: perform damage type identification processing on the fused feature vector to obtain the damage type, which includes missing damage, crack damage, and wear damage; determine the spectral feature difference vector and spatial morphological damage degree value of the component to be detected based on the fused feature vector; determine the vector distance between the spectral feature difference vector and the reference difference vector; determine the damage difference between the spatial morphological damage degree value and the preset damage threshold; and determine the damage level based on the vector distance and the damage difference.

[0145] In one exemplary embodiment of this disclosure, the remaining lifetime prediction module 640 includes a remaining lifetime prediction unit, configured to: acquire a health status index sequence of the component to be tested within a specified time period; determine the first component failure threshold and component auxiliary features corresponding to the component to be tested; determine the remaining lifetime prediction result based on the first component failure threshold, component auxiliary features, and health status index sequence; and determine the warning level of the component to be tested based on the comparison between the remaining lifetime prediction result and the remaining lifetime threshold, wherein the warning level is used to trigger a specified level of component remaining lifetime warning prompt.

[0146] In one exemplary embodiment of this disclosure, the chip tray remaining lifetime prediction device 600 further includes a model update module, configured to: acquire a pre-built component anomaly sample set, the component anomaly sample set including multiple anomaly labeled samples, the anomaly labeled samples including multiple samples whose prediction deviation of remaining lifetime prediction results reaches a preset deviation threshold; when a model update condition is detected, update the component lifetime prediction model based on the anomaly labeled samples to obtain an updated component lifetime prediction model.

[0147] The specific details of the virtual modules of the remaining lifetime prediction devices for each of the aforementioned chip trays have been described in detail in the corresponding remaining lifetime prediction methods for the chip trays. For any undisclosed details, please refer to the implementation methods in the method section, and therefore will not be repeated here.

[0148] It should be noted that although several modules or units of the chip tray remaining lifetime prediction device have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0149] An exemplary embodiment of this disclosure also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the aforementioned method for predicting the remaining lifetime of a chip tray.

[0150] In one implementation, the computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing the computer program. (See reference...) Figure 7 , Figure 7The schematic diagram illustrates a computer-readable storage medium 700 according to an exemplary embodiment of the present disclosure. The computer-readable storage medium 700 can be a storage medium based on signals such as electrical, magnetic, optical, electromagnetic, or infrared radiation, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory, hard disk drive (HDD), solid-state drive (SSD), etc. Exemplarily, a computer program product can be implemented as a non-volatile storage medium storing a computer program, such as read-only memory, NAND flash memory, etc.

[0151] In one implementation, the computer program product can be an intangible product containing a computer program. For example, the computer program product can be implemented as a virtual digital product, such as an executable file, installation package, or other digital file storing the computer program.

[0152] Computer program code can be written in one or more programming languages. Examples of programming languages ​​include C, Java, and C++. Program code can execute entirely on the user's computing device, partially on the user's computing device, or as a standalone software package. It can also execute partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, such as a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via an internet connection provided by a mobile network operator).

[0153] Computer programs can be carried or transmitted via signals such as electrical, magnetic, optical, electromagnetic, and infrared rays. Electronic devices can convert signals carrying computer programs into digital signals, thereby running the computer programs. When a computer program runs on an electronic device, its code is used to cause the electronic device to execute (more specifically, the processor of the electronic device to execute) the method steps of various exemplary embodiments of this disclosure, such as the remaining lifetime prediction method for the chip tray described above.

[0154] Exemplary embodiments of this disclosure also provide an electronic device, which may include a processor and a memory. The memory stores executable instructions of the processor, such as a computer program. The processor executes the executable instructions to perform the method steps of various exemplary embodiments of this disclosure. Furthermore, the electronic device may also include a display for displaying a graphical user interface.

[0155] The following is for reference. Figure 8 The electronic device is illustrated by way of a general-purpose computing device. It should be understood that... Figure 8The electronic device 800 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0156] like Figure 8 As shown, the electronic device 800 may include: a processor 810, a memory 820, a bus 830, an I / O (input / output) interface 840, a network adapter 850, and a display 860.

[0157] Memory 820 may include volatile memory, such as RAM 821 and cache unit 822, and may also include non-volatile memory, such as ROM 823. Memory 820 may also include one or more program modules 824, including but not limited to: operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. For example, program module 824 may include the modules in the above-described device.

[0158] The processor 810 may include one or more processing units, such as an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor, and / or an NPU (Neural-Network Processing Unit).

[0159] The processor 810 can be used to execute executable instructions stored in the memory 820, such as the remaining lifetime prediction method of the chip tray described above.

[0160] Bus 830 is used to connect different components of electronic device 800 and may include data bus, address bus and control bus.

[0161] Electronic device 800 can communicate with one or more external devices 900 (such as keyboard, mouse, external controller, etc.) through I / O interface 840.

[0162] Electronic device 800 can communicate with one or more networks via network adapter 850. For example, network adapter 850 can provide mobile communication solutions such as 3G / 4G / 5G, or wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication. Network adapter 850 can communicate with other modules of electronic device 800 via bus 830.

[0163] The electronic device 800 can display a graphical user interface via a display 860, such as an interface that displays the damage type of the component to be tested, the remaining life prediction results, and warning information.

[0164] although Figure 8 As not shown in the diagram, other hardware and / or software modules may also be configured in the electronic device 800, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) systems, tape drives, and data backup storage systems.

[0165] As can be seen from the above, the technical solutions disclosed herein can be implemented as methods, apparatus, systems, computer program products, storage media, electronic devices, etc. Those skilled in the art will understand that various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which may be referred to as "circuit," "module," or "system," respectively.

[0166] It should be understood that this disclosure is not limited to the specific methods, steps, or structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. Those skilled in the art will readily conceive of other embodiments based on the specific implementations provided in this disclosure. Therefore, the specific implementations provided in this disclosure are merely exemplary, and the scope and spirit of this disclosure are indicated by the claims, and should cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary technical means in the art not disclosed in this disclosure.

Claims

1. A method for predicting the remaining lifetime of a chip tray, characterized in that, include: Acquire data of the component to be tested, wherein the data of the component to be tested includes spatial image data and multi-band spectral data of the component to be tested; Feature extraction processing is performed on the multi-band spectral data and the spatial image data to obtain spectral feature extraction results and spatial feature extraction results; The spectral feature extraction results and the spatial feature extraction results are subjected to feature fusion and damage identification processing to obtain the damage identification results and health status index of the component to be detected; The remaining life prediction result of the component to be tested is determined based on the health status index.

2. The method according to claim 1, characterized in that, The acquisition of data of the component to be tested includes: A data acquisition signal is triggered in response to the arrival of the component to be tested at the testing station; The detection component is positioned based on the data acquisition signal, so that the positioning accuracy of the detection component is within a specified accuracy threshold range. Spatial image acquisition is performed on the component to be detected to obtain spatial image data; Multi-band spectral images are acquired on the component to be tested to obtain the multi-band spectral data.

3. The method according to claim 2, characterized in that, The step of acquiring multi-band spectral images of the component to be detected to obtain the multi-band spectral data includes: Determine the image acquisition environment parameters and the material type of the component to be detected; Based on the image acquisition environment parameters and the material type, determine the spectral image acquisition parameters corresponding to different wavebands; Based on the spectral image acquisition parameters of each band, images of the component to be detected are acquired in different bands to obtain a multi-band panoramic spectral image. The multi-band panoramic spectral image is preprocessed to obtain the multi-band spectral data.

4. The method according to claim 3, characterized in that, The step of preprocessing the multi-band panoramic spectral image to obtain the multi-band spectral data includes: Contour recognition and region segmentation are performed on the multi-band panoramic spectral image to obtain a multi-band fixed wall image; The images of the fixed wall in each of the aforementioned bands are subjected to image denoising processing to obtain multi-band denoised fixed wall images; The multi-band denoised fixed wall image is subjected to light intensity difference elimination processing and pixel alignment processing to obtain a multi-band aligned fixed wall image. The reflectance difference corresponding to different bands is determined, and the background separation processing is performed on the multi-band aligned fixed wall image based on the reflectance difference to obtain the multi-band spectral data. The multi-band spectral data is a standardized fixed wall image after format conversion processing.

5. The method according to claim 1, characterized in that, The step of performing feature extraction processing on the multi-band spectral data and the spatial image data to obtain spectral feature extraction results and spatial feature extraction results includes: Obtain a pre-trained component life prediction model, which includes a spectral feature extraction branch and a spatial feature extraction branch; Based on the spectral feature extraction branch, spectral feature extraction processing is performed on the multi-band spectral data to obtain the spectral feature extraction result; Based on the spatial feature extraction branch, spatial feature extraction processing is performed on the spatial image data to obtain the spatial feature extraction result.

6. The method according to claim 5, characterized in that, The multi-band spectral data includes multi-band spectral feature blocks. The spectral feature extraction branch includes a convolutional layer, an attention module, and an output branch. The spectral feature extraction process based on the spectral feature extraction branch on the multi-band spectral data yields the spectral feature extraction result, including: The convolutional layer of the spectral feature extraction branch performs spectral feature extraction processing on the multi-band spectral feature block for different bands to obtain the spectral features of each band. The attention module performs feature enhancement processing on the spectral features of each band based on the material type of the component to be detected, thereby obtaining the spectral enhancement features of each band. The spectral enhancement features of multiple bands are fused to obtain a latent damage feature map. Based on the latent damage feature map, the output branch outputs a damage cause probability vector and an internal anomaly score of the component. The damage cause probability vector includes the damage type of the component to be detected.

7. The method according to claim 5, characterized in that, The spatial feature extraction branch includes an encoder, a decoder, and an output head. The spatial feature extraction process performed on the spatial image data based on the spatial feature extraction branch to obtain the spatial feature extraction result includes: The encoder performs multi-level encoding processing on the spatial image data to obtain an encoded feature map, which includes multiple levels of encoded features and represents the spatial features of a fixed wall. The deepest level coding feature is determined from the coding features of the multiple levels; The decoder starts from the deepest level encoded features and performs upsampling decoding operations layer by layer. The decoded features are then fused with the shallow detail features corresponding to each level to obtain a decoded feature map. The output head outputs an explicit damage segmentation map and a quantization index vector based on the decoded feature map.

8. The method according to claim 5, characterized in that, The component life prediction model further includes a feature fusion module, which performs feature fusion and damage identification processing on the spectral feature extraction results and the spatial feature extraction results to obtain the damage identification results and health status index of the component to be detected, including: Determine the material type of the component to be detected, and determine the material calculation parameters and feature fusion weights corresponding to the component to be detected based on the material type; Based on the feature fusion weights, the spectral feature extraction results and the spatial feature extraction results are subjected to feature fusion processing to obtain a fused feature vector; Damage identification processing is performed based on the fused feature vector to obtain the damage identification result; The health status index is determined based on the material calculation parameters and the fused feature vector.

9. The method according to claim 8, characterized in that, The damage identification result includes damage type and damage level. The damage identification processing based on the fused feature vector to obtain the damage identification result includes: The fused feature vector is subjected to damage type identification processing to obtain the damage type, which includes missing damage, crack damage and wear damage. The spectral feature difference vector and the spatial morphological damage level value of the component to be detected are determined based on the fused feature vector. Determine the vector distance between the spectral feature difference vector and the reference difference vector, and determine the damage difference between the spatial morphological damage degree value and the preset damage threshold; The damage level is determined based on the vector distance and the damage difference.

10. The method according to claim 1 or 8, characterized in that, The step of determining the remaining lifespan prediction result of the component under test based on the health status index includes: Obtain the health status index sequence of the component to be detected within a specified time period; Determine the first-time component failure threshold and auxiliary features corresponding to the component to be detected; The remaining lifetime prediction result is determined based on the first component failure threshold, the component auxiliary features, and the health status index sequence; Based on the comparison between the remaining lifetime prediction result and the remaining lifetime threshold, the warning level of the component to be tested is determined, and the warning level is used to trigger a warning prompt for the remaining lifetime of the component at the specified level.

11. The method according to claim 5 or 8, characterized in that, The method further includes: Obtain a pre-constructed set of component anomaly samples, which includes multiple anomaly labeled samples. The anomaly labeled samples are those whose prediction deviation of the remaining lifetime prediction result reaches a preset deviation threshold. When the model update condition is met, the component life prediction model is updated based on the abnormal labeled samples to obtain the updated component life prediction model.

12. A device for predicting the remaining lifespan of a chip tray, characterized in that, include: The component data acquisition module is used to acquire data of the component to be detected, which includes spatial image data and multi-band spectral data of the component to be detected. The feature extraction module is used to perform feature extraction processing on the multi-band spectral data and the spatial image data to obtain spectral feature extraction results and spatial feature extraction results; The health status index determination module is used to perform feature fusion and damage identification processing on the spectral feature extraction results and the spatial feature extraction results to obtain the damage identification results and health status index of the component to be detected. The remaining life prediction module is used to determine the remaining life prediction result of the component to be tested based on the health status index.

13. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the chip tray remaining lifetime prediction method as described in any one of claims 1 to 11.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for predicting the remaining lifetime of the chip tray as described in any one of claims 1 to 11.

Images