A method and system for infrared imaging multimodal feature fusion detection of a transparent component

Abstract

The present disclosure relates to the field of image detection, and specifically relates to a method and system for multi-modal feature fusion detection of infrared imaging of a transparent component. The method includes the following steps: obtaining a preliminary inspection infrared image of the transparent component; based on the preliminary inspection infrared image, using the YOLOv8 algorithm to preliminarily classify the defects of the transparent component and obtain the area where the suspected defects are located; obtaining a refined inspection infrared image of the area where the suspected defects of the transparent component are located; based on the refined inspection infrared image, using a multi-modal feature fusion algorithm to identify the final defect type. The present disclosure realizes the alignment of different modal features in the time dimension and the matching in the information dimension, and then realizes the fusion of multi-modal features, and the detection result is more accurate.

Description

Technical Field

[0001] The present disclosure relates to the field of image detection, and particularly to a method and system for multi-modal feature fusion detection of infrared imaging of transparent components. Background Art

[0002] In the field of detection of transparent components, the detection process has its particularities, mainly reflected in the high light transmittance and reflectivity of the transparent components. Traditional machine vision detection methods usually rely on point, line or surface light sources to irradiate the object to be detected, capture defect features through the irradiation of the light source, and use algorithms for image processing and analysis. However, for transparent components, due to their material properties, most of the light will directly penetrate, resulting in unclear defect imaging features and being difficult to be accurately identified. In addition, the practice of increasing the light intensity to improve the imaging effect may cause excessive reflection, which further affects the quality of optical imaging.

[0003] Due to the above problems, traditional machine vision detection technologies are difficult to meet the high-precision online application standards in the defect detection of transparent components. At present, the detection of transparent components mainly relies on manual experience and requires multiple complex processes to ensure the detection quality, which is not only inefficient but also easily interfered by human factors. Summary of the Invention

[0004] The present disclosure provides a method and system for multi-modal feature fusion detection of infrared imaging of transparent components, which can solve at least one problem mentioned in the background art. To solve the above technical problems, the present disclosure provides the following technical solutions:

[0005] As an aspect of an embodiment of the present disclosure, there is provided a method for multi-modal feature fusion detection of infrared imaging of transparent components, characterized by including the following steps:

[0006] S10. Obtain the initial inspection infrared image of the transparent component;

[0007] S20. Based on the initial inspection infrared image, use the YOLOv8 algorithm to preliminarily classify the defects of the transparent component and obtain the area where the suspected defect is located;

[0008] S30. Obtain the refined inspection infrared image of the area where the suspected defect of the transparent component is located;

[0009] S40. Based on the refined inspection infrared image, use the multi-modal feature fusion algorithm to identify the final defect type.

[0010] Optionally, the initial inspection infrared image and / or the refined inspection infrared image are multiple photos taken by dynamic photographing multiple times.

[0011] Optionally, the preliminary classification includes the following: scratches, cracks, bubbles, inclusions, dirt, and no defects.

[0012] Optionally, the region is a rectangular region defined by the following data: the central X coordinate, the central Y coordinate, the bounding box width, and the bounding box height.

[0013] Optionally, the initial inspection infrared image includes r infrared imaging photos taken during the first dynamic photographing and m infrared imaging photos taken during the second dynamic photographing. Among them, the YOLOv8 algorithm is used to preliminarily classify the defects of the transparent component, including:

[0014] S201: Input the r infrared imaging photos taken during the first dynamic photographing into the YOLOv8 algorithm in sequence to obtain the class ID, central X coordinate, central Y coordinate, bounding box width, and bounding box height of the suspected defect;

[0015] S202: Count the class IDs of the suspected defects in the r infrared imaging photos respectively. If there are no suspected defects, the r infrared imaging photos taken during the current first dynamic photographing are detected as defect-free; if some or all have suspected defects, cluster and merge the central X coordinates and central Y coordinates of the suspected defect class IDs in the r infrared imaging photos to unify the information of the same suspected defect;

[0016] S203: Output the class ID, central X coordinate, central Y coordinate, bounding box width, and bounding box height of the suspected defect after clustering and merging;

[0017] S204: Process the m infrared imaging photos taken during the second dynamic photographing according to the steps of S201 - S203 to obtain the class ID, central X coordinate, central Y coordinate, bounding box width, and bounding box height of the suspected defect in the second dynamic photographing;

[0018] S205: Merge the class ID, central X coordinate, central Y coordinate, bounding box width, and bounding box height of the suspected defects in the first dynamic photographing and the second dynamic photographing to obtain the preliminary classification.

[0019] Optionally, the clustering and merging further includes the following steps: perform arithmetic mean operations on the central X coordinates and central Y coordinates of multiple adjacent 3 - 5 pixels, and take the maximum value of the bounding box width and bounding box height among them.

[0020] Optionally, based on the refined inspection infrared image, a multi-modal feature fusion algorithm is used to identify the final defect type, including:

[0021] For defects caused by static factors, wavelet transform is used to obtain high and low frequency components. At the same time, histogram of oriented gradients is used to statistically analyze the refined infrared image, and information with obvious gradient changes in the refined infrared image is obtained. Among them, the information with obvious gradient changes reflects the shape and details of the defects, and static features are obtained;

[0022] For the features of the refined infrared image caused by dynamic factors, a convolutional neural network is used to extract features from the refined infrared image, and an LSTM model is used to analyze the temporal correlation between adjacent key frames to obtain dynamic evolution features.

[0023] As another aspect of the embodiments of the present disclosure, a multi-modal feature fusion detection system for infrared imaging of a transparent component is provided, including:

[0024] A preliminary inspection infrared image acquisition unit for acquiring a preliminary inspection infrared image of the transparent component;

[0025] A preliminary classification unit, based on the preliminary inspection infrared image, using the YOLOv8 algorithm to preliminarily classify the defects of the transparent component and obtain the area where the suspected defects are located;

[0026] A refined inspection infrared image acquisition unit for acquiring a refined inspection infrared image of the area where the suspected defects of the transparent component are located;

[0027] A defect type acquisition unit, based on the refined inspection infrared image, using a multi-modal feature fusion algorithm to identify the final defect type.

[0028] As another aspect of the embodiments of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the multi-modal feature fusion detection method for infrared imaging of the transparent component is implemented.

[0029] As another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the multi-modal feature fusion detection method for infrared imaging of the transparent component is implemented.

[0030] Compared with the prior art, the present disclosure takes into account the characteristics of different modal features, such as different dimensions, scales, and spatio-temporal resolutions. It is impossible to directly fuse multi-modal features simply. It is necessary to consider not only the matching of information, but also the complementarity and collaboration of information. Therefore, the embodiments of the present disclosure achieve the alignment of different modal features in the time dimension and the matching in the information dimension, then realize the fusion of multi-modal features, and at the same time achieve the collaboration of features and the complementarity of information, making the detection result more accurate. Description of the Drawings

[0031] Figure 1It is a flowchart of the infrared imaging multimodal feature fusion detection method for the transparent component in Embodiment 1 of the present disclosure;

[0032] Figure 2 It is a schematic block diagram of the infrared imaging multimodal feature fusion detection method for the transparent component in Embodiment 1 of the present disclosure.

[0033] Figure 3 It is a schematic block diagram of the infrared imaging multimodal feature fusion detection system for the transparent component in Embodiment 2 of the present disclosure. Detailed implementation manners

[0034] Hereinafter, various exemplary embodiments, features and aspects of the present disclosure will be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements having the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0035] The special term "exemplary" here means "serving as an example, an embodiment or an illustration". Any embodiment described as "exemplary" here is not necessarily to be construed as superior to or better than other embodiments.

[0036] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following detailed implementation manners. Those skilled in the art should understand that the present disclosure can also be implemented without some specific details. In some instances, methods, means, elements and circuits well known to those skilled in the art are not described in detail so as to highlight the gist of the present disclosure.

[0037] It can be understood that the above-mentioned various method embodiments of the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not elaborate further.

[0038] In addition, the present disclosure also provides an infrared imaging multimodal feature fusion detection device, an electronic device, a computer-readable storage medium, and a program for the transparent component, all of which can be used to implement any one of the infrared imaging multimodal feature fusion detection methods provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be elaborated further.

[0039] The execution entity of the infrared imaging multi-modal feature fusion detection method for transparent components can be a computer or other devices capable of implementing the infrared imaging multi-modal feature fusion detection for transparent components. For example, the method can be executed by a terminal device, a server, or other processing devices. Among them, the terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementation manners, the infrared imaging multi-modal feature fusion detection method for transparent components can be implemented by a processor invoking computer-readable instructions stored in a memory.

[0040] Embodiment 1

[0041] This embodiment provides an infrared imaging multi-modal feature fusion detection method for transparent components, as Figure 1 shown, including the following steps:

[0042] S10. Obtain the initial inspection infrared image of the transparent component;

[0043] S20. Based on the initial inspection infrared image, use the YOLOv8 algorithm to preliminarily classify the defects of the transparent component and obtain the area where the suspected defects are located;

[0044] S30. Obtain the refined inspection infrared image of the area where the suspected defects of the transparent component are located;

[0045] S40. Based on the refined inspection infrared image, use the multi-modal feature fusion algorithm to identify the final defect type.

[0046] Based on the above configuration, the embodiments of the present disclosure achieve the alignment of different modal features in the time dimension and the matching in the information dimension, then realize the fusion of multi-modal features, and at the same time achieve the coordination of features and the complementarity of information.

[0047] The following will separately elaborate on each step of the embodiments of the present disclosure. As Figure 2 shown, it is a schematic block diagram of the infrared imaging multi-modal feature fusion detection method for transparent components in the embodiments of the present disclosure.

[0048] S10. Obtain the initial inspection infrared image of the transparent component;

[0049] Among them, infrared thermal imaging is a process in which infrared radiation emitted by an object is collected by a lens, screened by a grating, converted by a detector, and then processed by subsequent signal processing to finally generate an infrared thermal image. The initial inspection infrared image of the transparent component is obtained at the initial inspection station in the infrared imaging detection device. Among them, the initial inspection station has an infrared camera that can take pictures of the transparent component to obtain the initial inspection infrared image. Preferably, the initial inspection station can also perform thermal spraying on the transparent component to realize taking pictures after heating different transparent components;

[0050] Among them, the initial inspection infrared image is multiple photos obtained by the infrared camera taking dynamic pictures multiple times.

[0051] S20. Based on the initial inspection infrared image, use the YOLOv8 algorithm to preliminarily classify the defects of the transparent component and obtain the area where the suspected defect is located.

[0052] In some embodiments, the preliminary classification includes the following: scratches, cracks, bubbles, inclusions, dirt, and no defects.

[0053] Preferably, the area is a rectangular area, and the rectangular area is defined by the following data: center X coordinate, center Y coordinate, bounding box width, and bounding box height

[0054] In some embodiments, the initial inspection infrared image includes r infrared imaging photos taken in the first dynamic photographing and m infrared imaging photos taken in the second dynamic photographing. Among them, using the YOLOv8 algorithm to preliminarily classify the defects of the transparent component includes:

[0055] S201. Input the r infrared imaging photos taken in the first dynamic photographing into the YOLOv8 algorithm in sequence to obtain the category ID, center X coordinate, center Y coordinate, bounding box width, and bounding box height of the suspected defect;

[0056] S202. Count the category IDs of the suspected defects in the r infrared imaging photos respectively. If there are no suspected defects, the r infrared imaging photos taken in the current first dynamic photographing are detected as having no defects; if some or all have suspected defects, cluster and merge the center X coordinates and center Y coordinates of the suspected defect category IDs in the r infrared imaging photos to unify the information of the same suspected defect;

[0057] S203. Output the category ID, center X coordinate, center Y coordinate, bounding box width, and bounding box height of the suspected defect after clustering and merging;

[0058] S204. Process the m infrared imaging photos taken in the second dynamic photographing according to the steps of S201 - S203 to obtain the category ID, center X coordinate, center Y coordinate, bounding box width, and bounding box height of the suspected defect in the second dynamic photographing;

[0059] S205. Combine the category ID, center X coordinate, center Y coordinate, bounding box width, and bounding box height of the suspected defects in the first dynamic photographing and the second dynamic photographing to obtain a preliminary classification.

[0060] S30. Obtain a refined inspection infrared image of the area where the suspected defect of the transparent component is located;

[0061] Among them, the refined inspection infrared image of the transparent component is obtained by using the refined inspection station in the infrared imaging detection device. The refined inspection station has an infrared camera, which can take pictures of the position of the suspected defect obtained during the preliminary inspection of the transparent component to obtain a refined inspection infrared image. Preferably, the refined inspection infrared image is multiple photos obtained by the infrared camera through multiple dynamic photographings.

[0062] Preferably, the clustering and merging further includes the following steps: perform an arithmetic mean operation on the center X coordinates and center Y coordinates of multiple adjacent 3-5 pixels, and take the maximum value of the bounding box width and bounding box height among them.

[0063] S40. Based on the refined inspection infrared image, use a multi-modal feature fusion algorithm to identify the final defect type;

[0064] As Figure 2 shown, for defects caused by static factors, wavelet transform and high-frequency filtering are used to obtain high and low frequency components. At the same time, the histogram of oriented gradients is used to perform statistical analysis on the refined inspection infrared image to obtain information with obvious gradient changes in the refined inspection infrared image. Among them, the information with obvious gradient changes reflects the shape and details of the defect, and static features are obtained;

[0065] Among them, the obtained high and low frequency components are as follows:

[0066]

[0067] φ(t) is the low frequency component, is the high frequency component, k is the translation amount, h k belongs to the coefficients of the low-pass filter, g k belongs to the coefficients of the high-pass filter, and φ(2t - k) is the translated low frequency component.

[0068] For the characteristics of the refined inspection infrared image caused by dynamic factors, use a convolutional neural network to extract features from the refined inspection infrared image, use an LSTM model to analyze the temporal correlation between adjacent key frames, and obtain dynamic features. Then, perform alignment of multi-modal features and fusion of multi-modal features to finally realize the detection and identification of defects of the transparent component.

[0069] Embodiment 2

[0070] This embodiment provides a transparent component infrared imaging multimodal feature fusion detection device 100, as Figure 3 shown, including:

[0071] A preliminary inspection infrared image acquisition unit 1, which acquires a preliminary inspection infrared image of the transparent component;

[0072] A preliminary classification unit 2, which, based on the preliminary inspection infrared image, uses the YOLOv8 algorithm to preliminarily classify the defects of the transparent component and obtain the area where the suspected defects are located;

[0073] A refined inspection infrared image acquisition unit 3, which acquires a refined inspection infrared image of the area where the suspected defects of the transparent component are located;

[0074] A defect type acquisition unit 4, which, based on the refined inspection infrared image, uses a multimodal feature fusion algorithm to identify the final defect type.

[0075] In the preliminary inspection infrared image acquisition unit 1, infrared thermal imaging is a process in which infrared radiation emitted by an object passes through the collection of a lens, the screening of a grating, the conversion of a detector, and subsequent signal processing, and finally generates an infrared thermal image. The preliminary inspection infrared image of the transparent component is acquired using the preliminary inspection station in the infrared imaging detection device. Among them, the preliminary inspection station has an infrared camera, which can take a picture of the transparent component to obtain the preliminary inspection infrared image. Preferably, the preliminary inspection station can also perform thermal spraying on the transparent component to realize taking pictures after heating different transparent components;

[0076] Among them, the preliminary inspection infrared image is multiple photos obtained by the infrared camera taking dynamic photos multiple times.

[0077] In the preliminary classification unit 2, the preliminary classification includes the following: scratches, cracks, bubbles, inclusions, dirt, and no defects.

[0078] Preferably, the area is a rectangular area, and the rectangular area is defined by the following data: center X coordinate, center Y coordinate, bounding box width, and bounding box height

[0079] In some embodiments, the preliminary inspection infrared image includes r infrared imaging photos taken in the first dynamic photo and m infrared imaging photos taken in the second dynamic photo. Among them, using the YOLOv8 algorithm to preliminarily classify the defects of the transparent component includes:

[0080] Inputting the r infrared imaging photos taken in the first dynamic photo into the YOLOv8 algorithm in sequence to obtain the class ID, center X coordinate, center Y coordinate, bounding box width, and bounding box height of the suspected defects;

[0081] Statistically analyze the category IDs of suspected defects in r infrared images respectively. If there are no suspected defects, it means that the r infrared images taken in the first dynamic photographing are defect-free. If some or all of them have suspected defects, cluster and merge the central X coordinates and central Y coordinates of the suspected defect category IDs in the r infrared images to unify the information of the same suspected defect.

[0082] Output the category ID, central X coordinate, central Y coordinate, bounding box width, and bounding box height of the suspected defects after clustering and merging.

[0083] Process the m infrared images taken in the second dynamic photographing according to the steps of S201 - S203 to obtain the category ID, central X coordinate, central Y coordinate, bounding box width, and bounding box height of the suspected defects in the second dynamic photographing.

[0084] Merge the category ID, central X coordinate, central Y coordinate, bounding box width, and bounding box height of the suspected defects in the first dynamic photographing and the second dynamic photographing to obtain a preliminary classification.

[0085] In the fine inspection infrared image acquisition unit 3, use the fine inspection station in the infrared imaging detection device to obtain the fine inspection infrared image of the transparent component. Among them, the fine inspection station has an infrared camera, which can take pictures of the positions of the suspected defects obtained during the preliminary inspection on the transparent component to obtain the fine inspection infrared image. Preferably, the fine inspection infrared image is multiple pictures obtained by the infrared camera through multiple dynamic photographings.

[0086] Preferably, the clustering and merging further includes the following steps: perform arithmetic mean operations on the central X coordinates and central Y coordinates of multiple adjacent 3 - 5 pixels, and take the maximum value of the bounding box width and bounding box height among them.

[0087] In the defect type acquisition unit 4, for the defects caused by static factors, use wavelet transform and high-frequency filtering to obtain high and low frequency components. At the same time, use the histogram of oriented gradients to perform statistical analysis on the fine inspection infrared image to obtain the information with obvious gradient changes in the fine inspection infrared image. Among them, the information with obvious gradient changes reflects the shape and details of the defects, and static features are obtained.

[0088] Among them, the obtained high and low frequency components are as follows:

[0089]

[0090] φ(t) is the low-frequency component, is the high-frequency component, k is the translation amount, h k belongs to the coefficients of the low-pass filter, g k belongs to the coefficients of the high-pass filter, and φ(2t - k) is the translated low-frequency component.

[0091] Regarding the features of the refined inspection infrared images caused by dynamic factors, convolutional neural networks are used to extract features from the refined inspection infrared images, and the LSTM model is used to analyze the temporal correlation between adjacent key frames to obtain dynamic features. Then, alignment of multi-modal features and fusion of multi-modal features are carried out to ultimately achieve defect detection and recognition of transparent components.

[0092] Embodiment 3

[0093] An embodiment of the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the multi-modal feature fusion detection method for transparent component infrared imaging in Embodiment 1 is implemented.

[0094] Embodiment 3 of the present disclosure is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.

[0095] The electronic device may be presented in the form of a general-purpose computing device. For example, it may be a server device. The components of the electronic device may include, but are not limited to: at least one processor, at least one memory, and a bus connecting different system components (including the memory and the processor).

[0096] The bus includes a data bus, an address bus, and a control bus.

[0097] The memory may include volatile memory, such as random access memory (RAM) and / or cache memory, and may further include read-only memory (ROM).

[0098] The memory may further include program tools having a set (at least one) of program modules. Such program modules include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. The implementation of a network environment may be included in each or some combination of these examples.

[0099] The processor executes various functional applications and data processing by running the computer program stored in the memory.

[0100] The electronic device can also communicate with one or more external devices (such as a keyboard, a pointing device, etc.). Such communication can be carried out through an input / output (I / O) interface. Moreover, the electronic device can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter. The network adapter communicates with other modules of the electronic device through a bus. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in combination with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (redundant array of independent disks) systems, tape drives, and data backup storage systems, etc.

[0101] It should be noted that, although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more of the above-described units / modules can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0102] Embodiment 4

[0103] An embodiment of the present disclosure provides a computer-readable storage medium, where the readable storage medium stores a computer program, and when the program is executed by a processor, the steps of the transparent component infrared imaging multimodal feature fusion detection method in Embodiment 1 are implemented.

[0104] Among them, the more specific forms that the readable storage medium can adopt can include but are not limited to: portable disks, hard disks, random access memories, read-only memories, erasable programmable read-only memories, optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0105] In a possible implementation manner, the present disclosure can also be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to cause the terminal device to execute the steps of the transparent component infrared imaging multimodal feature fusion detection method described in Embodiment 1.

[0106] Among them, the program code for executing the present disclosure can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, executed as an independent software package, partially on the user device and partially on a remote device, or executed entirely on a remote device.

[0107] Although embodiments of the present disclosure have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present disclosure, and the scope of the present disclosure is defined by the appended claims and their equivalents.

Claims

1. A transparent component infrared imaging multi-modal feature fusion detection method, characterized in that: The steps include: S10, obtaining a preliminary inspection infrared image of the transparent component; S20, based on the initial inspection infrared image, using the YOLOv8 algorithm to preliminarily classify the defects of the transparent component and obtain the area where the suspected defects are located; the area is a rectangular area, and the rectangular area is defined by the following data: center X coordinate, center Y coordinate, bounding box width and bounding box height; the initial inspection infrared image includes r infrared imaging photos of the first dynamic photography and m infrared imaging photos of the second dynamic photography, wherein the YOLOv8 algorithm is used to preliminarily classify the defects of the transparent component, including: S201, inputting the r infrared imaging photos taken in the first dynamic shooting into the YOLOv8 algorithm in sequence to obtain the category ID, center X coordinate, center Y coordinate, bounding box width and bounding box height of the suspected defect; S202, respectively counting the category IDs of the suspected defects of the r infrared images, if there are no suspected defects, the r infrared images of the current first dynamic shooting are detected to be defect-free; if some or all of them have suspected defects, clustering and merging the center X coordinates and center Y coordinates of the suspected defect category IDs in the r infrared images to achieve information unification of the same suspected defect; S203, outputting the category ID, center X coordinate, center Y coordinate, bounding box width and bounding box height of the suspected defect after cluster merging; S204, processing the m infrared imaging photos of the second dynamic photography according to the steps of S201-S203 to obtain the category ID, center X coordinate, center Y coordinate, bounding box width and bounding box height of the suspected defect of the second dynamic photography; S205, combining the category ID, center X coordinate, center Y coordinate, bounding box width, and bounding box height of the suspected defects in the first dynamic photo and the second dynamic photo to obtain a preliminary classification; S30, obtaining a precise inspection infrared image of the area where the suspected defect of the transparent component is located; S40, based on the precise inspection infrared image, uses a multi-modal feature fusion algorithm to identify the final defect type, including: For defects caused by static factors, wavelet transform is used to obtain high and low frequency components, and directional gradient histogram is used to perform statistical analysis on the precision inspection infrared image to obtain information with obvious gradient changes in the precision inspection infrared image. The information with obvious gradient changes reflects the shape and details of the defect, and obtains static features. Aiming at the characteristics of precise inspection infrared images caused by dynamic factors, a convolutional neural network is used to extract features from precise inspection infrared images, and the LSTM model is used to analyze the temporal correlation between adjacent key frames to obtain dynamic evolution characteristics.

2. The infrared imaging multi-modal feature fusion detection method for transparent components according to claim 1, characterized in that: The initial inspection infrared image and / or the fine inspection infrared image are multiple photos taken dynamically multiple times.

3. The infrared imaging multi-modal feature fusion detection method for transparent components according to claim 1 or 2, characterized in that: The preliminary classification includes the following: scratches, cracks, bubbles, inclusions, dirt and no defects.

4. The infrared imaging multi-modal feature fusion detection method for transparent components according to claim 1, characterized in that: The cluster merging further comprises the following steps: performing an arithmetic average operation on the center X coordinates and the center Y coordinates of a plurality of neighboring 3 to 5 pixels, and taking the maximum value of the bounding box width and the bounding box height.

5. A transparent component infrared imaging multi-modal feature fusion detection system, characterized in that: include: A preliminary inspection infrared image acquisition unit, which acquires a preliminary inspection infrared image of the transparent component; A preliminary classification unit, based on the initial inspection infrared image, uses the YOLOv8 algorithm to preliminarily classify the defects of the transparent component and obtain an area where the suspected defects are located; the area is a rectangular area, and the rectangular area is defined by the following data: a center X coordinate, a center Y coordinate, a bounding box width, and a bounding box height; the initial inspection infrared image includes r infrared imaging photos of the first dynamic photography and m infrared imaging photos of the second dynamic photography, wherein the YOLOv8 algorithm is used to preliminarily classify the defects of the transparent component, including: Inputting the r infrared imaging photos taken in the first dynamic shooting into the YOLOv8 algorithm in sequence to obtain the category ID, center X coordinate, center Y coordinate, bounding box width and bounding box height of the suspected defect; The category IDs of the suspected defects of r infrared images are counted respectively. If there are no suspected defects, the r infrared images taken for the first dynamic shooting are detected to be defect-free. If some or all of them have suspected defects, the center X coordinates and center Y coordinates of the suspected defect category IDs in the r infrared images are clustered and merged to achieve the information unification of the same suspected defect. Output the category ID, center X coordinate, center Y coordinate, bounding box width and bounding box height of the suspected defect after cluster merging; Processing the m infrared imaging photos of the second dynamic photography according to steps S201-S203 to obtain the category ID, center X coordinate, center Y coordinate, bounding box width and bounding box height of the suspected defect of the second dynamic photography; Combine the category ID, center X coordinate, center Y coordinate, bounding box width and bounding box height of the suspected defects in the first dynamic photo and the second dynamic photo to obtain a preliminary classification A precision inspection infrared image acquisition unit is used to acquire a precision inspection infrared image of the area where the suspected defect of the transparent component is located; The defect type acquisition unit uses a multimodal feature fusion algorithm to identify the final defect type based on the precise inspection infrared image. For defects caused by static factors, wavelet transform is used to obtain high and low frequency components. At the same time, the directional gradient histogram is used to perform statistical analysis on the precise inspection infrared image to obtain information with obvious gradient changes in the precise inspection infrared image. The information with obvious gradient changes reflects the shape and details of the defect, and obtains static features. Aiming at the characteristics of precise inspection infrared images caused by dynamic factors, a convolutional neural network is used to extract features from precise inspection infrared images, and the LSTM model is used to analyze the temporal correlation between adjacent key frames to obtain dynamic evolution characteristics.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the infrared imaging multi-modal feature fusion detection method for transparent components described in any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the transparent component infrared imaging multi-modal feature fusion detection method described in any one of claims 1 to 4 is implemented.

Images