A semiconductor foreign matter detection method

Abstract

Embodiments of the present application provide a semiconductor foreign matter detection method, which comprises: acquiring a to-be-detected image; inputting the to-be-detected image into a pre-trained foreign matter detection model for analysis and processing to output a foreign matter detection result; wherein the foreign matter detection model at least comprises a relative gradient position encoding module, a plurality of feature interweaving attention modules, and an encoding-decoding module, the relative gradient position encoding module is used for structured gradient feature encoding processing of the to-be-detected image, the feature interweaving attention module is used for first feature cross attention processing of the output of the relative gradient position encoding module, and the encoding-decoding module is used for second feature multi-level dynamic encoding-decoding processing of the output of the feature interweaving attention module; and according to the foreign matter detection result, the foreign matter position information in the to-be-detected image is determined. The technical scheme effectively improves the detection precision and efficiency of tin spot welding point abnormalities and small foreign matter defects in semiconductor X-ray images.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for detecting semiconductor foreign objects. Background Technology

[0002] In the semiconductor manufacturing industry, quality inspection of semiconductor products is a critical link in the chip production process, and the accuracy of inspection determines the reliability and yield of the chips. Among these, detecting the presence of structural anomalies and / or foreign objects inside semiconductor products is crucial for improving product reliability.

[0003] Currently, existing methods employ high-resolution X-ray imaging equipment for non-destructive inspection of the interior of semiconductor product packages. However, in complex integrated environments, tiny foreign objects not only exhibit diverse shapes but also display extremely low contrast with the background in X-ray images, making it difficult to accurately identify and locate them, thus resulting in low detection efficiency. Summary of the Invention

[0004] This invention provides a semiconductor foreign object detection method to improve the detection accuracy and efficiency of foreign objects in X-ray images.

[0005] In a first aspect, embodiments of the present invention provide a semiconductor foreign object detection method, comprising: Acquire an image to be detected; wherein the image to be detected is an X-ray image including the semiconductor to be detected; The image to be detected is input into a pre-trained foreign object detection model for analysis and processing to output foreign object detection results. The foreign object detection model includes at least a relative gradient position encoding module, multiple feature interleaving attention modules, and an encoding / decoding module. The relative gradient position encoding module is used to encode the structured gradient features of the image to be detected. The feature interleaving attention module is used to perform cross-attention processing on the first feature output by the relative gradient position encoding module. The encoding / decoding module is used to perform multi-level dynamic encoding and decoding processing on the second feature output by the feature interleaving attention module. Based on the foreign object detection results, the location information of the foreign object in the image to be detected is determined.

[0006] Secondly, embodiments of the present invention provide a semiconductor foreign object detection device, comprising: An image acquisition module is used to acquire an image to be detected; wherein the image to be detected is an X-ray image including the semiconductor to be detected; The detection result output module is used to input the image to be detected into a pre-trained foreign object detection model for analysis and processing, so as to output the foreign object detection result; wherein, the foreign object detection model includes at least a relative gradient position encoding module, multiple feature interleaving attention modules, and an encoding and decoding module. The relative gradient position encoding module is used to encode the structured gradient features of the image to be detected. The feature interleaving attention module is used to perform cross-attention processing on the first feature output by the relative gradient position encoding module. The encoding and decoding module is used to perform multi-level dynamic encoding and decoding processing on the second feature output by the feature interleaving attention module. The foreign object location information determination module is used to determine the location information of the foreign object in the image to be detected based on the foreign object detection result.

[0007] Thirdly, embodiments of the present invention also provide an electronic device, comprising: At least one processor; and A memory that is communicatively connected to at least one processor; wherein, The memory stores a computer program that can be executed by at least one processor, such that the at least one processor can perform a semiconductor foreign object detection method as provided in any embodiment of the present invention.

[0008] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions for causing a processor to execute a semiconductor foreign object detection method as provided in any embodiment of the present invention.

[0009] Fifthly, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements a semiconductor foreign object detection method as described in any of the embodiments of the present disclosure.

[0010] The technical solution provided by this invention acquires an image of the semiconductor to be detected, and inputs the image into a pre-trained foreign object detection model for analysis and processing. The image undergoes at least a relative gradient position encoding module, multiple feature interleaving attention modules, and an encoding / decoding module. By enhancing the corresponding features of the image, the foreign object detection result is output, thereby improving the accuracy of foreign object detection. Based on the determined foreign object detection result, the location information of the foreign object in the image is determined. In summary, the technical solution of this invention effectively improves the accuracy and efficiency of detecting solder joint anomalies and minute foreign object defects in semiconductor X-ray images.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a semiconductor foreign object detection method provided in an embodiment of the present invention; Figure 2 A flowchart illustrating the model structure of a foreign object detection model provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a relative gradient position encoding module provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a feature interleaving attention module provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of an encoding / decoding module structure provided in an embodiment of the present invention; Figure 6 A schematic diagram of an encoding unit structure provided in an embodiment of the present invention; Figure 7 A schematic diagram of a decoding unit structure provided in an embodiment of the present invention; Figure 8 This is a flowchart of a semiconductor foreign object detection method provided in an embodiment of the present invention; Figure 9 This is an overall framework diagram of a semiconductor foreign object detection method provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of the structure of a semiconductor foreign object detection device provided in an embodiment of the present invention; Figure 11 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Before introducing the technical solutions provided by the embodiments of the present invention, the application scenarios can be illustrated by example. The embodiments of the present invention are applicable to application scenarios in semiconductor manufacturing and packaging processes, where X-ray imaging of a target object is used to detect the presence of foreign objects inside the target object. Optionally, it can be applied to the quality inspection stage of semiconductor devices such as integrated circuit chips, power modules, and sensors before they leave the factory. For example, after chip packaging is completed, two-dimensional or three-dimensional images of the internal structure of the chip need to be acquired using X-ray imaging equipment, and the images are inspected for process defects and / or contaminants. Process defects and / or contaminants can be solder ball voids, foreign dust, or metal debris, etc.

[0017] Figure 1 This is a flowchart illustrating a semiconductor foreign object detection method according to an embodiment of the present invention. This embodiment is applicable to the detection of foreign objects in X-ray images of semiconductors. The method can be executed by a semiconductor foreign object detection device, which can be implemented in hardware and / or software, and this device can be configured in a computing device. Figure 1 As shown, the method includes: S110. Obtain the image to be detected; wherein the image to be detected is an X-ray image including the semiconductor to be detected.

[0018] In this embodiment, the image to be inspected can be a raw image containing the semiconductor to be inspected. This image can be obtained directly by transmitting a high-resolution X-ray image of the semiconductor to be inspected using a high-resolution X-ray imaging device. The semiconductor to be inspected is a semiconductor device that requires internal structure and welding quality inspection during the manufacturing process.

[0019] It should be noted that after acquiring the image to be detected, image preprocessing can be performed to obtain a standardized image. Image preprocessing involves processing the image before inputting it into the model for analysis to optimize image quality and facilitate subsequent model analysis and recognition. Image preprocessing includes, but is not limited to, denoising, image cropping, contrast enhancement, and scaling. Denoising is the process of reducing unwanted noise in an image, such as eliminating thermal noise from electronic devices in X-ray images. Denoising methods include, but are not limited to, Gaussian filtering, median filtering, and nonlocal mean denoising. Image cropping involves extracting at least one rectangular region of interest from the image. Considering that semiconductor images may contain background elements unrelated to semiconductor devices, such as carriers and / or borders, image cropping can be used to extract the main detection area to eliminate interference from irrelevant elements. Contrast enhancement methods amplify the grayscale differences of the features of interest in the image. Contrast enhancement methods include, but are not limited to, histogram equalization, grayscale transformation, and adaptive histogram equalization. Scaling methods can change the size of an image, such as changing its width and height, so that the image can be input into the model.

[0020] For example, the main detection region is cropped from the image to be detected, and then the image is enhanced through image enhancement operations such as denoising and / or grayscale contrast enhancement. Further, the image to be detected is scaled to a uniform size for the model input data, such as a 1024×1024 resolution.

[0021] Specifically, an X-ray image of the semiconductor to be detected is obtained as the image to be detected, which facilitates the subsequent detection and judgment of the semiconductor to be detected in the image to be detected by the model.

[0022] S120. Input the image to be detected into the pre-trained foreign object detection model for analysis and processing, and output the foreign object detection result.

[0023] The foreign object detection model includes at least a relative gradient position encoding module, multiple feature interleaving attention modules, and an encoding / decoding module. The relative gradient position encoding module is used for structured gradient feature encoding of the image to be detected. The feature interleaving attention module is used for cross-attention processing of the first feature output by the relative gradient position encoding module. The encoding / decoding module is used for multi-level dynamic encoding and decoding processing of the second feature output by the feature interleaving attention module.

[0024] In this embodiment, the foreign object detection model is a model used to analyze the input image to be detected, identify whether there is a foreign object in the image, and accurately locate it. The anomaly detection model can be a trained artificial intelligence model, such as a convolutional neural network.

[0025] It should be noted that the foreign object detection model includes at least a relative gradient position encoding module, a multi-feature interleaving attention module, and an encoding / decoding module.

[0026] The relative gradient position encoding module captures the spatial distribution relationship between solder joints and foreign objects by introducing a relative position encoding mechanism between elements. This module receives the image to be detected and performs structured gradient feature encoding processing on it. Structured gradient feature encoding can be understood as the process of extracting gradient information from the input image to be detected and assembling the gradient information into feature representations, providing the model with deeper structured geometric features. The first feature can be the feature representation containing spatial relationships obtained after the image to be detected has been processed by the relative gradient position encoding module.

[0027] The feature interleaving attention module is a substructure within a deep learning model used to interact and fuse feature information from different semantics within the model. This allows the foreign object detection model to focus on more critical regions while suppressing irrelevant areas. Specifically, the feature interleaving attention module performs cross-attention processing on the first feature output from the relative gradient position encoding module.

[0028] The second feature can be the optimized feature representation obtained after being processed by at least one feature interleaving attention module, which is the image feature to be input into the encoding / decoding module, the next workflow module after the feature interleaving attention module. Cross-attention processing can be understood as interacting between two features to enhance the processing of semantic information.

[0029] It should be noted that the number of feature interleaving attention modules can be one or more. When there are multiple feature interleaving attention modules, the features output by the relative gradient position encoding module are processed by multiple feature interleaving attention modules, resulting in multiple first features. These multiple first features can be fused through feature concatenation operations to form a fused first feature.

[0030] For example, when the number of feature interleaving attention modules is 3, the features output by the relative gradient position encoding module are respectively input into three feature interleaving attention modules with identical structures, and three corresponding first features are output. Further, the three first features are concatenated to obtain the concatenated first feature.

[0031] The encoding / decoding module can contain both an encoding module and a decoding module. The encoding module contains at least multiple encoding units, and the decoding module contains at least multiple decoding units. The encoding module progressively extracts deep semantic features, while the decoding module progressively restores the spatial dimensions of the feature map and recovers the deep semantic information extracted by the encoding module back to a high-resolution space. The encoding / decoding module is used for multi-level dynamic encoding and decoding processing of the second feature output by the feature interleaving attention module. Multi-level dynamic encoding and decoding processing can be understood as first encoding the second feature for multi-scale feature extraction and information compression, and then decoding for multi-scale feature fusion and information reconstruction.

[0032] Furthermore, the anomaly detection model can be trained using multiple sample images and their corresponding sample label data. The multiple sample images are the image dataset used to train the anomaly detection model. These images are acquired using high-resolution X-ray imaging equipment and / or obtained from publicly available semiconductor foreign object detection datasets, containing images of semiconductors. Standardized images obtained after preprocessing these images can be used as sample images. It can be noted that the multiple sample images include, but are not limited to, defective images containing foreign objects and / or solder joint anomalies, as well as completely normal, good product images.

[0033] The sample label data consists of standard results corresponding to multiple sample images. The sample label data includes at least a label marking the label region containing the foreign object, the size information of the label region, and the coordinates of at least one vertex of the label region. The foreign object detection result can be data output by an anomaly detection model, characterizing anomalies in the image to be detected.

[0034] Specifically, the image to be detected is input into a pre-trained foreign object detection model for analysis and identification, and finally outputs a semiconductor anomaly detection result that characterizes the abnormality of the image to be detected.

[0035] The foreign object detection results output by the anomaly detection model contain multiple pieces of information. Furthermore, the specific content of the anomaly detection results is described in detail. Optionally, the foreign object detection results may include at least the region marker containing the location of the foreign object, the region size, and the region confidence level.

[0036] In this embodiment, the region marker containing the foreign object can be an image range identified and bounded in the image after analysis by the foreign object detection model. This image range is associated with the semiconductor foreign object. The visualization of the region marker can be a geometric shape. For example, the detected region containing the foreign object is marked with a rectangle.

[0037] The region size can include the dimensions of the region containing the foreign object and its vertex coordinates. The dimensions can be the width and height of the region. The vertex coordinates define the shape and position of the region containing the foreign object. For example, if the region marker is rectangular, the vertex positions are the coordinates of its four corner points; if the region marker is polygonal, the vertex coordinates are the coordinates of all key points that sequentially connect to form the region containing the foreign object. The coordinate information can be the x-coordinate and y-coordinate of the vertex in a coordinate system with the top-left corner of the image as the origin.

[0038] The region confidence score is the probability value given by the foreign object detection model that the target region is where the foreign object is located. The confidence score can be a value between 0 and 1, with a confidence score closer to 1 indicating a higher probability of the presence of a foreign object in the region.

[0039] Specifically, the foreign object detection results output by the foreign object detection model include at least the region marker, region size, and region confidence level of the area where the foreign object is located. Based on the foreign object detection results output by the model, it is determined whether the semiconductor to be detected has any abnormalities, and the specific location information of the abnormal region. This enables detection visualization and improves detection efficiency.

[0040] S130. Based on the foreign object detection results, determine the location information of the foreign object in the image to be detected.

[0041] In this embodiment, the foreign object location information refers to the specific location of the foreign object in the image to be detected, as determined by the foreign object detection model. This location information may include the coordinates of the area where the foreign object is located.

[0042] Specifically, the foreign object detection results are obtained, and the specific location information of the foreign object in the image to be detected is determined based on the region size and region confidence in the foreign object detection results.

[0043] The technical solution provided by this invention acquires an image to be detected and inputs it into a pre-trained foreign object detection model for analysis and processing. Based on the processing of modules such as the relative gradient position encoding module, feature interleaving attention module, and encoding / decoding module in the foreign object detection model, a foreign object detection result is output. Finally, based on the foreign object detection result, the location information of the foreign object in the image to be detected is determined to generate more accurate foreign object location information. In summary, the technical solution of this invention effectively improves the accuracy and efficiency of detecting solder joint anomalies and minute foreign object defects in semiconductor X-ray images.

[0044] Based on the foregoing embodiments, this embodiment further refines the model structure of the foreign object detection model. For specific implementation details, please refer to the description of this embodiment. Technical features that are the same as or similar to those in the foregoing embodiments will not be repeated here. Optionally, see... Figure 2 The foreign object detection model, based on the input-output relationship, includes the relative gradient position encoding module 21, multiple feature interleaving attention modules 22, the encoding / decoding module 23, at least one detection head module 24, and an output module 25.

[0045] First, for the relative gradient position encoding module, the relative gradient position encoding module processes the image to be detected in the following manner: For at least one pixel to be processed in the image to be detected, the second-order gradient values ​​of the pixel to be processed in multiple directions are determined based on the first pixel value of the pixel to be processed and the second pixel value of the associated pixels in the preset neighborhood. The multiple directions include a first direction, a second direction, a third direction and a fourth direction. The third direction is the direction that forms a first preset angle with the first direction, and the fourth direction is the direction that forms a second preset angle with the first direction. The first preset angle and the second preset angle are complementary.

[0046] In this embodiment, at least one pixel in the image to be detected can be the smallest constituent unit of the image or a basic processing unit for feature calculation. It should be noted that all pixels contained in the image to be detected can be referred to as pixels to be processed.

[0047] To determine the second-order gradient values ​​of the pixel to be processed in multiple directions, it is necessary to obtain the first pixel value of the pixel to be processed and the second pixel values ​​of the associated pixels within a preset neighborhood. The first pixel value can be understood as the pixel value corresponding to the pixel to be processed. The second pixel value can be understood as the pixel value of the pixel associated with the pixel to be processed within the preset neighborhood. The preset neighborhood is a rectangular area that slides across the pixel. This rectangular area has a fixed size, and the size value is preset. For example, the preset neighborhood can be a 3×3 rectangular sliding window. It should be noted that each pixel in the image to be detected can be used as a pixel to be processed, and the pixel to be processed is placed at the center of the preset neighborhood.

[0048] The first pixel value of the pixel to be processed and the second pixel value of the pixels adjacent to the pixel to be processed and located in a preset neighborhood are obtained. The second-order gradient values ​​of the pixel to be processed are calculated in multiple directions. The second-order gradient value can characterize the direction and rate of change of image grayscale values. By calculating the second-order gradient value of each pixel, the endpoints of edges, isolated spots, and texture undulations in the image to be detected are captured. The multiple directions include at least a first direction, a second direction, a third direction, and a fourth direction. Optionally, if the upper left corner of the image to be detected is taken as the origin, the positive x-axis is horizontally to the right, and the positive y-axis is vertically downward. The first direction can be the x-axis direction in the image coordinate system, used to capture the grayscale change pattern and curvature characteristics of the image to be detected in the horizontal direction; the second direction can be the y-axis direction in the image coordinate system, used to capture the structural strength and change pattern of the image in the vertical direction; the third direction is the direction at a first preset angle to the first direction, and the fourth direction is the direction at a second preset angle to the first direction, and the first and second preset angles are complementary. The first preset angle is the angle between the first direction and the third preset angle, and the second preset angle is the angle between the first direction and the fourth direction.

[0049] For example, if the first preset angle is 45° and the second preset angle is 35°, then the third direction is along the diagonal direction from the upper left to the lower right of the image, used to extract the geometric structure information of the image to be detected along the main diagonal direction. The fourth direction is along the diagonal direction from the upper right to the lower left of the image to be detected, used to extract the local texture changes of the image along the secondary diagonal direction.

[0050] Specifically, based on at least one pixel to be processed in the image to be detected, pixels associated with the pixel to be processed are determined based on a preset neighborhood, thereby determining a first pixel value and a second pixel value. Based on the first pixel value and the second pixel value, the second-order gradient values ​​of the pixel to be processed in multiple directions are determined, i.e.: in, The curvature that characterizes the grayscale of an image in the first direction, i.e., for Find the second-order partial derivative along the first direction. The curvature that characterizes the image grayscale in the second direction, i.e., for Find the second-order partial derivative along the second direction. Characterizes the interaction of image grayscale changes in the third and fourth directions. Indicates the image in coordinates The grayscale intensity value at that location. To represent partial derivatives, This represents a second-order partial differential.

[0051] The local structure matrix of at least one pixel to be processed is determined based on the second-order gradient values ​​of at least one pixel in each direction.

[0052] In this embodiment, the local structure matrix represents the geometric properties of grayscale changes within a preset neighborhood, the maximum eigenvalue of the moment matrix represents the salience of the local edge, and the corresponding eigenvector represents the dominant direction of gradient changes.

[0053] Specifically, the second-order gradient values ​​of at least one pixel to be processed are obtained in each direction. Based on the second-order gradient values ​​in each direction, a local structure matrix of at least one pixel to be processed is constructed, i.e.: ; in, The local structure matrix represents the pixel points. The second-order grayscale variation information within a preset neighborhood. Furthermore, by analyzing the eigenvalues ​​and eigenvectors of the local structure matrix, the local structure type and main edge direction of the pixel can be inferred.

[0054] Furthermore, to eliminate the inconsistency in feature dimensions caused by variations in illumination and differences in image acquisition conditions, and to enhance the contrast and comparability of gradient features between different pixels, the local structure matrix is ​​further normalized: ; in, It can be the normalized local structure matrix. Located at pixel coordinates The local structure matrix at that location. It can be the sum of the local structure matrix energies of all pixels in the image. It can be the matrix norm of the local structure matrix. This represents each pixel in the image. Normalizing the local structure matrix corresponding to each pixel in the image to be processed can be understood as standardizing the local structure matrix of each pixel obtained in the previous process to adjust the numerical range of the gradient features.

[0055] A first image feature is obtained by processing at least one pixel, wherein the first image feature is the image feature output by the relative gradient position encoding module.

[0056] In this embodiment, the first image feature is the image feature output by the relative gradient position encoding module. This image feature may contain multi-dimensional information extracted from the local structure matrix. For example, the multi-dimensional information may be information such as the gradient principal direction and local edge intensity.

[0057] For details, see Figure 3After the image to be detected 31 is input into the relative gradient position encoding module 32, each pixel in the image to be processed is first taken as the center pixel of the preset neighborhood, and the second-order gradient values ​​in multiple directions are calculated to extract the gray-level curvature information in the preset neighborhood and capture the subtle structural changes in the image to be detected. Further, a local structure matrix is ​​constructed based on the second-order gradients in multiple directions of each pixel to be processed. The eigenvalues ​​and eigenvectors of the local structure matrix correspond to the edge intensity and dominant direction, respectively. Further, the local structure matrix corresponding to each pixel to be processed is normalized to eliminate dimensional differences between images, improve feature contrast and stability, and enhance the model's sensitivity to low-contrast small foreign objects.

[0058] For example, after the second-order gradient value of pixel 33 in the relative gradient position encoding module is calculated by G(x,y), the second-order gradient value of the pixel 34 in the image to be detected is determined, and finally the first image feature 35 is output by the relative gradient position encoding module.

[0059] Secondly, for the feature interleaving attention module, multiple feature interleaving attention modules are arranged in parallel, and each feature interleaving attention module processes the first image features output by the relative gradient position encoding module in the following manner.

[0060] The first image features are enhanced by performing feature enhancement processing on at least two first depthwise convolutional units respectively, so as to obtain the first enhanced feature output by each first depthwise convolutional unit.

[0061] In this embodiment, after obtaining the first image features output by the relative gradient position encoding module, the first image features are simultaneously input into multiple feature interleaving attention modules. It should be noted that the multiple feature interleaving attention modules are arranged in parallel, and each feature interleaving attention module has the same structure. Parallel arrangement can be understood as multiple feature interleaving attention modules with identical structures being arranged side-by-side at the same network layer, processing the first image features separately, and finally concatenating and fusing the output structures of each feature interleaving attention module. Optionally, the number of feature interleaving attention modules can be adaptively adjusted according to the actual scenario requirements; this embodiment does not impose any limitations on this.

[0062] Next, the process of feature enhancement processing of the first image features by interleaved attention for each feature will be described in detail.

[0063] In this embodiment, the depthwise convolutional unit is a unit structure that includes depthwise convolution. The feature interleaving attention module includes at least a first depthwise convolutional unit and a second depthwise convolutional unit. The first depthwise convolutional unit includes at least a depthwise convolutional layer, a batch normalization layer, and an activation function. The second depthwise convolutional unit includes at least a depthwise convolutional layer and a batch normalization layer.

[0064] Depthwise convolution can be considered a lightweight convolution operation. It decomposes standard convolution into two parts: first, depthwise convolution, then pointwise convolution. Depthwise convolution can be understood as performing an independent convolution operation on each channel of the image features with a separate kernel. Pointwise convolution can be understood as using a 1×1 kernel to mix information from all channels, thus extracting image features while reducing the number of parameters and computational cost.

[0065] Batch normalization layers can be a standardization process used to accelerate deep network training, which normalizes each channel to a mean of 0 and a variance of 1 on each mini-batch of data.

[0066] Activation functions can be mathematical functions that introduce nonlinear transformations, enabling neural networks to fit nonlinear mappings. For example, the GeLU activation function can be used. The GeLU activation function multiplies the input by the probability of activation, and the probability of activation is determined by the cumulative probability of the input value itself in a standard normal distribution. Activation functions can perform nonlinear transformations on features normalized by batch normalization layers to enhance the model's expressive power.

[0067] The first enhanced feature can be a local image feature obtained after the first image feature has undergone feature enhancement processing by at least two first depthwise convolutional units. It should be noted that the at least two first depthwise convolutional units are ordered in parallel, that is, when the first image feature is simultaneously input into at least two first depthwise convolutional units, the output feature obtained is the first enhanced feature.

[0068] Specifically, after obtaining the first image features, the first image features are input into at least two first depthwise convolutional units to perform feature enhancement processing on the first image features, and the first enhanced features output by each first depthwise convolutional unit are obtained.

[0069] Based on the second depthwise convolutional unit, feature enhancement processing is performed on the first image features to obtain the second enhanced features.

[0070] In this embodiment, the second depthwise convolutional unit is a unit that includes at least a depthwise convolutional layer. The second enhanced feature can be the feature output after the first image feature has been enhanced by the second depthwise convolutional unit. Optionally, the second depthwise convolutional unit may include a depthwise convolutional layer and a batch normalization layer.

[0071] Specifically, after acquiring the first image features, the first image features are enhanced by passing them through a depthwise convolutional layer in the second depthwise convolutional unit, and then normalized by a batch normalization layer. Finally, the enhanced features are output as the second enhanced features. This process preserves the original information and enhances the global structural information of the first image features.

[0072] The third enhanced feature is obtained by superimposing the two first enhanced features and inputting them into the next first depthwise convolutional unit. The third enhanced feature is then aggregated based on the global average pooling unit to obtain the fourth enhanced feature.

[0073] In this embodiment, feature overlay can be understood as adding the two first enhanced features element-wise to generate a feature that fuses multiple information streams. The third enhanced feature can be the feature obtained by inputting the overlay of the first enhanced features into the next first depthwise convolutional unit.

[0074] After obtaining the third enhanced feature, it undergoes aggregation processing via a global average pooling unit. The global average pooling unit is used to compress the spatial dimension of the feature map and extract global contextual information. Based on the input-output relationship, the unit structure of the global average pooling unit consists of a global average pooling layer and an activation function. Global average pooling can be understood as averaging all pixel values ​​in each channel of the input image features and outputting a vector. It can be noted that the length of the output vector is consistent with the number of channels in the input image features, thus aggregating spatial location information into a global descriptor. Optionally, the activation function can be the ReLU activation function, which is a rectified linear unit that sets negative values ​​to zero and retains positive values. The fourth enhanced feature is the feature obtained after the third enhanced feature has been aggregated by the global average pooling unit.

[0075] Specifically, two first-level enhanced features are obtained, and after feature stacking, they are input into the first depthwise convolutional unit to output a third-level enhanced feature. Further, the third-level enhanced feature is input into a global average pooling unit for aggregation, resulting in a fourth-level enhanced feature. Global semantic context is extracted to provide pixel-level precise guidance information for subsequent operations.

[0076] The fourth enhanced feature is processed by at least two first depthwise convolutional units respectively. The fifth enhanced feature output by each first depthwise convolutional unit is then processed by superimposing the two fifth enhanced features and input into the next first depthwise convolutional unit to obtain the sixth enhanced feature.

[0077] In this embodiment, the fifth enhancement feature can be the feature output after the fourth enhancement feature is enhanced by the first depthwise convolution feature. It should be noted that at least two first depthwise convolution units are ordered in parallel; therefore, the fourth enhancement feature is input into at least two first depthwise convolution units, outputting at least two corresponding fifth enhancement features. The sixth enhancement feature can be the feature output after the two fifth enhancement features are superimposed and processed by the first depthwise convolution unit.

[0078] Specifically, after obtaining the fourth enhanced feature, it is input into at least two first depthwise convolutional units for feature enhancement processing, outputting at least two corresponding fifth enhanced features. Further, the obtained fifth enhanced features are processed by feature stacking and then input into the first depthwise convolutional units for processing, outputting a sixth enhanced feature. This focuses the features on local interactions along the channel dimension and improves the robustness of the output features.

[0079] The second enhancement feature is processed based on the signal function to obtain the seventh enhancement feature.

[0080] In this embodiment, the signal function is a function used to generate attention weights. Optionally, the signal function can be a sigmoid activation function, which compresses the input values ​​to the range (0,1) to generate a channel attention weight map and determine the importance score of each feature channel. The seventh enhancement feature is the feature obtained after processing the second enhancement feature using the signal function.

[0081] Specifically, after obtaining the second enhanced feature, a signal function is used to process the second enhanced feature to obtain the seventh enhanced feature. The resulting attention weights can suppress background noise and highlight the foreign object region.

[0082] The second image features are determined based on the sixth and seventh enhancement features; The second image feature is the feature output by the feature interleaving attention module; the first depthwise convolutional unit includes at least a depthwise convolutional layer, a batch normalization layer and an activation function; the global average pooling unit includes at least a global average pooling layer and an activation function; and the second depthwise convolutional unit includes at least a depthwise convolutional layer, a batch normalization layer and an activation function.

[0083] In this embodiment, the second image feature is the feature output by the feature interleaving attention module after the first image feature has passed through it.

[0084] Specifically, after obtaining the sixth and seventh enhanced features, the second image features are obtained through feature splicing, and adaptive feature fusion is performed to output more discriminative features.

[0085] For example, see Figure 4First, the first image feature I is simultaneously used as a feature. and characteristics The input is processed by two first depthwise convolutional units 41 and 42, with kernel sizes of 3×3 and 5×5 respectively, capturing features from two different receptive fields. Furthermore, the two first enhanced features at different scales are directly added and fused, allowing the features to simultaneously possess local details and contextual information. The fused features are then input again into a first depthwise convolutional unit 43, with a kernel size of 3×3, to achieve further interaction and integration of multi-scale information, outputting a third enhanced feature. The third enhanced feature is then input into a global average pooling unit 44, outputting a fourth enhanced feature. ,Right now: ; in, This is the fourth enhancement feature. as well as All are first image features, and DBG is the first depthwise convolutional unit. The kernel size is The first depthwise convolutional unit, The kernel size is The first depthwise convolutional unit, This is a global average pooling unit.

[0086] Furthermore, the fourth enhanced feature is simultaneously input into two first depthwise convolutional units 45 and 46 for processing. The kernel sizes of the depthwise convolutions in the two first depthwise convolutional units are 5×5 and 7×7, respectively. After passing through two first depthwise convolutional units of different sizes, two fifth enhanced features are generated and summed to fuse spatial context information from different ranges. The fused features are then processed through a convolutional kernel with a kernel size of... The first depthwise convolutional unit 47 outputs the sixth enhanced feature. ,Right now: ; in, The kernel size is The first depthwise convolutional unit, The kernel size is The first depthwise convolutional unit, This is the third enhancement feature.

[0087] At the same time, the first image features The input is fed into the second depthwise convolutional unit 48 to generate the second enhanced feature 47, and the second enhanced feature is processed using a signal function to obtain the seventh enhanced feature. The seventh enhanced feature represents the importance of each spatial location in the first image feature, i.e. ; in, This is the second depthwise convolutional unit. To and as well as Consistent first image features For splicing operations, This is the seventh enhanced feature.

[0088] Specifically, after acquiring the first image features, the first image features are input into at least two first depthwise convolutional units to perform feature enhancement processing, outputting a first enhanced feature. Further, the first image features are input into a second depthwise convolutional unit for feature enhancement processing to obtain a second enhanced feature. The two enhanced features are then superimposed and input into the next first depthwise convolutional unit to output a third enhanced feature. Further, based on a global average pooling unit, the third enhanced feature is aggregated to output a fourth enhanced feature. Further, based on at least two first depthwise convolutional units, the fourth enhanced feature is enhanced to obtain a fifth enhanced feature output by each first depthwise convolutional unit, and the two fifth enhanced features are superimposed. This processed feature is then input into a first depthwise convolutional unit to obtain a sixth enhanced feature. A signal function is used to process the second enhanced feature, outputting a seventh enhanced feature. Further, the sixth and seventh enhanced features are output to acquire the second image features. This process aggregates the spatial information of the image features into global information while maintaining the independence of each channel, simplifying the network structure and enhancing the model's robustness.

[0089] Furthermore, the encoding / decoding module includes multiple encoding units, which process the second image features based on the following method: The second image features are processed based on the normalization submodule to obtain the first feature to be used. The normalization submodule includes a normalization function and an activation function.

[0090] In this embodiment, the encoding and decoding module includes multiple encoding units. Each encoding unit, after receiving the second image features, outputs features that have undergone scale normalization and nonlinear transformation through a sub-module consisting of a normalization function and an activation function connected in series.

[0091] The normalization submodule performs channel-by-channel scale normalization on the input second image features before applying a non-linear transformation to enhance expressive power. This submodule includes a normalization function and an activation function. The normalization function eliminates the numerical scale differences between different channels and samples in the feature map. The first feature to be used is the feature obtained by normalizing the second image features.

[0092] For details, see Figure 5 After obtaining the second image features, based on the encoding part 51 in the encoding and decoding module, the normalization submodule BG is used to normalize the second image features and output the first feature to be used, providing a stable input feature for subsequent processes.

[0093] Before introducing the subsequent processing of the first feature to be used, the encoding unit in the encoding and decoding module will be described in detail. Optionally, the encoding unit includes multiple dynamic convolutional layers and multiple downsampling layers. The dynamic convolutional layer includes at least a dynamic convolutional group and the activation function, and at least part of the dynamic convolutional layer and the downsampling layer are connected.

[0094] In this embodiment, the dynamic convolutional layer can be a feature transformation layer consisting of a dynamic convolutional group followed by an activation function. The dynamic convolutional layer includes a dynamic convolutional group and an activation function. The dynamic convolutional group is used to dynamically generate convolutional kernel parameters based on the input features and perform convolution operations. The downsampling layer is used to reduce the spatial resolution of the feature map and expand the receptive field.

[0095] For example, see Figure 6 To obtain the input features, we need to consider input_1 and input_2. It should be noted that the input features are the features obtained from the output of the previous encoding unit. For example... Figure 5 The input features of the coding unit EU3 include the features output by the coding unit EU1 and the features output by the coding unit EU2. The features output by the coding unit EU1 can be used as input_1 and the features output by the coding unit EU2 can be used as input_2.

[0096] Inputs _1 and _2 are fed into the first dynamic convolutional layer 61. This dynamic convolutional layer includes dynamic convolutional groups and an activation function; optionally, the activation function can be the GeLU activation function. The features output from the first dynamic convolutional layer are then fed into the first downsampling layer 62 to perform spatial dimensionality reduction, initially expanding the receptive field while preserving key structural information. Furthermore, the dimensionality-reduced features are further processed by a second dynamic convolutional layer 64 for deep semantic encoding.

[0097] Furthermore, the features processed by the first downsampling layer are input into the second downsampling layer 63 for further processing, and the features output from the second downsampling layer are input into the third dynamic convolutional layer 65. Simultaneously, the features output from the second dynamic convolutional layer are input into the third dynamic convolutional layer for further processing, and the output features are input into the fourth dynamic convolutional layer 66 for further processing. The output features are then added element-wise with the features output from the second downsampling layer, and the first summed feature is output.

[0098] Furthermore, the features output from the first dynamic convolutional layer and the features output from the second downsampling layer are element-wise added to produce a second added feature. The first and second added features are then concatenated with the features output from the fourth dynamic convolutional layer to achieve feature fusion and enhance the semantic contextual information of the features.

[0099] Specifically, the input features are fed into the encoding unit, and multiple dynamic convolutional layers are used to extract features through dynamic convolution. During the process, downsampling layers are used to reduce the resolution of the feature map, which can perform differentiated processing for different regions and different samples, thereby improving the ability to distinguish complex backgrounds and foreign objects.

[0100] For multiple coding units, when processing based on the i-th coding unit, the output results of the (i-1)-th coding unit and the (i-2)-th coding unit are obtained to obtain the current output result of the i-th coding unit. Where, when i is a first preset value, the output result of the (i-2)-th coding unit is the first feature to be used output by the normalization submodule. In this embodiment, the encoding unit can be a module constituting the encoding part in the decoding encoding module, used to perform layer-by-layer compression and feature extraction of the input features. For example, if i is 4, the output results of the 3rd encoding unit and the output results of the 2nd encoding unit are input into the 4th encoding unit for processing to determine the current output result of the 4th encoding unit.

[0101] The first preset value is a pre-set parameter used to determine the position index of the feature output by the normalization submodule to be input into the encoding unit. For example, when the first preset value is set to 3, i is 3, and the result output by the first encoding unit is the first feature to be used output by the normalization submodule.

[0102] Specifically, after obtaining the first feature to be used output by the normalization submodule, the first feature to be used is input into the encoding unit, and the result is output after being processed by multiple encoding units.

[0103] When the i-th encoding unit is the first preset encoding unit, the output results of the (i-1)-th encoding unit and the (i-3)-th encoding unit are obtained, and the output results are processed based on the i-th preset encoding unit; In this embodiment, the first preset encoding unit is a pre-set encoding unit. When the currently processed encoding unit is the first preset encoding unit, the output results of the (i-1)th encoding unit and the (i-3)th encoding unit can be obtained and processed. For example, when i is 7, that is, when the currently processed encoding unit is the 7th encoding unit and the 7th encoding unit is the first preset encoding unit, the output results of the 6th encoding unit and the 4th encoding unit are obtained and processed, and the output result of the 7th encoding unit is output.

[0104] Specifically, during the processing of the encoding unit, it is necessary to determine whether the currently processed encoding unit is the first preset encoding unit. If the currently processed encoding unit is the first preset encoding unit, the output results of the (i-1)th encoding unit and the (i-3)th encoding unit are obtained. The output results of the (i-1)th encoding unit and the (i-3)th encoding unit are input into the ith encoding unit for output, and the output result of the ith encoding unit is output.

[0105] When the i-th preset coding unit is the second preset coding unit, the usable result corresponding to the odd-numbered coding units is obtained, and the second feature to be used is obtained based on the usable output result; In this embodiment, the second preset encoding unit is a pre-set encoding unit. When the currently processed encoding unit is the second preset encoding unit, the output result corresponding to the odd-numbered encoding units can be obtained. The usable result is the output result after processing by the encoding unit. The second feature to be used is the output result after processing by the eighth encoding unit.

[0106] For example, when the 8th encoding unit is the second preset encoding unit, the output results of the 3rd encoding unit, the 5th encoding unit, and the 7th encoding unit are obtained.

[0107] Specifically, in the i-th preset coding unit (which is the second preset coding unit), the output results corresponding to the odd-numbered coding units are obtained. The output results of the first coding unit and the third coding unit are concatenated to obtain the first concatenated feature. The first concatenated feature is concatenated with the output result of the fifth coding unit to obtain the usable output result; the usable output result and the output result of the seventh coding unit are input into the output result of the eighth coding unit as the second feature to be used.

[0108] The third image feature is obtained by using the second feature to be used and the usable output result.

[0109] In this embodiment, the third image feature is a feature that has been concatenated with the second feature to be used and the usable output result. This feature is the output feature of the encoding part in the encoding and decoding module, and also the input feature of the decoding part in the encoding and decoding module.

[0110] Optionally, the implementation of the encoding part 51 in the encoding / decoding module is described. The encoding part includes a normalization submodule and eight encoding units. The encoding units are EU1, EU2, EU3, EU4, EU5, EU6, EU7, and EU8, respectively. After obtaining the second image feature, the second image feature is input into the normalization submodule to obtain the first feature to be used. The first feature to be used and the second image feature are then input into encoding unit EU1 for processing, obtaining the output result of EU1. That is, the first feature to be used is used as input_1 in encoding unit EU1, and the second image feature is used as input_2 in encoding unit EU1. Following the same method, the first feature to be used and the output result of EU1 are input into encoding unit EU2 to obtain the output result of EU2; the output results of EU1 and EU2 are input into encoding unit EU3 to obtain the output result of EU3; the output results of EU2 and EU3 are input into encoding unit EU4 to obtain the output result of EU4; the output results of EU4 and EU3 are input into encoding unit EU5 to obtain the output result of EU5; the output results of EU4 and EU5 are input into encoding unit EU6 to obtain the output result of EU6; the output results of EU4 and EU6 are input into encoding unit EU7 to obtain the output result of EU7; the output results of EU1 and EU3 are concatenated to obtain the first concatenated feature, and the first concatenated feature is concatenated with the output result of EU5 to obtain the usable output result. Further, the second concatenated feature and the output result of EU7 are input into encoding unit EU8 to obtain the output result of EU8, which is the second feature to be used. The second feature to be used is concatenated with the usable output result to output the third image feature. Through feature extraction and compression mechanisms, the core information of the input data is gradually extracted, and the convolution kernel parameters are adaptively adjusted through dynamic convolution to capture diverse and complex image features and improve the modeling ability of global and local information.

[0111] Furthermore, for the encoding module, the encoding / decoding module includes a decoding unit, and at least one decoding unit and the splicing layer constitute a decoding submodule. Multiple decoding units in the encoding / decoding module process the third image features in the following manner: The third image features are processed based on the first decoding submodule to output the first decoded features. The first decoding submodule includes a first decoding unit, a second decoding unit, and a first stitching layer. The first stitching layer is used to stitch together the features output by the first decoding unit and the second decoding unit. In this embodiment, the first decoding submodule is an encapsulated module containing multiple decoding units and a concatenation layer, used for resolution restoration and multi-branch feature fusion of input features. The first decoding submodule includes a first decoding unit, a second decoding unit, and a first concatenation layer. The first decoding unit is used for spatial resolution enhancement and semantic information reconstruction of the input features. It should be noted that the first decoding unit and the second decoding unit have the same structure. The first concatenation layer can be used to perform concatenation operations on the input features.

[0112] Specifically, the third image features are input to the first decoding submodule for processing. After being processed by the first decoding unit in the first decoding submodule, the third image features are further input to the second decoding unit for processing and output. The features output by the first decoding unit and the features output by the second decoding unit are input to the first stitching layer for stitching processing, and the first decoded features are output.

[0113] Furthermore, the structure of the decoding unit in the decoding submodule is described in detail. Optionally, the decoding unit includes multiple dynamic convolutional layers and multiple upsampling layers, wherein the dynamic convolutional layer includes at least a dynamic convolutional group and the activation function, and at least a portion of the dynamic convolutional layer and the upsampling layer are connected.

[0114] In this embodiment, the upsampling layer can be used to increase the spatial resolution of the feature map in the network layer. For example, the upsampling of features in the upsampling layer can be performed by methods such as transposed convolution, nearest neighbor interpolation, or pixel rearrangement.

[0115] For example, see Figure 7Taking the first decoding submodule as an example, when the third image features are input to the first decoding submodule, they are processed based on the first dynamic convolutional layer 71. The dynamic convolutional group extracts features from the third image features, processes them using an activation function, and outputs the results. The second dynamic convolutional layer 72 receives the features output by the first dynamic convolutional layer and processes them, outputting the second dynamic convolutional layer output. The second dynamic convolutional layer output is input to the third dynamic convolutional layer for feature extraction, outputting the third dynamic convolutional layer 73 output. The first upsampling layer output and the third dynamic convolutional layer output are input to the second upsampling layer 75 for upsampling, and the second upsampling output is output. The first upsampling output and the second upsampling output are added pixel by pixel to output the summed features. The summed features are input to the fourth dynamic convolutional layer 76 for processing to output the first output features. Further, the third dynamic convolutional layer output and the second upsampling output are added pixel by pixel to output the second output features.

[0116] The second decoding submodule processes the features output by the first decoding feature and the second decoding unit to obtain the second decoding feature; wherein, the second decoding submodule includes a third decoding unit and a second splicing layer, and the second splicing layer is used to splice the features output by the third decoding unit and the second decoding unit; In this embodiment, the first decoding feature can be a feature output by the first decoding submodule. The second decoding feature is a feature output by the second decoding submodule. It should be noted that the second decoding submodule includes a third decoding unit and a second splicing layer.

[0117] Specifically, the first decoding feature output by the first decoding submodule is obtained, and the second decoding submodule processes the first decoding feature and the feature output by the second decoding unit, that is, the feature output by the first decoding feature after it is input into the third decoding unit is concatenated with the feature output by the second decoding unit to output the second decoding feature.

[0118] The third decoding submodule processes the second decoding feature and the feature output by the third encoding unit to obtain the third decoding feature. The third decoding submodule includes a fourth decoding unit and a third splicing layer. The third splicing layer is used to process the features output by the third decoding unit and the fourth decoding unit. In this embodiment, the third decoding submodule includes a fourth decoding unit and a third splicing layer. The third decoding feature is the feature output by the third decoding submodule.

[0119] Specifically, the second decoding feature and the feature output by the third encoding unit are obtained. The second decoding feature is input into the feature output by the third decoding unit and then concatenated with the feature output by the fourth decoding unit.

[0120] The fourth decoding submodule processes the features output by the fourth decoding unit and the third decoding features to obtain the fourth decoding features. The fourth decoding submodule includes a fifth decoding unit, a sixth decoding unit, and a fourth splicing layer. The fourth splicing layer is used to splice the features output by the sixth encoding unit and the features output by the fifth encoding unit. In this embodiment, the fourth decoding feature is the feature output by the fourth decoding submodule.

[0121] Specifically, the third decoding feature and the feature output by the fourth decoding unit are obtained. The third decoding feature is input to the sixth decoding unit for processing and outputting a feature. The feature output by the fourth decoding unit is input to the fifth decoding unit for processing and outputting a feature. The features output by the fifth and sixth decoding units are input to the fourth splicing layer for splicing and processing, and the fourth decoding feature is output.

[0122] The fifth decoding submodule processes the features output by the sixth decoding unit and the fourth decoding features to obtain the fifth decoding features. The fifth decoding submodule includes a seventh decoding unit and a fifth splicing layer. The fifth splicing layer is used to splice the features output by the sixth decoding unit and the features output by the seventh decoding unit. In this embodiment, the fifth decoding feature is the feature output after processing by the fifth decoding submodule.

[0123] Specifically, the fourth decoding feature and the features output by the sixth decoding unit are obtained. The fourth decoding feature is input into the seventh decoding unit for processing and output features. The features output by the sixth decoding unit and the features output by the seventh decoding unit are input into the fifth splicing layer for splicing processing, and the fifth decoding feature is output.

[0124] The sixth decoding feature is obtained by processing the fifth decoding feature and the feature output by the fifth decoding unit based on the sixth decoding submodule. The sixth decoding submodule includes an eighth decoding unit and a sixth splicing layer. The sixth splicing layer is used to splice the feature output by the eighth decoding unit, the feature output by the fifth decoding unit, and the feature output by the seventh decoding unit. In this embodiment, the sixth decoding feature can be the feature output by the sixth decoding submodule.

[0125] Specifically, the fifth decoding feature and the feature output by the fifth decoding unit are obtained. The fifth decoding feature is input to the feature output by the eighth decoding unit. The features output by the eighth decoding unit, the features output by the fifth decoding unit, and the features output by the seventh decoding unit are input together to the sixth splicing layer for splicing processing to output the sixth decoding feature.

[0126] The sixth decoding feature is processed based on the normalization submodule to obtain the fourth image feature.

[0127] In this embodiment, the fourth image feature can be the feature output after normalization processing by the normalization submodule.

[0128] Specifically, the sixth decoded feature is input into the normalization submodule for normalization processing and outputs the fourth image feature.

[0129] It should be noted that the second, third, fourth, fifth, sixth, seventh, and eighth decoding units have the same structure as the first decoding unit. The normalization submodule has the same structure as the normalization submodule in the encoding part of the encoding / decoding module.

[0130] For example, see Figure 5 The decoding part 52 in the encoding and decoding module, after acquiring the third image feature, inputs the third image feature into the first decoding unit DU1 for processing and outputs the DU1 output result; inputs the DU1 output result into the second decoding unit DU2 for processing to obtain the DU2 output result, and inputs the DU1 output result and the DU2 output result into the first splicing layer for splicing processing to output the first decoded feature.

[0131] Further, the first decoding feature is input to the third decoding unit DU3, which outputs the DU3 output. The DU3 output and the DU2 output are then input to the second splicing layer for splicing to obtain the second decoding feature. The second decoding feature is input to the fourth decoding unit DU4 for processing to obtain the DU4 output. The DU4 output and the DU3 output are then input to the third splicing layer for splicing to obtain the third decoding feature. The third decoding feature is input to the sixth decoding unit DU6 to determine the DU6 output. The DU4 output is input to the fifth decoding unit DU5 for processing to obtain the DU5 output. The DU5 and DU6 outputs are then input to the fourth splicing layer for splicing to obtain the fourth decoding feature. The fourth decoding feature is input to the seventh decoding unit DU7 for processing to obtain the DU7 output. The DU6 and DU7 outputs are then input to the fifth splicing layer for splicing to output the fifth decoding feature. The fifth decoding feature is input to the eighth decoding unit DU8 for processing to obtain the DU8 output. The DU8, DU7, and DU5 outputs are then input to the sixth splicing layer for splicing to output the sixth decoding feature. The sixth decoded feature is input into the normalization submodule BG for normalization processing, outputting the fourth image feature. The dynamic convolutional layer in the decoding unit optimizes the feature, restoring local structure and texture. The upsampling layer improves spatial resolution through interpolation and other methods. Finally, the decoding unit outputs the fourth image feature, solving the problem of detail loss in deep networks and enriching semantic features.

[0132] Furthermore, for at least one detection head module, the fourth image feature is processed according to the at least one detection head module to obtain the fifth image feature.

[0133] In this embodiment, the fifth image feature is a list of detection regions generated by at least one detection head module.

[0134] For example, three detection head modules are pre-configured, each with different detection region sizes and the number of detection boxes. Specifically, the detection region sizes for the three modules are set to 82×82 pixels, 56×56 pixels, and 23×23 pixels, respectively, with three detection regions generated for each size. After the fourth image feature is input in parallel to the three detection head modules, it is first processed by the detection head units within each module. The three detection head units map the third image feature onto the 82×82, 56×56, and 23×23 detection regions, respectively, thereby generating prediction matrices corresponding to the three detection head units. Each prediction matrix contains the predicted coordinate offsets of the four vertices of the three detection regions at the corresponding size, as well as the confidence level of the predicted region.

[0135] Furthermore, the prediction matrices output by the three detection head units are input into their respective feature processing units for processing. The feature processing unit predicts the offset based on the size information of the corresponding detection region and the coordinates in the prediction matrix, calculating the absolute coordinates of each detection box in the image to be detected. The calculated absolute coordinates are concatenated with the corresponding confidence scores to generate a list of detection regions corresponding to the detection head modules. This list contains the vertex coordinates and confidence scores of each predicted region. The list of detection regions generated by all detection head modules is then used as the fifth image feature output.

[0136] Specifically, the fourth image feature is input in parallel to at least one detection head module for calculation and analysis to generate a list of detection regions corresponding to at least one detection head module, which is then output as the fifth image feature. Through a multi-detector parallel inference medium, the highly discriminative features extracted by the aforementioned modules are transformed into detection results that can be directly used for production quality inspection.

[0137] Furthermore, for the output module, the features of the fifth image are analyzed and processed based on the output module to obtain the foreign object detection result.

[0138] In this embodiment, the output module is an algorithmic logic unit at the end of the foreign object detection model used to generate the final foreign object detection result. It is responsible for performing post-processing and outputting the result. Specifically, the output module includes at least the execution of a non-maximum suppression algorithm to remove redundant predicted regions from the detection box list. That is, among multiple overlapping predicted regions of the same target, only the predicted region with the highest confidence is retained as the target region, and the remaining predicted regions are removed.

[0139] The process of using the non-maximum suppression algorithm to filter at least one predicted region is as follows: After acquiring the fifth image features output by at least one detection head module, all predicted regions in the fifth image features are sorted from high to low confidence. Further, iterative selection is performed. First, the predicted region with the highest confidence is selected, and the overlap area between the predicted region with the highest confidence and all remaining predicted regions is calculated. A preset filtering threshold is set, and all predicted regions whose overlap area with the current highest confidence predicted region exceeds the preset filtering threshold are deleted from the remaining predicted regions. Further, the remaining predicted regions are sorted from high to low confidence, and the above operation is repeated until no predicted regions remain, indicating that only one predicted region is retained for a target foreign object. Finally, the retained predicted region is taken as the target region, and the target region, its corresponding size information, the coordinates of at least one vertex of the target region, and the confidence level corresponding to the coordinates of at least one vertex of the target region are output as the foreign object detection result.

[0140] Specifically, the fifth image features output by at least one detection head module are obtained, and the output module performs filtering processing on the fifth image features to obtain the final determined foreign object detection result.

[0141] The technical solution provided by the embodiments of the present invention acquires an image of the semiconductor to be detected, and sequentially inputs the image of the semiconductor to be detected into a relative gradient position encoding module, a multiple feature interleaving attention module, an encoding and decoding module, at least one detection head module, and an output module. The relative gradient position encoding module analyzes and processes the image to be detected to determine the first image features, thereby extracting finer-grained geometric and texture features of foreign objects in the image and improving the contrast between features. Further, the first image features are analyzed and processed by at least one feature interleaving attention module to obtain the second image features. The second image features output by at least one feature interleaving attention module are then concatenated to enhance the model's capture of key contextual information. Further, the features output by at least one feature interleaving attention module are analyzed and processed by the decoding encoding module to obtain the fourth image features, achieving efficient data representation and reconstruction, removing redundant information, and preserving the synergistic relationship between global and local information. Further, the fourth image features are processed by at least one detection head module to obtain the fifth image features, enabling the model to capture small targets while also focusing on the global information of large targets. Finally, the foreign object detection result is output by the output module based on the fifth image features output by at least one detection head module, improving the model's detection accuracy and robustness.

[0142] Figure 8 This is a flowchart of a semiconductor foreign object detection method provided by an embodiment of the present invention. Based on the foregoing embodiments, this embodiment elaborates on the training process of the foreign object detection model. Figure 8 As shown, the method includes: S210. Obtain multiple sample images for training the foreign object detection model.

[0143] Specifically, multiple sample images are used as input data to train the foreign object detection model.

[0144] S220. For multiple sample images, input the sample images into the initial foreign object detection model so that the foreign object detection model outputs sample prediction data.

[0145] The initial foreign object detection model is an untrained foreign object detection model, which includes a relative gradient position encoding module to be learned, at least one feature interleaving attention module, an encoding and decoding module, at least one detection head module, and an output module.

[0146] In this embodiment, the initial foreign object detection model is an initial neural network model whose parameters have not been adjusted through data training. The weights of the untrained initial foreign object detection model are typically randomly initialized. The sample prediction data is the prediction result output by the initial foreign object detection model through forward inference of the input sample image. The sample prediction data includes at least the predicted region containing the foreign object, the size information corresponding to the predicted region, the coordinates of at least one vertex of the predicted region, and the confidence level corresponding to the coordinates of at least one vertex of the predicted region.

[0147] Specifically, multiple sample images are sequentially input into an untrained initial foreign object detection model, and the model is analyzed and processed to output the sample prediction data corresponding to the sample images.

[0148] S230. Based on the sample prediction data, sample label data, first parameter, second parameter, and the loss function corresponding to the initial foreign object detection model, adjust the model parameters in the initial foreign object detection model.

[0149] The loss function includes the crossover ratio loss term, distance loss term, and shape area loss term of the foreign object detection results. The first parameter is the distance loss parameter determined based on the previous iteration, and the second parameter is the shape area loss parameter determined based on the previous iteration. In this embodiment, the loss function corresponding to the initial anomaly detection model is an error function used for model training, which quantifies the difference between the sample predicted data and the sample label data. During training, by minimizing this function value, the model is guided to learn how to accurately identify foreign objects and outline their locations.

[0150] The intersection-union ratio loss term is a loss function that calculates the ratio of the intersection area to the union area of ​​the predicted region and the label region in the sample prediction data, in order to maximize the overlap area between the predicted region and the label region.

[0151] The distance loss term penalizes the Euclidean distance between the center point of the predicted region and the center point of the label region. Based on the sample prediction data, the width and height of the predicted region are determined, and the coordinates of the center point, width, and height of the predicted region are calculated. Based on the sample label data, the height and width of the label region are determined, and the center coordinates and diagonal length of the label region are calculated. Based on the center point coordinates, width, height, and diagonal length of the predicted region, the distance loss is determined, i.e.: ; in, denoted as distance loss value, and b as the coordinates of the center point of the prediction region, including the x-coordinate and y-coordinate of the center point. The coordinates of the center point of the label area, including the x-coordinate of the center point. and the vertical axis . The squared Euclidean distance between the center point of the predicted region and the center point of the label region. W is the diagonal length of the label region, H is the width of the prediction region, and H is the height of the prediction region.

[0152] The shape-area loss term is a loss function that penalizes the relative errors in width and height between the predicted region and the label region. Furthermore, smaller targets are given stronger loss weights through coefficients. The width and height of the predicted region are determined based on the sample prediction data, and the width and height of the label region are determined based on the sample label data.

[0153] First, calculate the relative error between the predicted region and the label region based on width, that is: ; in, To predict the width of the region, The width of the label area.

[0154] And calculate the relative error between the predicted region and the labeled region based on high, that is: ; in, To predict the width of the region, The width of the label area.

[0155] Furthermore, based on the width and height of the predicted region, the width and height of the label region, and the relative error between the height and width, the shape area loss value is determined, i.e.: ; in, This is the area of ​​the label region, calculated from its height and width. This is for shape area loss. By exponentially penalizing the relative errors of width and height, the model can fit the shape more accurately.

[0156] The first parameter is the distance loss parameter determined in the previous iteration. The previous iteration can be understood as the training iterations completed before the current model parameters are updated, used to adjust the hyperparameters. The second parameter is the shape-area loss parameter determined in the previous iteration.

[0157] The loss function of the foreign object detection model optimizes object detection performance across multiple dimensions by combining intersection-union (IU), distance, and shape-area loss terms. Furthermore, during training, a first and a second parameter are introduced to control the different loss terms. The complete expression for the loss function of the foreign object detection model is: ; in, Here is the loss function corresponding to the foreign object detection model. The first parameter is the weighting factor corresponding to the distance loss term. This is the weighting factor corresponding to the shape area loss term, i.e., the second parameter.

[0158] It should be noted that the strategy of dynamically adjusting the weight factors adjusts the first and second parameters corresponding to the distance loss and shape / area loss terms based on the data characteristics of the current batch during model training. A batch can be understood as the model dividing multiple sample data into multiple batches for batch training. Based on the distance loss and shape / area loss corresponding to all sample images in the current batch, the first parameter corresponding to the distance loss in the current batch, and the second parameter corresponding to the shape / area loss in the current batch, the weight factors corresponding to the distance loss in the next batch and the weight factors corresponding to the shape / area loss in the current batch are determined, i.e.: ; in, The weighting factor corresponding to the distance loss in the next batch. This represents the weighting factor corresponding to the shape-area loss in the next batch. j represents the j-th sample image in a batch. N is the batch size, representing the number of sample images in a batch.

[0159] Specifically, the loss function of the foreign object detection model consists of an intersection-union (IU) loss term, a distance loss term, and a shape-area loss term. The IU loss term maximizes the overlap between the predicted region and the label region, ensuring tight localization. The distance loss term optimizes the positional offset of the predicted region. The shape-area loss term ensures that the predicted region matches the label region in shape and size, and improves the model's sensitivity to small targets. Furthermore, a dynamically adjusted weight factor is introduced during training. The first and second parameters corresponding to the distance loss term and the shape-area loss term in each batch are dynamically adjusted to optimize model training and balance regression accuracy and generalization performance.

[0160] S240. When the loss function converges, the initial foreign object detection model obtained is taken as the foreign object detection model.

[0161] In this embodiment, loss convergence can be understood as the loss function value gradually decreasing and stabilizing within a small range as the number of iterations increases during model training, without any significant decrease. Training stops when the loss function converges. Training stopping can be understood as the moment when a preset stopping condition is reached during model training. The preset stopping condition can be the training reaching a preset maximum number of iterations. The preset maximum number of iterations is determined based on multiple experimental data.

[0162] Specifically, training ends when the loss function converges, and the model at this point is used as the final deployable foreign object detection model.

[0163] The technical solution provided by this invention involves acquiring multiple sample images for training a foreign object detection model. Further, for each sample image, the images are input into an initial foreign object detection model to output sample prediction data. Based on the sample prediction data, sample label data, a first parameter, a second parameter, and the loss function corresponding to the initial foreign object detection model, the model parameters in the initial foreign object detection model are adjusted. The loss function includes an intersection-over-union (IoU) loss term, a distance loss term, and a shape-area loss term for the foreign object detection results. The first parameter is a distance loss parameter determined based on the previous iteration, and the second parameter is a shape-area loss parameter determined based on the previous iteration. When the loss function converges, the resulting initial foreign object detection model is used as the foreign object detection model. Through the multi-task loss function, the model is guided to learn the spatial, shape, and scale features of foreign objects from the data, achieving high-precision and highly robust foreign object detection capabilities.

[0164] Figure 9 This is an overall framework diagram of a semiconductor foreign object detection method provided in an embodiment of the present invention, combined with... Figure 9 Understand the technical solutions of the embodiments of the present invention.

[0165] like Figure 9 As shown, this embodiment explains the overall implementation of the solution based on the above optional implementation methods. Specifically, it includes: The system performs data preprocessing operations on the input image containing the semiconductor to be detected, including morphological operations, image enhancement operations, and uniform scaling operations, outputting the preprocessed image. This preprocessed image is then input into a pre-trained foreign object detection model for analysis and processing.

[0166] First, the image to be detected is input into the relative gradient position encoding module. For each pixel in the image, the second-order gradients in four main directions are extracted, including the horizontal, vertical, and two orthogonal diagonal directions, using a preset neighborhood as the calculation range. Based on the calculated second-order gradient values, a local structure matrix is ​​constructed for each pixel in the second image. To adjust the numerical range of the gradient features, the local structure matrix is ​​normalized to obtain the normalized value for each element, ultimately determining the features of the first image.

[0167] Furthermore, the first image features are synchronously input into multiple feature-interleaved attention modules, incorporating depthwise convolution, GeLU activation function, ReLU activation function, and global average pooling. By combining local feature enhancement and global attention mechanisms, the capture of key information is enhanced. The image features output from the multiple feature-interleaved attention modules are then concatenated to obtain the second image features.

[0168] Furthermore, the second image features are input into the encoding unit (encoding part) of the encoding and decoding module. The feature dimension is compressed through dynamic convolution and downsampling modules to extract deep semantic information, resulting in the third image features. The third image features are then input into the decoding unit (decoding part) of the encoding and decoding module. Spatial resolution is gradually restored through multiple dynamic convolutional layers and multiple upsampling layers. The deep semantic information is combined with the shallow spatial features to retain key information, outputting the fourth image features.

[0169] Furthermore, the fourth image feature is input into at least one detection head module for analysis and processing to obtain the fifth image feature, thereby capturing multi-scale features. The fifth image feature is then input into the output module to obtain the foreign object detection result. Finally, the location information of the foreign object in the image to be detected is determined based on the foreign object detection result.

[0170] The technical solution provided by this invention acquires an image of the semiconductor to be detected, and inputs the image into a pre-trained foreign object detection model for analysis and processing. The image undergoes at least a relative gradient position encoding module, multiple feature interleaving attention modules, and an encoding / decoding module. By enhancing the corresponding features of the image, the foreign object detection result is output, thereby improving the accuracy of foreign object detection. Based on the determined foreign object detection result, the location information of the foreign object in the image is determined. In summary, the technical solution of this invention effectively improves the accuracy and efficiency of detecting solder joint anomalies and minute foreign object defects in semiconductor X-ray images.

[0171] Figure 10 This is a schematic diagram of a semiconductor foreign object detection device provided in an embodiment of the present invention, as shown below. Figure 10 As shown, the device includes: an image acquisition module 310, a detection result output module 320, and a foreign object location information determination module 330.

[0172] The image acquisition module 310 is used to acquire an image to be detected, wherein the image to be detected is an X-ray image including the semiconductor to be detected; the detection result output module 320 is used to input the image to be detected into a pre-trained foreign object detection model for analysis and processing, so as to output a foreign object detection result; wherein the foreign object detection model includes at least a relative gradient position encoding module, multiple feature interleaving attention modules, and an encoding and decoding module, wherein the relative gradient position encoding module is used to encode the structured gradient features of the image to be detected, the feature interleaving attention module is used to perform first feature cross-attention processing on the output of the relative gradient position encoding module, and the encoding and decoding module is used to perform second feature multi-level dynamic encoding and decoding processing on the output of the feature interleaving attention module; The foreign object location information determination module 330 is used to determine the location information of the foreign object in the image to be detected based on the foreign object detection result.

[0173] The technical solution provided by this invention acquires an image of the semiconductor to be detected, and inputs the image into a pre-trained foreign object detection model for analysis and processing. The image undergoes at least a relative gradient position encoding module, multiple feature interleaving attention modules, and an encoding / decoding module. By enhancing the corresponding features of the image, the foreign object detection result is output, thereby improving the accuracy of foreign object detection. Based on the determined foreign object detection result, the location information of the foreign object in the image is determined. In summary, the technical solution of this invention effectively improves the accuracy and efficiency of detecting solder joint anomalies and minute foreign object defects in semiconductor X-ray images.

[0174] Based on the above technical solution, the foreign object detection model according to the input-output relationship includes the relative gradient position encoding module, multiple feature interleaving attention modules, the encoding and decoding module, at least one detection head module, and an output module; Based on the above technical solution, the relative gradient position encoding module processes the image to be detected in the following manner: The second-order gradient value determination module is used to determine the second-order gradient value of at least one pixel to be processed in the image to be detected in multiple directions based on the first pixel value of the pixel to be processed and the second pixel value of the associated pixels in a preset neighborhood. The multiple directions include a first direction, a second direction, a third direction, and a fourth direction. The third direction is a direction that forms a first preset angle with the first direction, and the fourth direction is a direction that forms a second preset angle with the first direction. The first preset angle and the second preset angle are complementary. The local structure matrix determination module is used to determine the local structure matrix of the at least one pixel to be processed based on the second-order gradient values ​​of the at least one pixel to be processed in each direction. A first image feature determination module is used to obtain a first image feature by processing the at least one pixel point, wherein the first image feature is the image feature output by the relative gradient position encoding module.

[0175] Based on the above technical solution, the multiple feature interleaving attention modules are arranged in parallel, and each feature interleaving attention module processes the first image features output by the relative gradient position encoding module in the following manner: The first enhanced feature determination module is used to perform feature enhancement processing on the first image features based on at least two first depthwise convolutional units respectively, to obtain the first enhanced feature output by each first depthwise convolutional unit; The second enhanced feature determination module is used to perform feature enhancement processing on the first image features based on the second depthwise convolution unit to obtain the second enhanced features; The fourth feature enhancement determination module is used to obtain a third enhancement feature by superimposing two first enhancement features and inputting them into the next first depthwise convolution unit, and then aggregate the third enhancement feature based on the global average pooling unit to obtain a fourth enhancement feature. The sixth enhancement feature module is used to perform feature enhancement processing on the fourth enhancement feature based on at least two first depthwise convolutional units respectively, and the fifth enhancement feature output by each first depthwise convolutional unit is input into the next first depthwise convolutional unit after the two fifth enhancement features are superimposed to obtain the sixth enhancement feature; The seventh enhancement feature determination module is used to process the second enhancement feature based on the signal function to obtain the seventh enhancement feature; The second image feature determination module is used to determine the second image features based on the sixth enhanced feature and the seventh enhanced feature; Wherein, the second image feature is the feature output by the feature interleaving attention module; the first depthwise convolutional unit includes at least a depthwise convolutional layer, a batch normalization layer and an activation function, the global average pooling unit includes at least a global average pooling layer and the activation function, and the second depthwise convolutional unit includes at least the depthwise convolutional layer and the batch normalization layer.

[0176] Based on the above technical solution, the encoding and decoding module includes multiple encoding units, and the multiple encoding units in the encoding and decoding module process the second image features in the following manner: The first feature determination module is used to process the second image features based on the normalization submodule to obtain the first feature to be used, wherein the normalization submodule includes a normalization function and an activation function; The first feature output module is used to obtain the output results of the (i-1)th and (i-2)th coding units when processing based on the i-th coding unit for multiple coding units, and to obtain the current output result of the i-th coding unit. When i is a first preset value, the output result of the (i-2)th coding unit is the first feature to be used output by the normalization submodule. The output result processing module is used to obtain the output result of the (i-1)th encoding unit and the output result of the (i-3)th encoding unit when the i-th encoding unit is the first preset encoding unit, so as to process the output result based on the i-th preset encoding unit; The second feature determination module is used to obtain the usable result corresponding to the odd-numbered coding units when the i-th preset coding unit is the second preset coding unit, and to obtain the second feature to be used based on the usable output result; The third image feature acquisition module is used to obtain the third image feature through the second feature to be used and the usable output result.

[0177] Based on the above technical solution, the encoding unit includes multiple dynamic convolutional layers and multiple downsampling layers. The dynamic convolutional layer includes at least a dynamic convolutional group and the activation function, and at least part of the dynamic convolutional layer and the downsampling layer are connected.

[0178] Based on the above technical solution, the encoding / decoding module includes a decoding unit. At least one decoding unit and a splicing layer constitute a decoding submodule. The multiple decoding units in the encoding / decoding module process the third image features in the following manner: The first decoding feature output module is used to process the third image features based on the first decoding submodule and output the first decoding feature. The first decoding submodule includes a first decoding unit, a second decoding unit, and a first splicing layer. The first splicing layer is used to splice the features output by the first decoding unit and the second decoding unit. The second decoding feature output module is used to process the first decoding feature and the feature output by the second decoding unit based on the second decoding submodule to obtain the second decoding feature; wherein, the second decoding submodule includes a third decoding unit and a second splicing layer, and the second splicing layer is used to splice the features output by the third decoding unit and the second decoding unit; The third decoding feature output module is used to process the second decoding feature and the feature output by the third encoding unit based on the third decoding submodule to obtain the third decoding feature. The third decoding submodule includes a fourth decoding unit and a third splicing layer. The third splicing layer is used to process the features output by the third decoding unit and the fourth decoding unit. The fourth decoding feature output module is used to process the features output by the fourth decoding unit and the third decoding feature based on the fourth decoding submodule to obtain the fourth decoding feature. The fourth decoding submodule includes a fifth decoding unit, a sixth decoding unit and a fourth splicing layer. The fourth splicing layer is used to splice the features output by the sixth encoding unit and the features output by the fifth encoding unit. The fifth decoding feature output module is used to process the features output by the sixth decoding unit and the fourth decoding feature based on the fifth decoding submodule to obtain the fifth decoding feature. The fifth decoding submodule includes a seventh decoding unit and a fifth splicing layer. The fifth splicing layer is used to splice the features output by the sixth decoding unit and the features output by the seventh decoding unit. The sixth decoding feature output module is used to process the fifth decoding feature and the feature output by the fifth decoding unit based on the sixth decoding submodule to obtain the sixth decoding feature. The sixth decoding submodule includes an eighth decoding unit and a sixth splicing layer. The sixth splicing layer is used to splice the feature output by the eighth decoding unit, the feature output by the fifth decoding unit, and the feature output by the seventh decoding unit. The fourth image feature output module is used to process the sixth decoding feature based on the normalization submodule to obtain the fourth image feature.

[0179] Based on the above technical solution, the decoding unit includes multiple dynamic convolutional layers and multiple upsampling layers. The dynamic convolutional layer includes at least a dynamic convolutional group and the activation function, and at least a portion of the dynamic convolutional layer and the upsampling layer are connected.

[0180] Based on the above technical solution, the model parameters in the foreign object detection model are determined in the following manner: The sample image acquisition module is used to acquire multiple sample images for training the foreign object detection model. The sample prediction data output module is used to input the sample images into the initial foreign object detection model for multiple sample images, so that the foreign object detection model outputs sample prediction data. The model parameter adjustment module is used to adjust the model parameters in the initial foreign object detection model based on the sample prediction data, the sample label data, the first parameter, the second parameter, and the loss function corresponding to the initial foreign object detection model. The loss function includes the cross-union ratio loss term, distance loss term, and shape area loss term of the foreign object detection result. The first parameter is the distance loss parameter determined based on the previous iteration, and the second parameter is the shape area loss parameter determined based on the previous iteration. The foreign object detection model acquisition module is used to obtain the initial foreign object detection model when the loss function converges as the foreign object detection model.

[0181] Based on the above technical solution, the foreign object detection result shall at least include the region marker containing the region where the foreign object is located, the region size, and the region confidence level.

[0182] The watermark embedding device in generated text provided in the embodiments of the present invention can execute a semiconductor foreign object detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method.

[0183] Figure 11 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0184] like Figure 11 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0185] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0186] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as a semiconductor foreign object detection method.

[0187] In some embodiments, a semiconductor foreign object detection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the semiconductor foreign object detection method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform a semiconductor foreign object detection method by any other suitable means (e.g., by means of firmware).

[0188] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0189] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0190] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0191] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0192] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0193] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0194] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0195] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for detecting semiconductor foreign objects, characterized in that, include: Acquire an image to be detected; wherein the image to be detected is an X-ray image including the semiconductor to be detected; The image to be detected is input into a pre-trained foreign object detection model for analysis and processing to output foreign object detection results. The foreign object detection model includes at least a relative gradient position encoding module, multiple feature interleaving attention modules, and an encoding / decoding module. The relative gradient position encoding module is used to encode the structured gradient features of the image to be detected. The feature interleaving attention module is used to perform cross-attention processing on the first feature output by the relative gradient position encoding module. The encoding / decoding module is used to perform multi-level dynamic encoding and decoding processing on the second feature output by the feature interleaving attention module. Based on the foreign object detection results, the location information of the foreign object in the image to be detected is determined.

2. The method according to claim 1, characterized in that, The foreign object detection model based on the input-output relationship includes the relative gradient position encoding module, multiple feature interleaving attention modules, the encoding and decoding module, at least one detection head module, and an output module.

3. The method according to claim 2, characterized in that, The relative gradient position encoding module processes the image to be detected in the following manner: For at least one pixel to be processed in the image to be detected, the second-order gradient value of the pixel to be processed in multiple directions is determined based on the first pixel value of the pixel to be processed and the second pixel value of the associated pixels in the preset neighborhood. The multiple directions include a first direction, a second direction, a third direction, and a fourth direction. The third direction is a direction that forms a first preset angle with the first direction, and the fourth direction is a direction that forms a second preset angle with the first direction. The first preset angle and the second preset angle are complementary. The local structure matrix of the at least one pixel to be processed is determined based on the second-order gradient values ​​of the at least one pixel to be processed in each direction. A first image feature is obtained by processing the at least one pixel, wherein the first image feature is the image feature output by the relative gradient position encoding module.

4. The method according to claim 2, characterized in that, The multiple feature-interleaved attention modules are arranged in parallel, and each feature-interleaved attention module processes the first image features output by the relative gradient position encoding module in the following manner: The first image features are enhanced based on at least two first depthwise convolutional units to obtain the first enhanced features output by each first depthwise convolutional unit. Based on the second depthwise convolutional unit, feature enhancement processing is performed on the first image features to obtain the second enhanced features; The third enhanced feature is obtained by superimposing the two first enhanced features and inputting them into the next first depthwise convolutional unit. The third enhanced feature is then aggregated based on the global average pooling unit to obtain the fourth enhanced feature. The fourth enhanced feature is enhanced by performing feature enhancement processing on the first enhanced feature based on at least two first depthwise convolutional units respectively, and the fifth enhanced feature output by each first depthwise convolutional unit is obtained. The second enhanced feature is then superimposed and input into the next first depthwise convolutional unit to obtain the sixth enhanced feature. The second enhancement feature is processed based on the signal function to obtain the seventh enhancement feature; Based on the sixth and seventh enhancement features, the second image features are determined; Wherein, the second image feature is the feature output by the feature interleaving attention module; the first depthwise convolutional unit includes at least a depthwise convolutional layer, a batch normalization layer and an activation function, the global average pooling unit includes at least a global average pooling layer and the activation function, and the second depthwise convolutional unit includes at least the depthwise convolutional layer and the batch normalization layer.

5. The method according to claim 2, characterized in that, The encoding / decoding module includes multiple encoding units, and the multiple encoding units in the encoding / decoding module process the second image features based on the following method: The second image feature is processed based on the normalization submodule to obtain the first feature to be used, wherein the normalization submodule includes a normalization function and an activation function; For multiple coding units, when processing based on the i-th coding unit, the output results of the (i-1)-th coding unit and the (i-2)-th coding unit are obtained to obtain the current output result of the i-th coding unit. Where, when i is a first preset value, the output result of the (i-2)-th coding unit is the first feature to be used output by the normalization submodule. When the i-th encoding unit is the first preset encoding unit, the output results of the (i-1)-th encoding unit and the (i-3)-th encoding unit are obtained, and the output results are processed based on the i-th preset encoding unit; When the i-th preset coding unit is the second preset coding unit, the usable result corresponding to the odd-numbered coding units is obtained, and based on the usable output result, the second feature to be used is obtained; The third image feature is obtained by using the second feature to be used and the usable output result.

6. The method according to claim 5, characterized in that, The encoding unit includes multiple dynamic convolutional layers and multiple downsampling layers. The dynamic convolutional layer includes at least a dynamic convolutional group and the activation function, and at least a portion of the dynamic convolutional layer and the downsampling layer are connected.

7. The method according to claim 2, characterized in that, The encoding / decoding module includes a decoding unit. At least one decoding unit and a splicing layer constitute a decoding submodule. The multiple decoding units in the encoding / decoding module process the third image features based on the following method: The third image features are processed based on the first decoding submodule to output the first decoded features. The first decoding submodule includes a first decoding unit, a second decoding unit, and a first splicing layer. The first splicing layer is used to splice the features output by the first decoding unit and the second decoding unit. The second decoding submodule processes the first decoding feature and the feature output by the second decoding unit to obtain the second decoding feature; wherein, the second decoding submodule includes a third decoding unit and a second splicing layer, and the second splicing layer is used to splice the features output by the third decoding unit and the second decoding unit; The third decoding feature is obtained by processing the second decoding feature and the feature output by the third encoding unit based on the third decoding submodule. The third decoding submodule includes a fourth decoding unit and a third splicing layer. The third splicing layer is used to process the features output by the third decoding unit and the fourth decoding unit. The fourth decoding submodule processes the features output by the fourth decoding unit and the third decoding features to obtain the fourth decoding features. The fourth decoding submodule includes a fifth decoding unit, a sixth decoding unit and a fourth splicing layer. The fourth splicing layer is used to splice the features output by the sixth encoding unit and the features output by the fifth encoding unit. The fifth decoding submodule processes the features output by the sixth decoding unit and the fourth decoding features to obtain the fifth decoding feature. The fifth decoding submodule includes a seventh decoding unit and a fifth splicing layer. The fifth splicing layer is used to splice the features output by the sixth decoding unit and the features output by the seventh decoding unit. The sixth decoding feature is obtained by processing the fifth decoding feature and the feature output by the fifth decoding unit based on the sixth decoding submodule. The sixth decoding submodule includes an eighth decoding unit and a sixth splicing layer. The sixth splicing layer is used to splice the feature output by the eighth decoding unit, the feature output by the fifth decoding unit, and the feature output by the seventh decoding unit. The sixth decoding feature is processed based on the normalization submodule to obtain the fourth image feature.

8. The method according to claim 7, characterized in that, The decoding unit includes multiple dynamic convolutional layers and multiple upsampling layers. Each dynamic convolutional layer includes at least a dynamic convolutional group and the activation function. At least a portion of the dynamic convolutional layer and the upsampling layer are connected.

9. The method according to claim 1, characterized in that, The model parameters in the foreign object detection model are determined based on the following method: Acquire multiple sample images for training the foreign object detection model; For multiple sample images, the sample images are input into the initial foreign object detection model so that the foreign object detection model outputs sample prediction data; Based on the sample prediction data, the sample label data, the first parameter, the second parameter, and the loss function corresponding to the initial foreign object detection model, the model parameters in the initial foreign object detection model are adjusted; The loss function includes the cross-union ratio loss term, distance loss term, and shape area loss term of the foreign object detection result. The first parameter is the distance loss parameter determined based on the previous iteration, and the second parameter is the shape area loss parameter determined based on the previous iteration. When the loss function converges, the initial foreign object detection model obtained is taken as the foreign object detection model.

10. The method according to claim 1, characterized in that, The foreign object detection results include at least the region marker containing the area where the foreign object is located, the region size, and the region confidence level.

Images