Semiconductor optoelectronic chip anomaly detection method and apparatus, and computing device

Abstract

This invention discloses a method, apparatus, and computing device for anomaly detection in semiconductor optoelectronic chips. The method includes: loading a first target chip image based on a first color model and converting it into a second target chip image based on a second color model, the second color model including a red channel, a saturation channel, and a brightness channel; performing superpixel segmentation on the second target chip image to obtain a target chip superpixel image, and acquiring the feature vector of each target superpixel in the target chip superpixel image; predicting multiple target key regions in the target chip superpixel image using a target visual large model; generating a corresponding OCSVM model for each target key region based on the feature vectors of each target superpixel; and performing anomaly detection on any target key region based on the OCSVM model corresponding to any target key region. This invention can perform accurate anomaly detection on a single key region of a semiconductor optoelectronic chip.

Description

Technical Field

[0001] This invention relates to the field of anomaly detection technology, and in particular to a method, device and computing equipment for anomaly detection of semiconductor optoelectronic chips. Background Technology

[0002] Semiconductor optoelectronic chips, as core components of modern electronics and lighting technologies, play a crucial role in various devices. Their high efficiency, long lifespan, and strong reliability have led to their widespread application in high-performance electronic equipment and various lighting applications. Despite the high production efficiency of semiconductor optoelectronic chips, the daily quality inspection volume is limited to only seven to eight hundred chips. Furthermore, anomalies on semiconductor optoelectronic chips are typically distributed across different areas, and multiple anomalies can occur simultaneously, making it difficult to identify all anomalies on the semiconductor optoelectronic chip at once.

[0003] In existing technologies, deep learning models or machine learning methods are generally used for anomaly detection in semiconductor optoelectronic chips. Deep learning models often require a large number of pre-labeled negative samples for training, which is disadvantageous for anomaly detection in semiconductor optoelectronic chips where negative samples are already scarce. Traditional machine learning methods for anomaly detection in semiconductor optoelectronic chips have the following problems: 1) Anomaly detection based on black-and-white image features yields insufficient information, leading to inaccurate detection results; 2) The lack of segmentation of key regions makes it impossible to detect anomalies in these areas, resulting in difficulty identifying small-sized anomaly regions.

[0004] Therefore, a method for detecting anomalies in semiconductor optoelectronic chips is needed to solve the problems existing in the above-mentioned technical solutions. Summary of the Invention

[0005] Therefore, the present invention provides a method and apparatus for detecting anomalies in semiconductor optoelectronic chips to solve or at least alleviate the problems mentioned above.

[0006] According to one aspect of the present invention, a method for anomaly detection of a semiconductor optoelectronic chip is provided, executed in a computing device, comprising: loading a first target chip image based on a first color model of a target semiconductor optoelectronic chip; converting the first target chip image into a second target chip image based on a second color model, the second color model including a red channel, a saturation channel, and a brightness channel; performing superpixel segmentation on the second target chip image to obtain a target chip superpixel image containing multiple target superpixels, and acquiring a feature vector of each target superpixel in the target chip superpixel image, the feature vector including the average value of the red channel, the average value of the saturation channel, and the average value of the brightness channel; predicting multiple target key regions in the target chip superpixel image using a target visual large model; generating a corresponding OCSVM model for each target key region based on the feature vector of each target superpixel in the target chip superpixel image; and performing anomaly detection on any target key region based on the OCSVM model corresponding to any target key region to determine an abnormal region in any target key region.

[0007] Optionally, in the semiconductor optoelectronic chip anomaly detection method according to the present invention, predicting multiple target key regions in the superpixel image of the target chip using a target visual large model includes: determining the average intensity value of each target superpixel in the target chip superpixel image; dividing the target chip superpixel image into multiple target initial key regions based on the average intensity value of each target superpixel; randomly generating a predetermined number of target cue points for each target initial key region as target cue point information for the target initial key region; inputting the target chip superpixel image and target cue point information into the target visual large model; and using the target visual large model to predict multiple target key regions in the target chip superpixel image based on the target cue point information of each target initial key region.

[0008] Optionally, in the semiconductor optoelectronic chip anomaly detection method according to the present invention, before predicting multiple target key regions in the superpixel image of the target chip using the target visual large model, the method further includes: using a freeze-fine-tuning method to fine-tune a pre-trained SAM2 visual large model based on multiple sample chip images with labeled information of semiconductor optoelectronic chips of various configurations, including freezing the parameters of the image encoder in the pre-trained SAM2 visual large model, and fine-tuning the parameters of the cue encoder and mask decoder in the pre-trained SAM2 visual large model to obtain the target visual large model.

[0009] Optionally, in the semiconductor optoelectronic chip anomaly detection method according to the present invention, the annotation information is used to identify the true key regions of each sample in the sample chip image; fine-tuning the pre-trained SAM2 visual large model based on multiple sample chip images with annotation information of semiconductor optoelectronic chips of various configurations to obtain the target visual large model includes: performing superpixel segmentation on the sample chip image to obtain a sample chip superpixel image containing multiple sample superpixels; determining the average intensity value of each sample superpixel, and dividing the sample chip superpixel image into multiple sample initial key regions based on the average intensity value of each sample superpixel; for each sample initial... For key regions, a predetermined number of sample cue points are randomly generated as the initial key region cue point information for the samples. The superpixel image of the sample chip and the sample cue point information are input into the pre-trained SAM2 visual large model. Using the pre-trained SAM2 visual large model, based on the sample cue point information of each initial key region of the samples, the predicted key regions of each sample in the superpixel image of the sample chip are predicted. The loss value between each predicted key region and each true key region of the samples is calculated. Based on the loss value, the parameters of the cue encoder and mask decoder in the pre-trained SAM2 visual large model are fine-tuned to obtain the target visual large model.

[0010] Optionally, in the semiconductor optoelectronic chip anomaly detection method according to the present invention, the target visual large model includes a target image encoder, a memory attention module, a target cue encoder, a target mask decoder, a memory encoder, and a memory repository; wherein, the memory encoder is adapted to generate memory information based on each previously predicted key region by the target mask decoder and store it in the memory repository; using the target visual large model, based on the target cue point information of each of the initial key regions of the target, predicting multiple target key regions in the superpixel image of the target chip includes: encoding the superpixel image of the target chip through the target image encoder to obtain an initial image embedding, and adjusting the initial image embedding based on the memory information in the memory repository through the memory attention module to obtain a target image embedding; encoding the target cue point information of each of the initial key regions of the target through the target cue encoder to obtain a target cue embedding; predicting multiple target key regions in the superpixel image of the target chip through the target mask decoder based on the target image embedding and the target cue embedding; and generating new memory information based on each target key region through the memory encoder and storing it in the memory repository.

[0011] Optionally, in the semiconductor optoelectronic chip anomaly detection method according to the present invention, based on the feature vectors of each target superpixel in the superpixel image of the target chip, generating a corresponding OCSVM model for each target key region includes: combining the average value of the saturation channel and the average value of the brightness channel in the feature vectors of each target superpixel into a first feature group, and combining the average value of the red channel and the average value of the brightness channel in the feature vectors of each target superpixel into a second feature group; for each target key region, generating an OCSVM model corresponding to the target key region based on the first feature group and / or the second feature group.

[0012] Optionally, in the semiconductor optoelectronic chip anomaly detection method according to the present invention, the plurality of target key regions include electrode lead regions, light-emitting regions, and epitaxial regions.

[0013] Optionally, in the semiconductor optoelectronic chip anomaly detection method according to the present invention, performing superpixel segmentation on the second target chip image to obtain a target chip superpixel image containing multiple target superpixels includes: using the SLIC algorithm to perform superpixel segmentation on the second target chip image to obtain a target chip superpixel image containing multiple target superpixels.

[0014] Optionally, in the semiconductor optoelectronic chip anomaly detection method according to the present invention, the first color model is an RGB color model and the second color model is an RSV color model.

[0015] According to one aspect of the present invention, a semiconductor optoelectronic chip anomaly detection device is provided, deployed in a computing device. The device includes: a loading unit adapted to load a first target chip image based on a first color model of a target semiconductor optoelectronic chip; a conversion unit adapted to convert the first target chip image into a second target chip image based on a second color model, the second color model including a red channel, a saturation channel, and a brightness channel; a superpixel segmentation unit adapted to perform superpixel segmentation on the second target chip image to obtain a target chip superpixel image containing multiple target superpixels, and to acquire a feature vector of each target superpixel in the target chip superpixel image, the feature vector including the average value of the red channel, the average value of the saturation channel, and the average value of the brightness channel; a key region segmentation unit adapted to predict multiple target key regions in the target chip superpixel image using a target visual large model; a generation unit adapted to generate a corresponding OCSVM model for each target key region based on the feature vector of each target superpixel in the target chip superpixel image; and an anomaly detection unit adapted to perform anomaly detection on any target key region based on the OCSVM model corresponding to any target key region, so as to determine the abnormal region in any target key region.

[0016] According to one aspect of the present invention, a computing device is provided, comprising: at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, the program instructions including instructions for performing the semiconductor optoelectronic chip anomaly detection method as described above.

[0017] According to one aspect of the present invention, a computer program product is provided, comprising a computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the method as described above.

[0018] According to one aspect of the present invention, a readable storage medium storing program instructions is provided, which, when read and executed by a computing device, causes the computing device to perform the semiconductor optoelectronic chip anomaly detection method as described above.

[0019] According to the technical solution of the present invention, a method for anomaly detection in semiconductor optoelectronic chips is provided. This method involves converting a first target chip image based on an RGB color model into a second target chip image based on an RSV color model. Then, superpixel segmentation is performed on the second target chip image to obtain a superpixel image of the target chip. Multiple key target regions are extracted from the superpixel image using a large target visual model. Furthermore, an OCSVM model is generated for each key target region based on the feature vectors of each superpixel in the superpixel image. Thus, anomaly detection can be performed on any key target region based on the OCSVM model corresponding to any key target region. It is evident that the present invention performs superpixel segmentation and key region extraction on semiconductor optoelectronic chip images using an RSV color model. The RSV color model effectively highlights abnormal regions on the semiconductor optoelectronic chip, facilitating more accurate and efficient identification and detection of abnormal regions. The large target visual model obtained by fine-tuning according to the present invention is suitable for segmenting key regions of semiconductor optoelectronic chip images with various configurations and can accurately and efficiently extract multiple key regions from the semiconductor optoelectronic chip image. This enables precise anomaly detection for each key region of the semiconductor optoelectronic chip, improving the accuracy and precision of anomaly detection in semiconductor optoelectronic chips.

[0020] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0021] To achieve the foregoing and related objectives, certain illustrative aspects are described herein in conjunction with the following description and accompanying drawings. These aspects indicate various ways in which the principles disclosed herein may be practiced, and all aspects and their equivalents are intended to fall within the scope of the claimed subject matter. The foregoing and other objectives, features, and advantages of the invention will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings. Throughout the invention, the same reference numerals generally refer to the same parts or elements.

[0022] Figure 1 A schematic diagram of a computing device 100 provided according to an embodiment of the present invention is shown;

[0023] Figure 2 A schematic flowchart of a semiconductor optoelectronic chip anomaly detection method 200 provided according to an embodiment of the present invention is shown;

[0024] Figure 3 A schematic diagram showing a first target chip image and the division of its various key regions according to some embodiments of the present invention is provided.

[0025] Figure 4 A schematic diagram illustrating the display effect of a typical appearance anomaly of a target semiconductor optoelectronic chip under different color channels according to an embodiment of the present invention is shown.

[0026] Figure 5 This diagram illustrates the superpixel segmentation results of abnormal regions in chip images under three color models.

[0027] Figure 6 A schematic diagram illustrating the principle of fine-tuning the SAM2 visual large model according to some embodiments of the present invention is shown.

[0028] Figure 7 A schematic diagram illustrating the segmentation results of images of semiconductor optoelectronic chips of various configurations using a target vision large model according to some embodiments of the present invention is shown.

[0029] Figure 8 A schematic diagram of the structure of a target visual large model 800 according to some embodiments of the present invention is shown;

[0030] Figure 9 A schematic diagram illustrating the anomaly detection results of three key regions of a target semiconductor optoelectronic chip according to some embodiments of the present invention is shown.

[0031] Figure 10 A schematic diagram of a semiconductor optoelectronic chip anomaly detection device 1000 provided according to an embodiment of the present invention is shown. Detailed Implementation

[0032] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0033] To address the problem that existing anomaly detection schemes for semiconductor optoelectronic chips are not accurate enough and cannot detect anomalies in key areas, this invention proposes an anomaly detection method for semiconductor optoelectronic chips. Based on the RSV color model, it can effectively highlight abnormal areas on the semiconductor optoelectronic chip, thereby enabling more accurate and efficient identification and detection of abnormal areas. Furthermore, by utilizing a large target vision model, multiple key areas can be accurately extracted from the semiconductor optoelectronic chip image, thus enabling precise anomaly detection for each key area of ​​the semiconductor optoelectronic chip.

[0034] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0035] Figure 1 A schematic diagram of a computing device 100 according to an embodiment of the present invention is shown. Figure 1 As shown, in a basic configuration, computing device 100 includes at least one processing unit 102 and system memory 104. According to one aspect, depending on the configuration and type of the computing device, the processing unit 102 may be implemented as a processor. System memory 104 includes, but is not limited to, volatile memory (e.g., random access memory), non-volatile memory (e.g., read-only memory), flash memory, or any combination of such memories. According to one aspect, system memory 104 includes an operating system 105.

[0036] According to one aspect, operating system 105 is, for example, suitable for controlling the operation of computing device 100. Furthermore, examples are practiced in conjunction with graphics libraries, other operating systems, or any other applications, and are not limited to any particular application or system. Figure 1 The basic configuration is illustrated by the components within the dashed lines. According to one aspect, the computing device 100 has additional features or functions. For example, according to one aspect, the computing device 100 includes additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. This additional storage... Figure 1 The middle part is shown by removable storage device 109 and non-removable storage device 110.

[0037] As stated above, according to one aspect, program module 103 is stored in system memory 104. According to one aspect, program module 103 may include one or more applications. The present invention does not limit the type of application; for example, applications may include: email and contact applications, word processing applications, spreadsheet applications, database applications, slideshow applications, drawing or computer-aided applications, web browser applications, etc.

[0038] According to one aspect, program module 103 may include a plurality of program instructions adapted to execute the semiconductor optoelectronic chip anomaly detection method 200 of the present invention, such that computing device 100 is configured to execute the semiconductor optoelectronic chip anomaly detection method 200 of the present invention.

[0039] According to one aspect, program module 103 may include a semiconductor optoelectronic chip anomaly detection device 1000, which may be configured to perform the semiconductor optoelectronic chip anomaly detection method 200 of the present invention.

[0040] According to one aspect, examples can be practiced on circuits including discrete electronic components, packaged or integrated electronic chips containing logic gates, circuits utilizing microprocessors, or on a single chip containing electronic components or a microprocessor. For example, it can be practiced via wherein... Figure 1 Each or many of the components shown can be implemented as an example by integrating a System-on-a-Chip (SOC) on a single integrated circuit. According to one aspect, such an SOC device may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all integrated (or “burned in”) as a single integrated circuit onto a chip substrate. When operating via the SOC, the functions described herein can be operated via dedicated logic integrated on a single integrated circuit (chip) with other components of the computing device 100. Embodiments of the invention can also be implemented using other techniques capable of performing logical operations (e.g., AND, OR, and NOT), including but not limited to mechanical, optical, fluid, and quantum technologies. Additionally, embodiments of the invention can be implemented within a general-purpose computer or in any other circuit or system.

[0041] According to one aspect, computing device 100 may also have one or more input devices 112, such as a keyboard, mouse, pen, voice input device, touch input device, etc. It may also include output devices 114, such as a display, speaker, printer, etc. The foregoing devices are examples and other devices may also be used. Computing device 100 may include one or more communication connections 116 that allow communication with other computing devices 118. Examples of suitable communication connections 116 include, but are not limited to: RF transmitter, receiver and / or transceiver circuitry; Universal Serial Bus (USB), parallel and / or serial ports.

[0042] As used herein, the term computer-readable medium includes computer storage medium. Computer storage medium can include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information (e.g., computer-readable instructions, data structures, or program module 103). System memory 104, removable storage device 109, and non-removable storage device 110 are examples of computer storage media (i.e., memory storage). Computer storage media can include random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape, magnetic tape, disk storage or other magnetic storage devices, or any other article of manufacture that can be used to store information and is accessible by computing device 100. According to one aspect, any such computer storage medium can be part of computing device 100. Computer storage media does not include carrier waves or other transmitted data signals.

[0043] According to one aspect, the communication medium is implemented by computer-readable instructions, data structures, program modules 103, or other data in a modulated data signal (e.g., a carrier wave or other transmission mechanism), and includes any information transmission medium. According to one aspect, the term "modulated data signal" describes a signal having one or more sets of characteristics or altered in a manner that encodes information in the signal. By way of example and not limitation, the communication medium includes wired media such as wired networks or direct wired connections, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0044] In an embodiment of the present invention, a computing device 100 is configured to execute the semiconductor optoelectronic chip anomaly detection method 200 of the present invention. The computing device 100 includes one or more processors and one or more readable storage media storing program instructions, which, when configured to be executed by the one or more processors, cause the computing device to execute the semiconductor optoelectronic chip anomaly detection method 200 of the present invention.

[0045] Figure 2 A schematic flowchart of a semiconductor optoelectronic chip anomaly detection method 200 according to an embodiment of the present invention is shown. The semiconductor optoelectronic chip anomaly detection method 200 can be executed in a computing device (e.g., the aforementioned computing device 100).

[0046] In an embodiment of the present invention, the computing device 100 used to execute the semiconductor optoelectronic chip anomaly detection method 200 of the present invention may be a terminal or a server.

[0047] like Figure 2 As shown, the semiconductor optoelectronic chip anomaly detection method 200 includes the following steps 210 to 260.

[0048] Step 210: The computing device 100 may load a first target chip image based on a first color model of the target semiconductor optoelectronic chip (i.e., the semiconductor optoelectronic chip to be detected).

[0049] It can be understood that the color model of the first target chip image loaded by the computing device 100 is a first color model. In some embodiments, the first color model can be an RGB color model, which includes three color channels: red channel (R channel), green channel (G channel), and blue channel (B channel).

[0050] In this embodiment of the invention, the first target chip image may contain multiple key regions. The semiconductor optoelectronic chip anomaly detection method 200 provided by this embodiment of the invention can accurately detect anomalies in each key region. It should be noted that the technical solution of this invention is applicable to anomaly detection of semiconductor optoelectronic chips with various configurations, and does not limit the specific configuration of the target semiconductor optoelectronic chip. Therefore, it does not limit the specific division of each key region in the first target chip image corresponding to the target semiconductor optoelectronic chip.

[0051] Figure 3 A schematic diagram illustrating a first target chip image and the division of its key regions according to some embodiments of the present invention is shown. Figure 3 As shown, several key regions in the first target chip image may include electrode lead regions, light-emitting regions, and epitaxial regions.

[0052] Step 220: The computing device 100 can convert the first target chip image into a second target chip image based on the second color model.

[0053] In this embodiment of the invention, the second color model may include a red channel (R channel), a saturation channel (S channel), and a luminance channel (V channel). That is, the second color model may be the RSV color model.

[0054] It should be noted that by converting the first target chip image based on the first color model into a second target chip image based on the second color model (RSV color model), it is possible to focus on highlighting various abnormal regions on the target semiconductor optoelectronic chip, thereby facilitating more accurate and efficient identification and detection of abnormal regions. The red channel (R channel), saturation channel (S channel), and brightness channel (V channel) in the second color model can be obtained from the RGB color model and the HSV color model, respectively.

[0055] Figure 4 This illustration shows the display effects of a typical appearance anomaly of the target semiconductor optoelectronic chip under different color channels according to an embodiment of the present invention. Specifically, as shown... Figure 4 As shown, electrode pin marks and similar anomalies are typically located in the electrode lead area within a high-contrast environment. The saturation channel effectively highlights the corresponding abnormal area, allowing for accurate identification. However, anomalies such as lead scratches in the electrode lead area, luminous area breakdown holes and foreign objects in the luminous area, and epitaxial defects in the epitaxial area often have properties similar to the surrounding normal areas, making them prone to misidentification. The red and luminance channels can accurately detect these abnormal areas.

[0056] Step 230: The computing device 100 can perform superpixel segmentation on the second target chip image based on the second color model to obtain a target chip superpixel image containing multiple target superpixels. Subsequently, the feature vector of each target superpixel in the target chip superpixel image can be acquired. Here, the feature vector includes the average value of the red channel, the average value of the saturation channel, and the average value of the brightness channel.

[0057] It should be understood that the color model of the target chip superpixel image obtained by superpixel segmentation of the second target chip image based on the second color model is also the second color model (RSV color model).

[0058] In some embodiments, in step 230, the computing device 100 may use the SLIC algorithm to perform superpixel segmentation on the second target chip image to obtain a target chip superpixel image containing multiple target superpixels.

[0059] It should be noted that the SLIC algorithm initializes a corresponding number of cluster centers based on a preset target superpixel count. Then, based on color similarity and spatial proximity, each pixel in the second target chip image is assigned to the nearest cluster center. The superpixel boundaries are optimized by iteratively updating the cluster centers. Specifically, during the superpixel segmentation of the second target chip image using the SLIC algorithm, the computing device 100 determines the cluster center for each pixel by calculating the color difference and spatial distance between each pixel and each cluster center. This allows the second target chip image to be segmented into target chip superpixel images with multiple target superpixels. Here, the color difference can be measured using Euclidean distance in the RSV color space, and the spatial distance can be the Euclidean distance between the pixel coordinates and the cluster center coordinates. These two distances together determine which cluster center the pixel is ultimately assigned to. By continuously recalculating the position of each cluster center to reflect the average color and average coordinates of all pixels in that cluster, the superpixel boundaries can be optimized. In each iteration, the algorithm reassigns pixels and updates the cluster centers until the cluster centers no longer change. This iterative process gradually adjusts the superpixel boundaries, eventually enabling the superpixel boundaries to closely conform to the boundary of abnormal regions in the target semiconductor optoelectronic chip.

[0060] Figure 5 The diagram illustrates the superpixel segmentation results of abnormal regions in chip images under three color models. For example... Figure 5 As shown, in the RGB color model, brightness or color differences are more prominent in color channels with large variance, providing a stronger ability to describe details and abnormal areas in images. By observing the variances of the R, G, and B channels of the target semiconductor optoelectronic chip, the average value of the R channel (red channel average value) with the largest variance is ultimately selected as one of the feature vectors of the superpixel. In the HSV color model, the S channel is sensitive to saturation, and typical defects in electrodes (electrode pin marks) are located in a high-saturation environment. Therefore, the average value of the S channel (saturation channel average value) can accurately distinguish defective electrodes from normal electrodes. The average value of the V channel (luminance channel average value) can also effectively highlight abnormalities in each region. Based on this, the average values ​​of the red, saturation, and luminance channels of the target superpixel can be used as the feature vectors of the target superpixel.

[0061] Step 240: The computing device 100 can use the target visual large model to predict multiple key target regions in the superpixel image of the target chip.

[0062] Specifically, in step 240, the computing device 100 can use the target visual big model to predict the target key region corresponding to each target superpixel in the target chip superpixel image, thereby realizing the accurate segmentation of the target chip superpixel image into multiple target key regions.

[0063] It should be noted that by performing step 240, multiple target key regions are extracted from the superpixel image of the target chip, so that anomaly detection can be performed on each target key region of the target semiconductor optoelectronic chip in the subsequent process.

[0064] In some embodiments, in step 240, the computing device 100 first determines the average intensity value of each target superpixel in the target chip superpixel image. Then, based on the average intensity value of each target superpixel, the target chip superpixel image is initially divided into multiple initial key regions. Next, a predetermined number of target cue points are randomly generated for each initial key region, serving as target cue point information for that initial key region. Finally, the target chip superpixel image and the target cue point information are input into a target visual model. Using this model, multiple target key regions (i.e., segmentation masks) in the target chip superpixel image can be predicted based on the target cue point information for each initial key region.

[0065] It should be noted that the present invention is not limited to the specific type of target visual large model. For example, in some embodiments, the target visual large model may be a fine-tuned SAM2 visual large model.

[0066] Figure 6 A schematic diagram illustrating the principle of fine-tuning the SAM2 visual large model according to some embodiments of the present invention is shown.

[0067] In some embodiments, before performing step 240, that is, before predicting multiple key target regions in the superpixel image of the target chip using the target vision big model, a freeze-fine-tuning method can be used to fine-tune the pre-trained SAM2 vision big model based on multiple sample chip images with labeled information of semiconductor optoelectronic chips of various configurations. Specifically, as Figure 6 As shown, in the process of fine-tuning the pre-trained SAM2 visual large model using the freeze-fine-tuning method, the parameters of the image encoder in the pre-trained SAM2 visual large model can be frozen and the parameters of the cue encoder (the embedding layer of the cue encoder) and the mask decoder (the self-attention mechanism and output layer of the mask decoder) in the pre-trained SAM2 visual large model can be fine-tuned to obtain the target visual large model (i.e., the fine-tuned SAM2 visual large model).

[0068] It should be noted that the annotation information in the sample chip image is used to identify the true key regions of each sample in the sample chip image. Specifically, the sample chip image can be a chip image based on the second color model (RSV color model) after color model conversion.

[0069] In some embodiments, the specific process of fine-tuning a pre-trained SAM2 vision model using multiple sample chip images with labeled information from semiconductor optoelectronic chips of various configurations is as follows: First, superpixel segmentation can be performed on the sample chip images based on the second color model (RSV color model) to obtain sample chip superpixel images containing multiple sample superpixels. Then, the average intensity value of each sample superpixel can be determined, and the sample chip superpixel image can be divided into multiple initial key regions based on the average intensity value of each sample superpixel. Next, a predetermined number of sample cue points can be randomly generated for each initial key region, serving as the sample cue point information for the initial key region. Finally, the sample chip superpixel images and sample cue point information can be input into the pre-trained SAM2 vision model. Using the pre-trained SAM2 vision model, the predicted key regions of each sample in the sample chip superpixel image can be predicted based on the sample cue point information of each initial key region. Finally, by calculating the loss value between the predicted key region of each sample and the real key region of each sample, the parameters of the cue encoder (embedding layer of the cue encoder) and mask decoder (self-attention mechanism and output layer of the mask decoder) in the pre-trained SAM2 visual large model are fine-tuned based on the loss value, and the target visual large model can be obtained.

[0070] It is understandable that the target visual model obtained by the above fine-tuning is suitable for segmenting key regions of semiconductor optoelectronic chip images of various configurations, and can further improve image segmentation efficiency. Figure 7 The diagram illustrates the segmentation results of images of semiconductor optoelectronic chips with various configurations using a target vision large model according to some embodiments of the present invention.

[0071] In some embodiments, a semiconductor optoelectronic chip image dataset can be pre-constructed, containing multiple sample chip images (e.g., 1200 sample chip images) with annotation information for various configurations of semiconductor optoelectronic chips. This dataset can be divided into a training set (e.g., 950 sample chip images with annotation information for various configurations of semiconductor optoelectronic chips) and a test set (e.g., 250 sample chip images with annotation information for various configurations of semiconductor optoelectronic chips). Then, the latest SAM2 vision model is selected as the base model, and the SAM2 vision model is constructed and trained using the PyTorch deep learning framework to obtain a pre-trained SAM2 vision model. Subsequently, the freeze-fine-tuning method can be used to fine-tune the pre-trained SAM2 vision model based on the individual sample chip images with annotation information for various configurations of semiconductor optoelectronic chips in the training set.

[0072] Figure 8 A schematic diagram of the structure of a target visual large model 800 according to some embodiments of the present invention is shown.

[0073] like Figure 8 As shown, the target vision large model 800 includes a target image encoder, a memory attention module, a target cue encoder, a target mask decoder, a memory encoder, and a memory repository. The target image encoder, memory attention module, target cue encoder, memory encoder, and memory repository are sequentially coupled. Furthermore, the target cue encoder is coupled to the target mask decoder, and the memory attention module is coupled to the memory repository. It should be noted that the memory encoder can generate memory information based on the previously predicted key regions (the previously predicted key regions in the previous superpixel image of the target chip) by the target mask decoder, and store the memory information in the memory repository.

[0074] In step 240, the specific method for predicting multiple target key regions in the target chip superpixel image using the target visual large model and based on the target cue point information of each initial key region of the target is as follows: The target image encoder of the target visual large model 800 encodes the target chip superpixel image (the current target chip superpixel image) to obtain an initial image embedding. Then, the memory attention module of the target visual large model 800 adjusts the initial image embedding based on the memory information in the memory repository to obtain the target image embedding. The target cue encoder of the target visual large model 800 encodes the target cue point information of each initial key region of the target to obtain the target cue embedding. Next, the target mask decoder of the target visual model 800 predicts multiple target key regions in the target chip superpixel image based on the above target image embedding and target cue embedding. Furthermore, the memory encoder of the target visual large model 800 can generate new memory information based on each target key region currently predicted by the target mask decoder and store it in the memory repository.

[0075] According to the above embodiments, multiple key target regions can be accurately extracted from the superpixel image of the target semiconductor optoelectronic chip using a target vision large model.

[0076] Step 250: The computing device 100 can generate a corresponding OCSVM model for each target key region based on the feature vectors of each target superpixel in the target chip superpixel image.

[0077] Step 260: The computing device 100 can perform anomaly detection on any target key region based on the OCSVM model corresponding to any target key region, so as to determine the abnormal region in any target key region.

[0078] It's important to note that OCSVM (One-Class Support Vector Machine) projects input data into a high-dimensional space using a kernel function and constructs a hyperplane for classification. This transformation separates data points in the feature space, making the projected data relatively linear. It achieves the separation between positive and negative samples by maximizing the inter-class distance and maximizing the wrapping of positive samples. The defined hyperplane helps in inferring normal attributes or features; it is also considered a decision function that determines whether a new observation is inside the hyperplane (normal) or outside (outlier). Specifically, if there are no defects in the critical region, OCSVM's decision function will classify the corresponding feature values ​​as normal, belonging to the same distribution as positive samples; otherwise, these feature values ​​will be marked as outliers. Even with scarce negative samples in semiconductor optoelectronic chips, the OCSVM model can achieve high-precision anomaly detection.

[0079] Therefore, the present invention can achieve precise anomaly detection in various key areas of semiconductor optoelectronic chips, thereby improving the accuracy and precision of anomaly detection in semiconductor optoelectronic chips.

[0080] It should also be noted that, in this embodiment of the invention, superpixel segmentation and key region extraction are performed on the semiconductor optoelectronic chip image based on the RSV color model. The RSV color model can effectively highlight abnormal regions on the semiconductor optoelectronic chip, thereby enabling more accurate and efficient identification and detection of abnormal regions.

[0081] In some embodiments, since different typical defects have different characteristics, the feature vectors of each target superpixel collected in step 230 can be divided into a first feature group F1 and a second feature group F2 according to the characteristics of the typical defects, so as to meet the task requirements of subsequent anomaly detection for key regions of a single target.

[0082] It should be noted that the average value of the red channel can highlight abnormal areas with color characteristics similar to normal areas, the average value of the saturation channel can highlight electrode defects (electrode pin marks) under high saturation environments, and the average value of the luminance channel has a certain highlighting effect on abnormalities in different areas. Based on this, in some embodiments of the present invention, after the computing device 100 performs target superpixel segmentation on the second target chip image, it can combine the average value of the saturation channel and the average value of the luminance channel in the feature vector of each target superpixel in the target chip superpixel image into a first feature group F1. The first feature group F1 can be used to detect electrode abnormalities (electrode pin marks) in the electrode lead area; and the average value of the red channel and the average value of the luminance channel in the feature vector of each target superpixel in the target chip superpixel image can be combined into a second feature group F2. The second feature group F2 can be used to detect lead abnormalities (lead scratches) in the electrode lead area, light-emitting area breakdown holes and foreign objects in the light-emitting area, and epitaxial defects in the epitaxial area. Furthermore, in step 250, the computing device can generate an OCSVM model corresponding to each target key region based on the first feature group and / or the second feature group.

[0083] Figure 10 A schematic diagram of a semiconductor optoelectronic chip anomaly detection device 1000 according to an embodiment of the present invention is shown. The semiconductor optoelectronic chip anomaly detection device 1000 can be deployed in a computing device 100, and the semiconductor optoelectronic chip anomaly detection device 1000 is configured to perform the semiconductor optoelectronic chip anomaly detection method 200 of the present invention.

[0084] like Figure 10As shown, in an embodiment of the present invention, the semiconductor optoelectronic chip anomaly detection device 1000 includes a loading unit 1010, a conversion unit 1020, a superpixel segmentation unit 1030, a key region segmentation unit 1040, a generation unit 1050, and an anomaly detection unit 1060 that are sequentially connected in communication.

[0085] The loading unit 1010 is used to load the first target chip image based on the first color model of the target semiconductor optoelectronic chip.

[0086] The conversion unit 1020 is used to convert the first target chip image into a second target chip image based on a second color model, the second color model including a red channel, a saturation channel and a brightness channel.

[0087] The superpixel segmentation unit 1030 is used to perform superpixel segmentation on the second target chip image to obtain a target chip superpixel image containing multiple target superpixels, and to collect the feature vector of each target superpixel in the target chip superpixel image. The feature vector includes the average value of the red channel, the average value of the saturation channel, and the average value of the brightness channel.

[0088] The key region segmentation unit 1040 is used to predict multiple key regions of the target in the superpixel image of the target chip by utilizing the target visual large model.

[0089] The generation unit 1050 is used to generate a corresponding OCSVM model for each target key region based on the feature vector of each target superpixel in the target chip superpixel image.

[0090] The anomaly detection unit 1060 is used to perform anomaly detection on any target key region based on the OCSVM model corresponding to any target key region, so as to determine the abnormal region in any target key region.

[0091] It should be noted that the loading unit 1010, the conversion unit 1020, the superpixel segmentation unit 1030, the key region segmentation unit 1040, the generation unit 1050, and the anomaly detection unit 1060 are respectively used to execute the aforementioned steps 210 to 260. Here, the specific execution logic of each unit can be found in the description of steps 210 to 260 in the previous method 200, and will not be repeated here.

[0092] The semiconductor optoelectronic chip anomaly detection method 200 according to an embodiment of the present invention involves converting a first target chip image based on an RGB color model into a second target chip image based on an RSV color model. Then, superpixel segmentation is performed on the second target chip image to obtain a target chip superpixel image. Multiple target key regions are extracted from the target chip superpixel image using a target visual large model. Furthermore, an OCSVM model is generated for each target key region based on the feature vectors of each target superpixel in the target chip superpixel image. Thus, anomaly detection can be performed on any target key region based on the OCSVM model corresponding to any target key region. It is evident that the present invention performs superpixel segmentation and key region extraction on semiconductor optoelectronic chip images based on an RSV color model. The RSV color model effectively highlights abnormal regions on the semiconductor optoelectronic chip, facilitating more accurate and efficient identification and detection of abnormal regions. The target visual large model obtained by fine-tuning according to the present invention is suitable for segmenting key regions of semiconductor optoelectronic chip images with various configurations and can accurately and efficiently extract multiple key regions from the semiconductor optoelectronic chip image. This enables precise anomaly detection for each key region of the semiconductor optoelectronic chip, improving the accuracy and precision of anomaly detection for semiconductor optoelectronic chips.

[0093] Furthermore, embodiments of the present invention also disclose: A8, a method as described in any one of A1-A7, wherein performing superpixel segmentation on the second target chip image to obtain a target chip superpixel image containing multiple target superpixels includes: using the SLIC algorithm to perform superpixel segmentation on the second target chip image to obtain a target chip superpixel image containing multiple target superpixels. A9, a method as described in any one of A1-A8, wherein the first color model is an RGB color model, and the second color model is an RSV color model.

[0094] The various techniques described herein can be implemented in combination with hardware or software, or a combination thereof. Thus, the methods and apparatus of the present invention, or certain aspects or portions thereof, can take the form of program code (i.e., instructions) embedded in a tangible medium, such as a removable hard disk, USB flash drive, floppy disk, CD-ROM, or any other machine-readable storage medium, wherein when the program is loaded into and executed by a machine such as a computer, the machine becomes an apparatus for practicing the present invention.

[0095] When the program code is executed on a programmable computer, the mobile terminal generally includes a processor, a processor-readable storage medium (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store program code; the processor is configured to execute the semiconductor optoelectronic chip anomaly detection method of the present invention according to instructions in the program code stored in the memory.

[0096] By way of example, and not limitation, readable media include readable storage media and communication media. Readable storage media stores information such as computer-readable instructions, data structures, program modules, or other data. Communication media generally embodies computer-readable instructions, data structures, program modules, or other data in the form of modulated data signals such as carrier waves or other transmission mechanisms, and includes any information delivery medium. Any combination of the above is also included within the scope of readable media.

[0097] In the specification provided herein, the algorithms and displays are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used with the examples of this invention. The required structure for constructing such systems is apparent from the above description. Furthermore, this invention is not directed to any particular programming language. It should be understood that the contents of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0098] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0099] Similarly, it should be understood that, in order to streamline this disclosure and aid in understanding one or more of the various aspects of the invention, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof.

[0100] Those skilled in the art will understand that modules, units, or components of the devices disclosed in the examples herein can be arranged in the devices described in this embodiment, or alternatively, can be located in one or more devices different from the devices in this example. The modules in the foregoing examples can be combined into a single module or, in addition, can be divided into multiple sub-modules.

[0101] Unless otherwise specified, the use of ordinal numbers such as “first,” “second,” “third,” etc., to describe ordinary objects merely indicates different instances of similar objects and is not intended to imply that the objects being described must have a given order in time, space, ordering, or any other manner.

Claims

1. A method for detecting anomalies in a semiconductor optoelectronic chip, executed in a computing device, comprising: Loading an image of the first target chip based on the first color model of the target semiconductor optoelectronic chip; The first target chip image is converted into a second target chip image based on a second color model, the second color model including a red channel, a saturation channel and a brightness channel; The second target chip image is segmented into superpixels to obtain a target chip superpixel image containing multiple target superpixels. The feature vector of each target superpixel in the target chip superpixel image is collected. The feature vector includes the average value of the red channel, the average value of the saturation channel, and the average value of the brightness channel. Using a target vision big model, multiple target key regions in a target chip superpixel image are predicted, including: determining the average intensity value of each target superpixel in the target chip superpixel image; dividing the target chip superpixel image into multiple initial target key regions based on the average intensity value of each target superpixel; randomly generating a predetermined number of target cue points for each target initial key region as target cue point information for the target initial key region; inputting the target chip superpixel image and target cue point information into the target vision big model; and using the target vision big model to predict multiple target key regions in the target chip superpixel image based on the target cue point information of each target initial key region. The target vision big model includes a target image encoder, a memory attention module, a target cue encoder, a target mask decoder, a memory encoder, and a memory repository. The memory encoder is adapted to predict multiple target key regions in the target chip superpixel image based on the target mask decoder's previously pre-defined... Each previously detected key region generates memory information and stores it in the memory repository. Specifically, using the target visual large model, based on the target cue point information of each of the initial key regions of the target, multiple target key regions in the superpixel image of the target chip are predicted. This includes: encoding the superpixel image of the target chip using the target image encoder to obtain an initial image embedding; adjusting the initial image embedding based on the memory information in the memory repository using the memory attention module to obtain a target image embedding; encoding the target cue point information of each of the initial key regions of the target using the target cue encoder to obtain a target cue embedding; predicting multiple target key regions in the superpixel image of the target chip using the target mask decoder based on the target image embedding and the target cue embedding; and generating new memory information based on each target key region using the memory encoder and storing it in the memory repository. Based on the feature vectors of each target superpixel in the superpixel image of the target chip, a corresponding OCSVM model is generated for each target key region. Based on the OCSVM model corresponding to any target key region, anomaly detection is performed on the target key region to identify the abnormal regions within the target key region.

2. The method as described in claim 1, wherein, Before using a large target vision model to predict multiple key target regions in the superpixel image of the target chip, the process also includes: Using the freeze-fine-tuning method, a pre-trained SAM2 visual large model is fine-tuned based on multiple sample chip images with labeled information from semiconductor optoelectronic chips of various configurations. This includes freezing the parameters of the image encoder in the pre-trained SAM2 visual large model and fine-tuning the parameters of the cue encoder and mask decoder in the pre-trained SAM2 visual large model to obtain the target visual large model.

3. The method as described in claim 2, wherein, The annotation information is used to identify the real key regions of each sample in the sample chip image; based on multiple sample chip images with annotation information of semiconductor optoelectronic chips of various configurations, the pre-trained SAM2 visual large model is fine-tuned to obtain the target visual large model, including: The sample chip image is segmented into superpixels to obtain a sample chip superpixel image containing multiple sample superpixels; Determine the average intensity value of each sample superpixel, and divide the sample chip superpixel image into multiple sample initial key regions based on the average intensity value of each sample superpixel; For each initial key region of the sample, a predetermined number of sample cue points are randomly generated as sample cue point information for the initial key region of the sample. The superpixel image of the sample chip and the sample cue point information are input into the pre-trained SAM2 visual big model. Using the pre-trained SAM2 visual big model, based on the sample cue point information of each sample's initial key region, the predicted key region of each sample in the superpixel image of the sample chip is predicted. Calculate the loss value between the predicted key region of each sample and the true key region of each sample. Based on the loss value, fine-tune the parameters of the cue encoder and mask decoder in the pre-trained SAM2 visual large model to obtain the target visual large model.

4. The method according to any one of claims 1-3, wherein, Based on the feature vectors of each target superpixel in the target chip superpixel image, an OCSVM model is generated for each target key region, including: The average value of the saturation channel and the average value of the brightness channel in the feature vector of each target superpixel are combined into a first feature group, and the average value of the red channel and the average value of the brightness channel in the feature vector of each target superpixel are combined into a second feature group; For each target key region, an OCSVM model corresponding to the target key region is generated based on the first feature group and / or the second feature group.

5. The method according to any one of claims 1-3, wherein, The multiple target key regions include the electrode lead region, the light-emitting region, and the epitaxial region.

6. A semiconductor optoelectronic chip anomaly detection device, deployed in a computing device, suitable for performing the method as described in any one of claims 1-5, the device comprising: A loading unit, adapted to load a first target chip image based on a first color model of the target semiconductor optoelectronic chip; The conversion unit is adapted to convert the first target chip image into a second target chip image based on a second color model, the second color model including a red channel, a saturation channel and a brightness channel; The superpixel segmentation unit is adapted to perform superpixel segmentation on the second target chip image to obtain a target chip superpixel image containing multiple target superpixels, and to collect the feature vector of each target superpixel in the target chip superpixel image, wherein the feature vector includes the red channel average value, the saturation channel average value, and the brightness channel average value. The key region segmentation unit is suitable for using a large target vision model to predict multiple key target regions in the superpixel image of the target chip; The generation unit is adapted to generate a corresponding OCSVM model for each target key region based on the feature vector of each target superpixel in the target chip superpixel image; An anomaly detection unit is adapted to perform anomaly detection on any target key region based on the OCSVM model corresponding to any target key region, so as to determine the abnormal region in the target key region.

7. A computing device, comprising: At least one processor; and A memory storing program instructions, wherein the program instructions are configured to be processed by the at least one processor, the program instructions including instructions for processing the method as described in any one of claims 1-5.

8. A computer program product comprising a computer program / instructions, wherein, When the computer program / instructions are executed by the processor, they implement the method as described in any one of claims 1-5.

Images