Chip image labeling method and device, computing device, and storage medium

Abstract

The application discloses a chip image labeling method and device, computing equipment and a storage medium, and relates to the technical field of visual detection. The method comprises the following steps: receiving labeling data of a chip image, determining the number of categories in the chip image as a first number according to the labeling data; inputting the labeled chip image into a full convolution network model for processing, and adjusting the network parameters of the full convolution network model by calculating the category loss of each pixel in the chip image sample, so that the number of categories reaches a second number, and a trained full convolution network model is obtained; inputting a chip image to be labeled into the trained full convolution network model, and outputting a corresponding second segmentation image based on the second number of categories; and merging each region in the second segmentation image into the first number of categories according to the average pixel value and the region position of each region in the second segmentation image, so as to obtain a first segmentation image. According to the technical scheme of the application, the automatic labeling effect is realized, and the labeling precision is high.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual detection, and in particular to a chip image labeling method and device, a computing device and a storage medium. BACKGROUND

[0002] Image-based visual detection means has occupied a very important position in the industrial manufacturing process. For example, through visual detection means, the surface defects of LED chip products and the size measurement of products are detected. The non-contact and high-precision detection characteristics of the visual detection means enable it to replace manual work to complete special work.

[0003] Image segmentation is a means for classifying pixels according to certain rules and image features to extract the target region of interest. In engineering applications, due to the high precision, high efficiency and high flexibility requirements of image segmentation, traditional image segmentation methods have been unable to meet the requirements. Therefore, intelligent segmentation methods based on deep learning have emerged, which have the characteristics of high precision, high efficiency and high flexibility, replacing traditional image segmentation.

[0004] However, intelligent segmentation models based on deep learning require a large amount of labeled data to train the model. For a segmented object, dozens to hundreds of images need to be labeled, and the labeled categories on an image may include dozens of categories. At the same time, high-precision segmentation requires high-precision labeling, which makes people have to label pixel by pixel. Moreover, different types of objects need to be re-labeled, which leads to a large amount of manpower and material resources being invested in the labeling data link.

[0005] Therefore, a chip image labeling method is needed to solve the above technical problems. SUMMARY

[0006] Therefore, the present application provides a chip image labeling method and device to solve or at least alleviate the above problems.

[0007] According to an aspect of the present application, a chip image labeling method is provided, which is executed in a computing device and includes: receiving labeling data of a chip image, determining a number of classes in the chip image as a first number according to the labeling data; inputting the labeled chip image into a full convolutional network model for processing, and adjusting network parameters of the full convolutional network model by calculating class loss of each pixel in the chip image sample, so that the number of classes reaches a second number, to obtain a trained full convolutional network model, wherein the second number is greater than or equal to the first number; inputting a chip image to be labeled into the trained full convolutional network model, to output a second segmentation image corresponding to the chip image to be labeled based on the second number of classes; and merging each region in the second segmentation image into a first number of classes according to average pixel values and region positions of each region in the second segmentation image, to obtain a first segmentation image.

[0008] Optionally, in the chip image labeling method according to the present application, the method further includes: determining a class label value of each pixel in the chip image to be labeled according to the first segmentation image.

[0009] Optionally, in the chip image labeling method according to the present application, inputting the labeled chip image into the full convolutional network model for processing, and adjusting the network parameters of the full convolutional network model by calculating the class loss of each pixel in the chip image sample, includes: inputting the labeled chip image into the full convolutional network model to extract pixel features in the labeled chip image; predicting a class prediction value of each pixel according to the pixel features, and determining a class true value of each pixel according to the labeling data; calculating a cross-entropy loss of each pixel according to a difference between the class true value and the class prediction value of each pixel, to adjust the network parameters of the full convolutional network model, so that the number of classes reaches the second number.

[0010] Optionally, in the chip image labeling method according to the present application, merging each region in the second segmentation image into the first number of classes includes: merging each region in the second segmentation image into the first number of classes by using a hierarchical clustering algorithm.

[0011] Optionally, in the chip image labeling method according to the present application, merging each region in the second segmentation image into the first number of classes according to the average pixel values and the region positions of each region in the second segmentation image includes: calculating a similarity of each region according to the average pixel values and the region positions of each region in the second segmentation image; and merging the regions into the first number of classes according to the similarity of each region.

[0012] Optionally, in the chip image labeling method according to the present application, before receiving the labeling data of the chip image, the method further includes: loading the chip image.

[0013] Optionally, in the chip image labeling method according to the present application, before loading the chip image, the method comprises: judging whether the size of the chip image is greater than or equal to a threshold value; if yes, performing a reduction operation on the chip image so that the reduced chip image is smaller than the threshold value, and loading the reduced chip image.

[0014] Optionally, in the chip image labeling method according to the present application, the fully convolutional network model comprises a feature extraction module, a hybrid attention module and a multi-layer feature fusion module coupled in sequence.

[0015] According to an aspect of the present application, there is provided a chip image labeling device residing in a computing device, comprising: a receiving module adapted to receive labeling data of a chip image, and determine that the number of categories in the chip image is a first number according to the labeling data; a training module adapted to input the labeled chip image into a fully convolutional network model for processing, and adjust the network parameters of the fully convolutional network model by calculating the category loss of each pixel in the chip image sample, so that the number of categories reaches a second number, to obtain a trained fully convolutional network model, wherein the second number is greater than or equal to the first number; a processing module adapted to input a chip image to be labeled into the trained fully convolutional network model, to output a second segmentation image corresponding to the chip image to be labeled based on the second number of categories; and a clustering module adapted to merge each region in the second segmentation image into the first number of categories according to the average pixel value of each region and the position of each region in the second segmentation image, to obtain a first segmentation image.

[0016] According to an aspect of the present application, there is provided a computing device, comprising: at least one processor; a memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, and the program instructions comprise instructions for executing the chip image labeling method as described above.

[0017] According to an aspect of the present application, there is provided a readable storage medium storing program instructions, which, when read and executed by a computing device, cause the computing device to execute the chip image labeling method as described above.

[0018] According to the technical scheme of the present application, a chip image labeling method is provided. By receiving labeling data of a chip image, the number of categories in the chip image is determined to be a first number according to the labeling data, the labeled chip image is input into a full convolution network model for processing, and the network parameters of the full convolution network model are adjusted by calculating the category loss of each pixel in the chip image sample, so that the number of categories reaches a second number greater than or equal to the first number, and a trained full convolution network model is obtained. Then, the chip image to be labeled can be directly input into the trained full convolution network model for processing, so as to output a second segmentation image based on the second number of categories, and to merge each region in the second segmentation image into the first number of categories according to the average pixel value and the region position of each region in the second segmentation image, so as to obtain a first segmentation image labeled with the first number of categories. In this way, according to the technical scheme of the present application, for different types of chip images, a large number of training sets are not required to train the model on a large scale, and automatic labeling of the chip images of this type can be realized, and the classification based on the pixel level improves the labeling accuracy. For the same type of chip image, only the trained full convolution network model and the clustering algorithm are used to realize automatic labeling of the chip image. It can be seen that the present application realizes automatic labeling effect, has high labeling accuracy, and can greatly reduce the workload of manual labeling of images.

[0019] The above description is only a summary of the technical scheme of the present application. In order to more clearly understand the technical means of the present application, the content of the specification can be implemented, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described below. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to achieve the above and related purposes, certain illustrative aspects will be described herein in connection with the following description and drawings, which are indicative of the various ways in which the principles disclosed herein can be practiced and all aspects and equivalents thereof are intended to fall within the scope of the claimed subject matter. The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings in which like reference numerals refer to like components or elements throughout the drawings. The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings in which like reference numerals refer to like components or elements throughout the drawings.

[0021] Figure 1 A schematic diagram of a computing device 100 according to one embodiment of the present application is shown;

[0022] Figure 2 A flowchart of a chip image labeling method 200 according to one embodiment of the present application is shown;

[0023] Figure 3 A schematic diagram of a labeled chip image in one embodiment of the present application is shown;

[0024] Figure 4 A schematic diagram of the structure of a fully convolutional network model 400 according to an embodiment of the present invention is shown;

[0025] Figure 5 A schematic diagram of a chip image annotation apparatus 500 according to an embodiment of the present invention is shown. Detailed Implementation

[0026] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0027] Figure 1 A schematic diagram of a computing device 100 according to an embodiment of the present invention is shown. Figure 1 As shown, in a basic configuration, computing device 100 includes at least one processing unit 102 and system memory 104. According to one aspect, depending on the configuration and type of the computing device, the processing unit 102 may be implemented as a processor. System memory 104 includes, but is not limited to, volatile memory (e.g., random access memory), non-volatile memory (e.g., read-only memory), flash memory, or any combination of such memories. According to one aspect, system memory 104 includes an operating system 105.

[0028] According to one aspect, operating system 105 is, for example, suitable for controlling the operation of computing device 100. Furthermore, examples are practiced in conjunction with graphics libraries, other operating systems, or any other applications, and are not limited to any particular application or system. Figure 1 The basic configuration is illustrated by the components within the dashed lines. According to one aspect, the computing device 100 has additional features or functions. For example, according to one aspect, the computing device 100 includes additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. This additional storage... Figure 1 The middle part is shown by removable storage device 109 and non-removable storage device 110.

[0029] As stated above, according to one aspect, program module 103 is stored in the system memory 104. According to one aspect, program module 103 can include one or more applications that can be an application program interface (API), middleware, a foundation library, a STON library, a virtual machine, or various program modules providing a function according to the present application. The application program can include, for example, an e-mail and contact application, a word processor application, a spreadsheet application, a database application, a slide presentation application, a painting or computer-aided application, a web browser application, etc.

[0030] According to one aspect, the chip image labeling apparatus 500 includes a plurality of program instructions adapted to perform the chip image labeling method 200 of the present application.

[0031] According to one aspect, the example can be practiced in a circuit comprising discrete electronic elements, packaged or integrated electronic chips containing logic gates, a circuit utilizing a microprocessor, or on a single chip containing electronic elements or a microprocessor. For example, the example can be practiced via one or more components of a personal computer system, such as a system on a chip (SoC), or via one or more components of a mobile device, such as a system on a chip (SoC). Figure 1 The example can be practiced in a system on a chip (SoC) where each or many of the components illustrated in FIG. 1 can be integrated on a single integrated circuit, according to one aspect. According to one aspect, such a SoC device can include one or more processing units, graphics units, communications units, system virtualization units, and various application functionality all of which are integrated (or "burned") onto the chip substrate as a single integrated circuit (or "chip"). When implemented via an SoC, the functionality described herein can be operated via a processor or central processing unit (CPU) of a main or primary core of the SoC that is dedicated to operating system (OS) and / or virtual machine (VM) functions, a graphics processing unit (GPU) that can be dedicated to graphics functionalities, an application processing unit that can be dedicated to application software functionalities, or a combination thereof. The functionality described herein can be operated via a dedicated logic circuitry integrated on the same chip substrate as the other components of the computing device 100. The example embodiments of this application can also be practiced using other techniques for implementing an

[0032] According to one aspect, the computing device 100 can also have one or more input device(s) 112 such as a keyboard, a mouse, a pen, a microphone, a touch input device, etc. One or more output device(s) 114 such as a display, speakers, a printer, etc. can also be included. The aforementioned devices are examples and others can also be used. The computing device 100 can include one or more communication connections 116 allowing communications with other computing devices 118. Examples of suitable communication connections 116 include, but are not limited to: RF transmitter, receiver, and / or transceiver circuitry; universal serial bus (USB), parallel, and / or serial ports.

[0033] The term computer readable media as used herein includes computer storage media. Computer storage media can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, or program modules 103. The system memory 104, the removable storage device 109, and the non-removable storage device 110 are all computer storage media examples (i.e., memory storage.) Computer storage media can include Random Access Memory (RAM), Read-Only Memory (ROM), Electronically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other article of manufacture which can be used to store information and which can be accessed by the computing device 100. According to an aspect, any of such computer storage media can be part of the computing device 100. Computer storage media does not include a carrier wave or other propagated data signal.

[0034] According to an aspect, communication media typically embodies computer readable instructions, data structures, program modules 103, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. According to an aspect, the term "modulated data signal" describes a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0035] In an embodiment according to the present application, the computing device 100 can be configured to perform the chip image labeling method 200 according to the present application. The computing device 100 includes one or more processors, and one or more readable storage media storing program instructions which, when configured to be executed by the one or more processors, cause the computing device to perform the chip image labeling method 200 in the embodiment of the present application. By performing the chip image labeling method 200 of the present application, the precision and detection efficiency of defect detection on the appearance of the chip can be improved.

[0036] According to an embodiment of the present application, a chip image labeling apparatus 500 is deployed in the computing device 100, and the chip image labeling apparatus 500 is configured to perform the chip image labeling method 200 according to the present application. The chip image labeling apparatus 500 contains a plurality of program instructions for performing the chip image labeling method 200 of the present application, and these program instructions can instruct the processor to perform the chip image labeling method 200 according to the present application.

[0037] Figure 2 A flowchart of a chip image labeling method 200 according to an embodiment of the present application is shown. The chip image labeling method 200 is suitable to be executed in a computing device (e.g., the computing device 100 as described above), and specifically can be executed in a chip image labeling apparatus 500 of the computing device 100.

[0038] According to the chip image labeling method 200 of the present application, the chip can be specifically an LED chip.

[0039] As shown in Figure 2 The method 200 starts from step 210.

[0040] Before step 210 is executed, a chip image (original chip image without labeling) can be loaded first. In an embodiment, before the chip image is loaded, it can be determined whether the size of the chip image is greater than or equal to a threshold value. If the size of the chip image is greater than or equal to the threshold value, a reduction operation can be performed on the chip image so that the reduced chip image is smaller than the threshold value, and then the reduced chip image can be loaded. In this way, the calculation time of the subsequent image segmentation process can be saved.

[0041] In step 210, the labeling data of the loaded chip image is received by a user, and a labeled chip image is obtained. According to the labeling data, the number of classes in the chip image can be determined as a first number. Here, the first number can be represented as k, for example.

[0042] Figure 3 A schematic diagram of a labeled chip image according to an embodiment of the present application is shown.

[0043] As shown in Figure 3 In an embodiment, the user can select multiple regions of interest on the chip image, and can label the multiple regions of interest as a first number (k) of classes, so as to realize preliminary labeling of the chip image. For example, different colors can be used to distinguish the labeled classes, and each color corresponds to a class. It is worth noting that the present application only needs to be scribbled once for the same type of chip image.

[0044] It should be noted that the chip image labeled by step 210 can be used to train a full convolutional network model (FCN). The full convolutional network model can perform pixel-level classification on an image to realize semantic-level image segmentation.

[0045] Subsequently, in step 220, the labeled chip image is input into the full convolutional network model for processing, and the network parameters of the full convolutional network model are adjusted by calculating the class loss of each pixel in the chip image sample (pixel-by-pixel loss calculation) to make the number of classes reach a second number, and the training is ended, and a trained full convolutional network model is obtained. The second number is greater than or equal to the first number, for example, the second number can be represented as k+n, where n≥0.

[0046] In this way, the trained full convolutional network model can be used to process one or more to-be-labeled chip images of subsequent same type chips to realize automatic labeling of the to-be-labeled chip images. It should be noted that for the to-be-labeled chip images of subsequent same type chips, only steps 230-240 described below need to be performed to realize automatic labeling of the to-be-labeled chip images.

[0047] In step 230, the to-be-labeled chip image (the chip in the to-be-labeled chip image is a same type chip as the chip in the above chip image) is input into the trained full convolutional network model for processing to output a second segmentation image corresponding to the to-be-labeled chip image based on the second number of classes.

[0048] Here, it can be understood that the second segmentation image corresponding to the to-be-labeled chip image contains a plurality of regions formed by the trained full convolutional network model segmenting the to-be-labeled chip image, and the plurality of regions correspond to the second number of classes. In other words, the second segmentation image is labeled with the second number of classes.

[0049] Figure 4 A structural schematic diagram of a full convolutional network model 400 according to an embodiment of the present application is shown.

[0050] As shown in Figure 4 The full convolutional network model 400 includes a feature extraction module 410, a hybrid attention module 420, and a multi-layer feature fusion module 430 coupled in sequence. The feature extraction module 410 includes four 3x3 convolutional layers. The chip image (to-be-labeled chip image) input into the full convolutional network model 400 is first processed by the feature extraction module 410 to output the feature of each pixel in the chip image, and then the pixel features are sequentially processed by the hybrid attention module 420 and the multi-layer feature fusion module 430 to generate and output a second segmentation image corresponding to the chip image.

[0051] Finally, in step 240, each region in the second segmentation image (originally corresponding to the second number of categories) can be merged into the first number (k) of categories according to the average pixel value of each region in the second segmentation image and the position of each region (as a clustering feature), so that the first segmentation image corresponding to the first number of categories is obtained. It can be understood that the first segmentation image is labeled with the first number of categories.

[0052] Specifically, the clustering algorithm can be used to merge each region in the second segmentation image into the first number of categories. In an embodiment, the clustering algorithm can be a hierarchical clustering algorithm, that is, the hierarchical clustering algorithm can be used to merge each region in the second segmentation image into the first number of categories.

[0053] In this way, the full convolutional network model trained by the chip image labeling method of the present application and the clustering algorithm can automatically label the chip image to be labeled into the first number (k) of categories.

[0054] Further, the computing device can determine the category label value of each pixel in the chip image to be labeled according to the first segmentation image, and pixels corresponding to different category label values exhibit different colors, thereby achieving automatic labeling of the chip image to be labeled.

[0055] It should be noted that the chip image labeling method 200 of the present application does not require a large training set to train the model on a large scale. For different types of chip images, the automatic labeling of the chip images of this type can be achieved by performing the above steps 210-240. For the same type of chip image, the automatic labeling of the chip image can be achieved by using the trained full convolutional network model in combination with the clustering algorithm. It can be seen that the present application achieves automatic labeling effect and can greatly reduce the workload of manually labeling images.

[0056] In addition, it should be pointed out that for the first segmentation image generated this time, the user can also modify, withdraw, redo and other operations on the basis of the first segmentation image until the desired segmentation effect is achieved.

[0057] According to an embodiment of the present application, in step 240, the similarity of each region can be calculated according to the average pixel value of each region in the second segmentation image and the position of each region. Further, each region in the second segmentation image is merged into the first number of categories according to the similarity of each region.

[0058] According to an embodiment of the present application, in step 220, the labeled chip image is input into the full convolutional network model for processing, and the network parameters of the full convolutional network model are adjusted by calculating the category loss of each pixel in the chip image sample. Specifically, it can include the following steps.

[0059] First, the labeled chip image is input into the full convolution network model to extract the pixel features in the labeled chip image. Specifically, the labeled chip image is input into the feature extraction module of the full convolution network model, and the pixel features in the labeled chip image are extracted through the feature extraction module.

[0060] Next, the class prediction value of each pixel is predicted according to the pixel features, and the class true value of each pixel is determined according to the labeled data.

[0061] Finally, the cross-entropy loss of each pixel is calculated according to the difference between the class true value and the class prediction value of each pixel, and the network parameters of the full convolution network model are adjusted according to the cross-entropy loss of each pixel to make the number of classes reach the second number (k+n). In this way, the trained full convolution network model can be obtained.

[0062] In an implementation, the cross-entropy loss of each pixel can be calculated using the following loss function.

[0063]

[0064] where p(x i ) is the probability of the class true value occurring, and q(x i ) is the probability of the class prediction value occurring.

[0065] The optimizer can be used to iteratively adjust the network parameters in reverse, as shown in the following formula. Optionally, the optimizer can be, for example, SGD.

[0066] ω t+1 = ω t - α·g t

[0067] where ω t+1 is the parameter of the next optimized model, ω t is the parameter of the current model, α is the learning rate, and g t is the gradient of the current parameter.

[0068] Accordingly, the loss function can converge to make the number of classes reach k+n, n≥0.

[0069] After that, for the chip image to be labeled of the same type of chip, the trained full convolution network model can be input for processing, so as to output the second segmentation image corresponding to the chip image to be labeled based on the second number of classes using the trained full convolution network model, and the second segmentation image is labeled with the second number of classes.

[0070] Figure 5A schematic diagram of a chip image labeling apparatus 500 according to an embodiment of the present application is shown. The chip image labeling apparatus 500 resides in the computing device 100. The chip image labeling apparatus 500 can be configured to perform the chip image labeling method 200 of the present application.

[0071] As shown in Figure 5 The chip image labeling apparatus 500 includes, in sequence, a receiving unit 510, a training unit 520, a processing unit 530, and a clustering unit 540.

[0072] The receiving module 510 can receive labeling data of chip images, and determine the number of classes in the chip images as a first number according to the labeling data. The training module 520 can input the labeled chip images into a fully convolutional network model for processing, and adjust the network parameters of the fully convolutional network model by calculating the class loss of each pixel in the chip image sample, so as to make the number of classes reach a second number, and obtain a trained fully convolutional network model, wherein the second number is greater than or equal to the first number. The processing module 530 can input a chip image to be labeled into the trained fully convolutional network model, so as to output a second segmentation image corresponding to the chip image to be labeled based on the second number of classes. The clustering module 540 can merge each region in the second segmentation image into the first number of classes according to the average pixel value of each region in the second segmentation image and the position of each region, and obtain a first segmentation image.

[0073] It should be noted that the receiving unit 510, the training unit 520, the processing unit 530, and the clustering unit 540 are respectively used to perform the aforementioned steps 210-240. Here, the specific execution logic of the receiving unit 510, the training unit 520, the processing unit 530, and the clustering unit 540 can be referred to the description of steps 210-240 in the method 200 above, which will not be repeated here.

[0074] According to the chip image labeling method, the labeling data of the chip image is received, the number of categories in the chip image is determined as a first number according to the labeling data, the chip image after labeling is input into a full convolution network model for processing, and the network parameters of the full convolution network model are adjusted by calculating the category loss of each pixel in the chip image sample, so that the number of categories reaches a second number greater than or equal to the first number, and a trained full convolution network model is obtained. After that, the chip image to be labeled can be directly input into the trained full convolution network model for processing, so as to output a second segmentation image based on the second number of categories, and to merge each region in the second segmentation image into the first number of categories according to the average pixel value of each region in the second segmentation image and the position of each region, so as to obtain a first segmentation image labeled with the first number of categories. In this way, according to the technical scheme of the present application, for different types of chip images, a large number of training sets are not required to train the model on a large scale, and automatic labeling of the chip images of this type can be realized, and the classification based on the pixel level improves the labeling accuracy. For the same type of chip image, only the trained full convolution network model is used in combination with the clustering algorithm, and automatic labeling of the chip image can be realized. It can be seen that the present application realizes automatic labeling effect, has high labeling accuracy, and can greatly reduce the workload of manually labeling images.

[0075] A8. The method of any one of A1-A7, wherein the full convolution network model comprises a feature extraction module, a hybrid attention module, and a multi-layer feature fusion module coupled in sequence.

[0076] The various techniques described herein can be implemented in connection with hardware or software, or a combination of both. Thus, the methods and apparatus of the present application, or certain aspects or portions thereof, can take the form of program code (i.e., instructions) embodied in tangible media, such as removable hard disks, USB flash drives, floppy diskettes, CD-ROMs, or any other machine-readable storage medium wherein, when the program code is loaded into an apparatus, such as a computer, the apparatus becomes an implementation of the present application.

[0077] Where the program code is executed on a programmable computing device, the mobile terminal generally includes a processor, a memory that is readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store program code, and the processor is configured to execute instructions in the program code stored in the memory to perform the chip image labeling method and / or the grabbing control method of the present application.

[0078] By way of example, and not limitation, a readable medium can include a readable storage medium and a communication medium. A readable storage medium stores information such as computer readable instructions, data structures, program modules or other data. A communication medium conveys information such as computer readable instructions, data structures, program modules or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. Combinations of the any of the above are also included within the scope of readable media.

[0079] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order to not obscure the understanding of this description.

[0080] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order to not obscure the understanding of this description.

[0081] Similarly, it is to be understood that the above description is one example of inventive aspects of the application and that not all of the features and aspects described above are necessarily comprised in each example of the application. For the sake of brevity and clarity, certain inventive aspects of the application have not been described in detail.

[0082] Those skilled in the art will understand that the modules, or units, or components of the devices in the examples disclosed herein can be arranged in a device as described in the examples, or alternatively can be located in one or more devices different from the devices in the examples. The modules in the foregoing examples can be combined into one module or further divided into multiple sub-modules.

[0083] Those skilled in the art will understand that the modules in the devices in the examples can be adaptively changed and arranged in one or more devices different from the examples. The modules or units or components in the examples can be combined into one module or unit or component, and further can be divided into multiple sub-modules or sub-units or sub-components.

[0084] Furthermore, those skilled in the art will understand that the combination of features of different examples means within the scope of the application and forms different examples, although some examples described herein include certain features but not others of the examples described herein. Furthermore, those skilled in the art will understand that the combination of features of different examples means within the scope of the application and forms different examples, although some examples described herein include certain features but not others of the examples described herein.

[0085] Furthermore, some of the embodiments described herein are of a "method" or a "process" that can be embodied in software, firmware or both, and that is performed by a computer system that "includes a processor" that executes instructions. A computer process typically consists of instructions that implement the desired functionality.

[0086] As used herein, unless otherwise indicated, the use of the ordinal adjectives "first", "second", "third" and so on, merely to distinguish different instances of an object and does not imply a time, spatial, ranking, ordinal, or causal relationship, between the instances of that object.

[0087] While the present application has been described in connection with a limited number of embodiments, one of ordinary skill in the art will be able to devise numerous other arrangements without departing from the scope of the application as described herein. Furthermore, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes and can not have been selected to expressly convey the scope of the subject matter of the present application.

Claims

1. A chip image annotation method, executed in a computing device, comprising: Receive annotation data for a chip image, and determine the number of categories in the chip image as a first quantity based on the annotation data; The labeled chip image is input into a fully convolutional network model for processing. The network parameters of the fully convolutional network model are adjusted by calculating the class loss of each pixel in the chip image sample so that the number of classes reaches a second number, and the trained fully convolutional network model is obtained, wherein the second number is greater than or equal to the first number. The chip image to be labeled is input into the trained fully convolutional network model to output a second segmentation image corresponding to the chip image to be labeled based on a second number of categories; Based on the average pixel value and location of each region in the second segmented image, the regions in the second segmented image are merged into a first number of categories to obtain the first segmented image.

2. The method as described in claim 1, wherein, Also includes: Based on the first segmented image, determine the category label value of each pixel in the chip image to be labeled.

3. The method as described in claim 1 or 2, wherein, The labeled chip images are input into a fully convolutional network model for processing. The network parameters of the fully convolutional network model are adjusted by calculating the class loss of each pixel in the chip image samples, including: The labeled chip image is input into a fully convolutional network model to extract the pixel features of the labeled chip image; Based on the features of each pixel, the predicted category value of each pixel is predicted, and based on the labeled data, the true category value of each pixel is determined. The cross-entropy loss of each pixel is calculated based on the difference between the true value and the predicted value of the class for each pixel, so as to adjust the network parameters of the fully convolutional network model to achieve the second number of classes.

4. The method according to any one of claims 1-3, wherein, The regions in the second segmented image are merged into a first number of categories, including: Using a hierarchical clustering algorithm, the regions in the second segmented image are merged into a first number of categories.

5. The method according to any one of claims 1-4, wherein, Based on the average pixel value and location of each region in the second segmented image, the regions in the second segmented image are merged into a first number of categories, including: The similarity of each region is calculated based on the average pixel value and the location of each region in the second segmented image. Based on the similarity of the regions, the regions are merged into a first number of categories.

6. The method according to any one of claims 1-5, wherein, Before receiving the annotation data for the chip image, the process also includes: Load the chip image.

7. The method of claim 6, wherein, Before loading the chip image, the process includes: Determine whether the size of the chip image is greater than or equal to a threshold; If so, the chip image is scaled down so that the scaled-down chip image is smaller than the threshold, and the scaled-down chip image is loaded.

8. The method according to any one of claims 1-7, wherein, The fully convolutional network model includes a feature extraction module, a hybrid attention module, and a multi-layer feature fusion module that are coupled in sequence.

9. A chip image annotation apparatus, residing in a computing device, comprising: A receiving module is adapted to receive annotation data of a chip image and determine the number of categories in the chip image as a first quantity based on the annotation data; The training module is adapted to input the labeled chip image into the fully convolutional network model for processing, and to adjust the network parameters of the fully convolutional network model by calculating the class loss of each pixel in the chip image sample so that the number of classes reaches a second number, thereby obtaining the trained fully convolutional network model, wherein the second number is greater than or equal to the first number; The processing module is adapted to input the chip image to be labeled into the trained fully convolutional network model, so as to output the second segmentation image corresponding to the chip image to be labeled based on the second number of categories; The clustering module is adapted to merge the regions in the second segmented image into a first number of categories based on the average pixel value and the location of each region in the second segmented image, thereby obtaining the first segmented image.

10. A computing device, comprising: At least one processor; as well as A memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, the program instructions including instructions for performing the method as described in any one of claims 1-8.

11. A readable storage medium storing program instructions that, when read and executed by a computing device, cause the computing device to perform the method as described in any one of claims 1-8.

Images