Metallographic particle image segmentation method and device based on visual large model

Abstract

The application discloses a metallographic particle image segmentation method and device based on a visual large model, and the method comprises the following steps: obtaining a saliency segmentation image of a metallographic particle original image through a pre-trained visual large model, and extracting a whole particle image from the saliency segmentation image; based on a plurality of configuration metallographic particle training images, a fine-tuning method is used to fine-tune the pre-trained visual large model to obtain a target visual large model; performing morphological corrosion processing on the whole particle image to obtain prompt point information of each coating layer in the whole particle image; optimizing the prompt point information of each coating layer to make the prompt point positions of each coating layer uniformly distributed; and through the target visual large model, based on the optimized prompt point information of each coating layer, each coating layer mask image is predicted to obtain a particle image coating layer segmentation result. The application can accurately segment each coating layer in a plurality of configuration metallographic particle original images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and particularly relates to a metallographic particle image segmentation method based on a visual large model and a metallographic particle image segmentation device. BACKGROUND

[0002] Energy particles, as the basic unit of energy, are composed of a core and a ring layer attached to the outside of the core. The main function of the ring layer is to prevent the leakage of fission products. The thickness of the ring layer directly affects the performance and safety of the energy. Therefore, accurately measuring the thickness of the ring layer (coating) is crucial for the effectiveness and safety of the energy.

[0003] According to the existing deep learning model, when segmenting multi-configuration metallographic particle coatings, there is a problem of insufficient applicability. For example, when the deep convolutional network CNN processes different configurations of metallographic particles, it often relies too much on image annotation information, and when the detection content changes, the model performance will decrease sharply.

[0004] The visual large model is a neural network-based machine learning method that has shown excellent generalization and transferability in the field of image segmentation. The pre-trained visual large model focuses on natural image segmentation, and the significant data shift between metallographic particle images and natural images will affect the generalization performance of the large model. Therefore, the visual large model needs to be fine-tuned to eliminate the influence of data shift on precision.

[0005] Therefore, a metallographic particle image segmentation method based on a visual large model is needed to solve the problems in the above technical solutions. SUMMARY

[0006] Therefore, the present application provides a metallographic particle image segmentation method and device based on a visual large model to solve or at least alleviate the above problems.

[0007] According to one aspect of the present application, a visual large model-based metallographic particle image segmentation method is provided, which is executed in a computing device, the metallographic particle includes one or more coatings, and the method comprises: acquiring a metallographic particle original image to be segmented; processing the metallographic particle original image by a pre-trained visual large model to obtain a saliency segmentation image of the metallographic particle original image, and extracting an entire particle image from the saliency segmentation image; based on a plurality of configuration metallographic particle training images with label information of each coating of the metallographic particle, a fine-tuning method is used to fine-tune the pre-trained visual large model to obtain a target visual large model; performing morphological erosion processing on the entire particle image to obtain hint point information of each coating in the entire particle image; optimizing the hint point information of each coating to make the hint point positions of each coating uniformly distributed to obtain optimized hint point information of each coating; and predicting each coating mask image in the metallographic particle original image based on the optimized hint point information of each coating by the target visual large model to obtain a coating segmentation result of the particle image.

[0008] Optionally, in the visual large model-based metallographic particle image segmentation method according to the present application, the step of predicting each coating mask image in the metallographic particle original image based on the optimized hint point information of each coating by the target visual large model to obtain a coating segmentation result of the particle image comprises: inputting the metallographic particle original image, the optimized hint point information of each coating, and a hint point type into the target visual large model, the hint point type being used to indicate a segmentation region or a non-segmentation region; and predicting each coating mask image in the metallographic particle original image based on the optimized hint point information of each coating and the hint point type by the target visual large model to obtain a coating segmentation result of the particle image.

[0009] Optionally, in the visual large model-based metallographic particle image segmentation method according to the present application, the step of predicting each coating mask image in the metallographic particle original image based on the optimized hint point information of each coating by the target visual large model to obtain a coating segmentation result of the particle image comprises: determining a label value of each pixel in the metallographic particle original image based on the optimized hint point information of each coating by the target visual large model, and determining a corresponding coating of each pixel based on the label value of each pixel to obtain each coating mask image, wherein the label value of the pixel is used to represent a probability that the pixel belongs to each coating; and performing a hole filling operation on each coating mask image and adding them to obtain a coating segmentation result of the particle image.

[0010] Optionally, in the metallographic particle image segmentation method based on a visual large model according to the present application, the target visual large model comprises a target image encoder, a target prompt encoder, and a target mask decoder; and the target visual large model is used to predict each coating mask image in the metallographic particle original image based on the optimized prompt point information and the prompt point type of each coating, so as to obtain the coating segmentation result of the particle image, including: the target image encoder is used to encode the metallographic particle original image to obtain image block embedding features; the target prompt encoder is used to encode the optimized prompt point information and the prompt point type of each coating to obtain a prompt embedding vector; and the target mask decoder is used to predict each coating mask image based on the image block embedding features and the prompt embedding vector, and obtain the coating segmentation result of the particle image based on each coating mask image.

[0011] Optionally, in the metallographic particle image segmentation method based on a visual large model according to the present application, obtaining the coating segmentation result of the particle image based on each coating mask image includes: performing a hole filling operation on each coating mask image and adding them together to obtain the coating segmentation result of the particle image.

[0012] Optionally, in the metallographic particle image segmentation method based on a visual large model according to the present application, performing a morphological erosion process on the entire particle image to obtain the prompt point information of each coating in the entire particle image includes: for each coating, performing a morphological erosion process on the entire particle image based on a disc structure element with a radius corresponding to the coating to obtain an eroded particle image corresponding to the coating; and extracting a plurality of boundary points of the eroded particle from the eroded particle image, and performing a downsampling operation on the plurality of boundary points to obtain less than a predetermined number of boundary points as the prompt point information of the coating.

[0013] Optionally, in the metallographic particle image segmentation method based on a visual large model according to the present application, the prompt point information of each coating is optimized to make the prompt point positions of each coating uniformly distributed to obtain the optimized prompt point information of each coating, including: for each coating, calculating the mean and standard deviation of each prompt point in the coating based on the prompt point information of the coating, and determining a deviation threshold based on the standard deviation; and removing one or more prompt points in the coating whose deviation from the mean exceeds the deviation threshold to obtain the optimized prompt point information of the coating.

[0014] Optionally, in the metallographic particle image segmentation method based on a visual large model according to the present application, extracting the entire particle image from the saliency segmentation image includes: extracting the entire particle image from the saliency segmentation image based on an entire particle mask.

[0015] Optionally, in the metallographic particle image segmentation method based on a visual large model according to the present application, the pre-trained visual large model is fine-tuned by using a frozen fine-tuning method, including: freezing the parameters of an image encoder in the pre-trained visual large model and fine-tuning the parameters of a prompt encoder and a mask decoder in the pre-trained visual large model by using the frozen fine-tuning method.

[0016] Optionally, in the metallographic particle image segmentation method based on a visual large model according to the present application, the visual large model is a SAM visual large model.

[0017] According to an aspect of the present application, a metallographic particle image segmentation device is provided, which is deployed in a computing device, the metallographic particle includes one or more coatings, and the device comprises: an acquisition unit adapted to acquire a metallographic particle original image to be segmented; an extraction unit adapted to process the metallographic particle original image by a pre-trained visual large model to obtain a saliency segmentation image of the metallographic particle original image, and extract an entire particle image from the saliency segmentation image; a fine-tuning unit adapted to fine-tune the pre-trained visual large model based on a plurality of configuration metallographic particle training images with labeled information of each coating of the metallographic particle by using a frozen fine-tuning method to obtain a target visual large model; an erosion processing unit adapted to perform morphological erosion processing on the entire particle image to obtain prompt point information of each coating in the entire particle image; an optimization unit adapted to optimize the prompt point information of each coating to make the prompt point positions of each coating uniformly distributed to obtain optimized prompt point information of each coating; and a segmentation unit adapted to predict each coating mask image in the metallographic particle original image based on the optimized prompt point information of each coating by using the target visual large model to obtain a coating segmentation result of the particle image.

[0018] According to an aspect of the present application, a computing device is provided, comprising: at least one processor; a memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, and the program instructions comprise instructions for executing the metallographic particle image segmentation method based on a visual large model as described above.

[0019] According to an aspect of the present application, a computer program product is provided, comprising computer programs / instructions, wherein the computer programs / instructions are executed by a processor to implement the method as described above.

[0020] According to an aspect of the present application, a readable storage medium storing program instructions is provided, when the program instructions are read and executed by a computing device, the computing device performs the metallographic particle image segmentation method based on a visual large model as described above.

[0021] According to the technical scheme of the present application, a metallographic particle image segmentation method based on a visual large model is provided. First, a saliency segmentation image of a metallographic particle original image is obtained by pre-training a visual large model, and an entire particle image is extracted from the saliency segmentation image. Subsequently, based on a plurality of configuration metallographic particle training images, a target visual large model is obtained by fine-tuning the pre-trained visual large model using a freeze fine-tuning method, and prompt point information of each coating layer is obtained by performing morphological erosion processing on the entire particle image. The prompt point information of each coating layer is optimized, and finally, each coating layer mask image in the metallographic particle original image can be predicted based on the optimized prompt point information of each coating layer by using the target visual large model, so as to obtain the coating layer segmentation result of the particle image. Thus, the present application can achieve accurate segmentation of each coating layer in the metallographic particle image. Moreover, the target visual large model fine-tuned according to the present application can eliminate the influence of data bias on the accuracy of the visual large model, and is suitable for coating layer segmentation of a plurality of metallographic particle images with different configurations, and has high segmentation accuracy.

[0022] The above description is only a summary of the technical scheme of the present application. In order to more clearly understand the technical means of the present application, the present application can be implemented according to the content of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to achieve the above and related purposes, certain illustrative aspects will be described herein in connection with the following description and drawings, which are indicative of the various ways in which the principles disclosed herein can be practiced, and all aspects and equivalents thereof are intended to fall within the scope of the claimed subject matter. The above and other objects, features and advantages of the present application will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings in which like reference numerals indicate like elements or features. Throughout the specification, like reference numerals indicate like elements or features.

[0024] Figure 1 A schematic diagram of a computing device 100 according to an embodiment of the present application is shown;

[0025] Figure 2 A flowchart of a metallographic particle image segmentation method 200 based on a visual large model according to an embodiment of the present application is shown;

[0026] Figure 3 A schematic diagram of a plurality of configuration metallographic particle images according to an embodiment of the present application is shown;

[0027] Figure 4 A schematic diagram of extracting an entire particle image from a saliency segmentation image according to an embodiment of the present application is shown;

[0028] Figure 5 A schematic diagram showing the principle of fine-tuning a pre-trained visual large model using a frozen fine-tuning method is shown.

[0029] Figure 6 A schematic diagram of a prompt point information image according to an embodiment of the present application is shown.

[0030] Figure 7 A schematic diagram of an optimized prompt point information image according to an embodiment of the present application is shown.

[0031] Figure 8 A schematic diagram showing the respective coating segmentation results of the particle images corresponding to the metallographic particle original images of the three configurations according to an embodiment of the present application is shown.

[0032] Figure 9 A schematic diagram showing the principle of segmentation processing of the metallographic particle original image by the target visual large model 900 according to an embodiment of the present application is shown.

[0033] Figure 10 A schematic diagram of an etched particle image corresponding to a silicon carbide layer according to an embodiment of the present application is shown.

[0034] Figure 11 A schematic diagram of a prompt point information image of a silicon carbide layer according to an embodiment of the present application is shown.

[0035] Figure 12 A schematic diagram of a metallographic particle image segmentation device 1200 according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0036] Exemplary embodiments of the present application will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it is understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present application can be more thoroughly understood and so that the scope of the present application can be conveyed to those skilled in the art.

[0037] In view of the problem that existing deep learning models cannot segment metallographic particle coatings of different configurations and have insufficient applicability, the present application proposes a metallographic particle image segmentation method based on a visual large model, which fine-tunes a visual large model based on metallographic particle images of multiple configurations, can eliminate the influence of data bias on the accuracy of the visual large model, can accurately segment each coating in metallographic particle original images of multiple configurations, and improves the applicability.

[0038] Embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0039] Figure 1A schematic diagram of a computing device 100 according to an embodiment of the present invention is shown. Figure 1 As shown, in a basic configuration, computing device 100 includes at least one processing unit 102 and system memory 104. According to one aspect, depending on the configuration and type of the computing device, the processing unit 102 may be implemented as a processor. System memory 104 includes, but is not limited to, volatile memory (e.g., random access memory), non-volatile memory (e.g., read-only memory), flash memory, or any combination of such memories. According to one aspect, system memory 104 includes an operating system 105.

[0040] According to one aspect, operating system 105 is, for example, suitable for controlling the operation of computing device 100. Furthermore, examples are practiced in conjunction with graphics libraries, other operating systems, or any other applications, and are not limited to any particular application or system. Figure 1 The basic configuration is illustrated by the components within the dashed lines. According to one aspect, the computing device 100 has additional features or functions. For example, according to one aspect, the computing device 100 includes additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. This additional storage... Figure 1 The middle part is shown by removable storage device 109 and non-removable storage device 110.

[0041] As stated above, according to one aspect, program module 103 is stored in system memory 104. According to one aspect, program module 103 may include one or more applications. The present invention does not limit the type of application; for example, applications may include: email and contact applications, word processing applications, spreadsheet applications, database applications, slideshow applications, drawing or computer-aided applications, web browser applications, etc.

[0042] According to one aspect, program module 103 may include a plurality of program instructions adapted to execute the visual large model-based metallographic particle image segmentation method 200 of the present invention, such that computing device 100 is configured to execute the visual large model-based metallographic particle image segmentation method 200 of the present invention.

[0043] According to one aspect, program module 103 may include metallographic grain image segmentation device 1200, which may be configured to perform the visual large model-based metallographic grain image segmentation method 200 of the present invention.

[0044] According to an aspect, examples can be practiced with electronic circuitry integrated on a single integrated circuit chip, with separate electronic elements interconnected off the chip, with a microprocessor, or with any other physical configuration. For instance, examples can be implemented via a general purpose computer, a special purpose computer, a microprocessor, or a state machine. Examples can be implemented using any of a wide variety of Figure 1 microprocessors of one or more processors of a multi-processor core, micro-controllers, digital signal processors, dedicated circuitry, or any other circuitry or processor. Examples can be implemented using a computer-readable medium having stored thereon computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform steps necessary to implement the examples. Examples can be implemented using any of a wide variety of such programmable data processing apparatuses.

[0045] According to an aspect, the computing device 100 can also have one or more input device(s) 112 such as a keyboard, a mouse, a pen, a microphone, a touch input device, etc. One or more output device(s) 114 such as a display, speakers, a printer, etc. can also be included. The aforementioned devices are examples and others can also be used. The computing device 100 can include one or more communication connections 116 allowing communications with other computing devices 118. Examples of suitable communication connections 116 include, but are not limited to: RF transmitter, receiver, and / or transceiver circuitry; universal serial bus (USB), parallel, and / or serial ports.

[0046] The term computer readable media as used herein includes computer storage media. Computer storage media can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, or program modules 103. The system memory 104, the removable storage device 109, and the non-removable storage device 110 are all computer storage media examples (i.e., memory storage.) Computer storage media can include Random Access Memory (RAM), Read-Only Memory (ROM), Electronically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other article of manufacture which can be used to store information and which can be accessed by the computing device 100. According to an aspect, any of such computer storage media can be part of the computing device 100. Computer storage media does not include a carrier wave or other propagated data signal.

[0047] According to an aspect, communication media typically embodies computer readable instructions, data structures, program modules 103, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. According to an aspect, the term "modulated data signal" describes a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0048] In embodiments according to the present application, the computing device 100 is configured to perform the visual large model based metallographic particle image segmentation method 200 of the present application. The computing device 100 comprises one or more processors, and one or more readable storage media having stored program instructions which, when configured to be executed by the one or more processors, cause the computing device to perform the visual large model based metallographic particle image segmentation method 200 in embodiments of the present application.

[0049] Figure 2 A flowchart of a visual large model based metallographic particle image segmentation method 200 according to an embodiment of the present application is shown. The visual large model based metallographic particle image segmentation method 200 can be performed in a computing device, such as the aforementioned computing device 100.

[0050] In embodiments of the present application, the computing device 100 for performing the visual large model based metallographic particle image segmentation method 200 of the present application can be a terminal or a server.

[0051] In embodiments of the present application, the metallographic particle includes one or more coating layers. The metallographic particle image is a metallographic image of the metallographic particle.

[0052] According to the metallographic particle image segmentation method 200 based on the visual large model in the embodiments of the present application, each coating layer in the metallographic particle image of various configurations can be accurately segmented.

[0053] Figure 3 The schematic diagrams of the metallographic particle images of various configurations provided by embodiments of the present application are shown. As shown in Figure 3 each configuration of the metallographic particle can include a core and one or more coating layers attached to the outside of the core. For example, the multiple coating layers of the metallographic particle of the first configuration include a porous loose layer, an inner dense pyrolytic carbon layer, a silicon carbide layer, and an outer dense pyrolytic carbon layer. The multiple coating layers of the metallographic particle of the second configuration include a porous loose layer and a silicon carbide layer. The metallographic particle of the third configuration only includes one layer, i.e., a porous loose layer. It should be pointed out that Figure 3 only three configurations of metallographic particle images are exemplarily shown in the above embodiments, but neither the fine-tuning process of the pre-trained visual large model nor the segmentation process of the metallographic particle image based on the fine-tuned visual large model is limited to the above three configurations of metallographic particle images.

[0054] In the following embodiments, in order to distinguish between the metallographic particle images used for training and the metallographic particle images to be segmented, the metallographic particle images to be segmented can be referred to as "metallographic particle original images", and the metallographic particle images used for training can be referred to as "metallographic particle sample images". The metallographic particle original image and the metallographic particle sample image can refer to Figure 3 the metallographic particle images shown in

[0055] In some embodiments of the present application, before performing the method 200, a visual large model can be constructed and trained based on a PyTorch deep learning framework to obtain a pre-trained visual large model.

[0056] As shown in Figure 2 the metallographic particle image segmentation method 200 based on the visual large model includes steps 210-260.

[0057] Step 210, the computing device 100 acquires a metallographic particle original image to be segmented.

[0058] It should be noted that, referring to Figure 3 , the metallographic particle original image to be segmented can be any one of the metallographic particle images of various configurations.

[0059] Step 220, the computing device 100 can process the metallographic particle original image through the pre-trained visual large model to obtain a saliency segmentation image of the metallographic particle original image, and extract a whole particle image from the saliency segmentation image.

[0060] Figure 4 A schematic diagram of extracting a whole particle image from a saliency segmentation image according to an embodiment of the present application is shown. As shown in Figure 4 After obtaining the saliency segmentation image of the metallographic particle original image through the pre-trained visual large model, the whole particle image can be extracted from the saliency segmentation image based on a whole particle mask.

[0061] It should be understood that in the saliency segmentation image, in addition to the background region, the whole particle (single complete particle) is the mask with the largest connected region, therefore, by controlling the size of the connected region, the whole particle mask for extracting the whole particle image can be obtained, and then the whole particle image can be extracted from the saliency segmentation image based on the whole particle mask.

[0062] Step 230, the computing device 100 can fine-tune the pre-trained visual large model based on a plurality of configuration metallographic particle training images by using a freeze-tuning method to obtain a target visual large model (i.e., a fine-tuned visual large model). Wherein, the metallographic particle training images have annotation information of each coating of the metallographic particle.

[0063] It should be noted that the metallographic particle training images have annotation information of each coating of the metallographic particle. Specifically, the metallographic particle training images contain metallographic particle sample images for training (fine-tuning) (which can refer to Figure 3 the metallographic particle images in the metallographic particle training images) and corresponding annotation images, wherein the annotation images contain annotation information of each coating in the metallographic particle sample images.

[0064] In an embodiment of the present application, the visual large model can be a SAM (Segment Anything Model) visual large model. The pre-trained visual large model is a pre-trained SAM visual large model. The target visual large model is a fine-tuned SAM visual large model.

[0065] Figure 5 A schematic diagram of fine-tuning the pre-trained visual large model by using the freeze-tuning method is shown. The pre-trained visual large model includes an image encoder, a prompt encoder, and a mask encoder. Wherein, the image encoder and the prompt encoder are coupled with the mask encoder.

[0066] According to an embodiment of the present application, as shown in Figure 5As shown, in the process of fine-tuning the pre-trained visual large model by using the Freeze-tuning method, the parameters of the image encoder in the pre-trained visual large model can be frozen, and only the parameters of the prompt encoder and the mask encoder in the pre-trained visual large model are fine-tuned. Specifically, the metallographic particle training images of multiple configurations can be input into the pre-trained visual large model, and the pre-trained visual large model is fine-tuned based on the metallographic particle training images of multiple configurations by using the Freeze-tuning method. Among them, the image encoder of the pre-trained visual large model can encode the metallographic particle sample image to obtain an image embedding; the prompt encoder can encode the labeled image (the labeled information of each coating) to obtain a prompt vector. The mask encoder can predict each coating prediction mask image in the metallographic particle sample image based on the image embedding and the prompt vector, and then the parameters (part of the layer parameters) of the mask encoder and the prompt encoder are fine-tuned by calculating the loss between each coating prediction mask image and each coating labeled information.

[0067] In some embodiments, before step 230 is performed, a metallographic particle image dataset of multiple configurations can be generated in advance, and the metallographic particle image dataset can include metallographic particle training images of multiple configurations, wherein each configuration of the metallographic particle training images can include multiple metallographic particle training images belonging to the configuration. For example, in the metallographic particle image dataset, there are 500 metallographic particle training images with four layers, 600 metallographic particle training images with two layers, and 400 metallographic particle training images with one layer. Subsequently, the metallographic particle image dataset can be divided into a training set (for example, including 1200 metallographic particle training images involving multiple configurations) and a test set (for example, including 300 metallographic particle training images involving multiple configurations). Further, each metallographic particle training image in the training set can be input into the pre-trained visual large model, and the pre-trained visual large model is fine-tuned by using the Freeze-tuning method.

[0068] Step 240, the computing device 100 can perform morphological erosion processing on the entire particle image to obtain prompt point information of each coating in the entire particle image.

[0069] It should be noted that the prompt point information includes multiple prompt points (coordinates of the points). Figure 6 A schematic diagram of a prompt information image according to an embodiment of the present application is shown. As shown in Figure 6 As shown, each coating has multiple prompt points (green points) respectively.

[0070] At step 250, the computing device 100 can optimize the prompt point information of each coating obtained at step 240, so that the prompt point positions of each coating are uniformly distributed, thereby obtaining the optimized prompt point information of each coating.

[0071] In an embodiment of the present application, the optimized prompt point information of each coating is used to indicate the segmentation of each coating.

[0072] Figure 7 A schematic diagram of an optimized prompt point information image according to an embodiment of the present application is shown. As shown in the figure, by optimizing the prompt point information of each coating, the prompt point positions of each coating can be uniformly distributed. Figure 7

[0073] In some embodiments, the computing device 100 can remove one or more prompt points deviating from the mean value too much from each coating to obtain the optimized prompt point information of each coating (containing multiple uniformly distributed prompt points that are not removed). Specifically, for each coating, the computing device 100 can calculate the mean value and standard deviation of each prompt point in the coating based on the prompt point information of the coating, and determine a deviation threshold based on the standard deviation. Further, one or more prompt points in the coating deviating from the mean value by more than the deviation threshold can be removed to obtain the optimized prompt point information of the coating. Specifically, the deviation of each prompt point in the coating from the mean value can be calculated, and it is determined whether the deviation of the prompt point from the mean value exceeds the deviation threshold. If it exceeds the deviation threshold, the prompt point is removed; if it does not exceed the deviation threshold, the prompt point is retained; and the optimized prompt point information of the coating can be obtained based on the multiple prompt points finally retained in the coating.

[0074] At step 260, the computing device 100 can input the metallographic particle original image (which can be any of a plurality of metallographic particle original images) and the optimized prompt point information of each coating into the target visual large model, and then the target visual large model can be used to predict the coating mask image of each coating in the metallographic particle original image based on the optimized prompt point information of each coating, and further, the coating segmentation result of the particle image can be obtained from the coating mask image.

[0075] Thus, the precise segmentation of each coating in the metallographic particle image can be achieved. Moreover, according to the fine-tuned target visual large model of the present application, the influence of data bias on the accuracy of the visual large model can be eliminated, which is suitable for coating segmentation of a plurality of metallographic particle images of different configurations, and has high segmentation accuracy.

[0076] ​It is worth noting that the target visual large model can perform mask prediction on the metallographic particle original image layer by layer. Specifically, the mask image of the corresponding coating in the metallographic particle original image can be predicted based on the optimized prompt point information of each coating, so as to realize the layer-by-layer segmentation of each coating.

[0077] In some embodiments, based on the optimized prompt point information of each coating, the target visual large model can determine the label value of each pixel in the metallographic particle original image. Based on the label value of each pixel, the coating corresponding to each pixel can be determined to obtain the coating mask image. Here, the label value of the pixel is used to represent the probability of the pixel belonging to each coating.

[0078] Further, by performing a hole filling operation on each coating mask image and adding them together, the coating segmentation result of the particle image (i.e., a comprehensive mask image containing multiple coating label values) can be obtained. Here, Figure 8 The schematic diagram of the coating segmentation result of the particle image corresponding to the metallographic particle original image of the three configurations in the embodiment of the present application is shown.

[0079] Figure 9 The schematic diagram of the principle of the target visual large model 900 performing segmentation processing on the metallographic particle original image in the embodiment of the present application is shown. The process of the target visual large model 900 performing segmentation processing on the metallographic particle original image will be described below in combination with Figure 9

[0080] As shown in Figure 9 , in step 260, first, the metallographic particle original image (which can be any of the metallographic particle original images of multiple configurations), the optimized prompt point information of each coating, and the prompt point type are input into the target visual large model 900. It should be noted that the prompt point type is used to indicate a segmentation region or a non-segmentation region. Further, based on the optimized prompt point information of each coating and the prompt point type, the target visual large model 900 can predict the coating mask image in the metallographic particle original image, and further obtain the coating segmentation result of the particle image based on the coating mask image. Based on this, the accurate segmentation of the selected particle coating can be realized.

[0081] In some embodiments, as shown in Figure 9 , the target visual large model 900 includes a target image encoder, a target prompt encoder, and a target mask decoder. The target image encoder and the target prompt encoder are respectively coupled to the target mask decoder.

[0082] The specific process of predicting the coating mask image in the metallographic particle original image based on the optimized prompt point information of each coating and the prompt point type by the target visual large model 900 is as follows: ​

[0083] Firstly, the metallographic particle original image can be input into a target image encoder of the target visual large model 900, and the optimized prompt point information and the prompt point type can be input into a target prompt encoder of the target visual large model 900.

[0084] Subsequently, the metallographic particle original image can be encoded by the target image encoder to obtain image block embedding features. The optimized prompt point information and the prompt point type of each coating layer can be encoded by the target prompt encoder to obtain prompt embedding vectors. Further, each coating layer mask image can be predicted based on the image block embedding features and the prompt embedding vectors by the target mask decoder. Finally, the particle image coating layer segmentation result can be obtained based on each coating layer mask image.

[0085] Specifically, after each coating layer mask image is predicted, the particle image coating layer segmentation result (comprehensive mask image) can be obtained by performing a hole filling operation on each coating layer mask image and adding them together.

[0086] In addition, in some embodiments, in step 220, the computing device 100 can also extract the kernel image from the saliency segmentation image based on the kernel mask. In step 240, the computing device 100 can perform morphological erosion processing on the entire particle image, or perform dilation processing on the kernel image, to obtain the prompt point information of each coating layer in the entire particle image.

[0087] In some embodiments, in step 240, considering the near-circular shape of the metallographic particle, a disc structure element can be used to perform erosion processing on the entire particle image (or dilation processing on the kernel image) to reduce the occurrence of jagged edges.

[0088] Figure 10 A schematic diagram of the erosion particle image corresponding to the silicon carbide layer in the embodiment of the present application is shown. Figure 11 A schematic diagram of the prompt point information image of the silicon carbide layer in the embodiment of the present application is shown.

[0089] Specifically, the following method can be used to perform morphological erosion processing on the entire particle image: first, for each coating layer, a disc structure element with a radius corresponding to the coating layer can be used to perform morphological erosion processing on the entire particle image to obtain an erosion particle image corresponding to the coating layer. For example, for the silicon carbide layer, the radius corresponding to the silicon carbide layer is 59 pixels, so a disc structure element with a radius of 59 pixels can be used to perform morphological erosion processing on the entire particle image to obtain an erosion particle image corresponding to the silicon carbide layer. The erosion particle image corresponding to the silicon carbide layer is shown in Figure 10 .

[0090] Secondly, a plurality of boundary points of the corrosion particles can be extracted from the corrosion particle image, and then the plurality of boundary points can be down-sampled to reduce the number of boundary points to obtain boundary points less than a predetermined number, and the boundary points less than the predetermined number are taken as the prompt point information of the coating. Here, see Figure 11 The prompt point information image of the silicon carbide layer is shown. In one specific embodiment, when the plurality of boundary points are down-sampled, one boundary point can be extracted from every two boundary points in a loop, and this process continues until the number of extracted boundary points converges to less than the predetermined number. It should be understood that the predetermined number can be reasonably set according to actual needs, for example, the predetermined number can be set to 100.

[0091] Figure 12 A schematic diagram of a metallographic particle image segmentation device 1200 according to an embodiment of the present application is shown. The metallographic particle image segmentation device 1200 can be deployed in the computing device 100, and the metallographic particle image segmentation device 1200 is configured to perform the visual large model-based metallographic particle image segmentation method 200 of the present application.

[0092] As Figure 12 shown, in an embodiment of the present application, the metallographic particle image segmentation device 1200 includes an acquisition unit 1210, an extraction unit 1220, a fine-tuning unit 1230, a corrosion processing unit 1240, an optimization unit 1250, and a segmentation unit 1260, which are sequentially and communicatively connected.

[0093] The acquisition unit 1210 is configured to acquire a metallographic particle original image to be segmented.

[0094] The extraction unit 1220 can process the metallographic particle original image by using a pre-trained visual large model to obtain a saliency segmentation image of the metallographic particle original image, and extract an entire particle image from the saliency segmentation image.

[0095] The fine-tuning unit 1230 can fine-tune the pre-trained visual large model based on a plurality of metallographic particle training images of various configurations to obtain a target visual large model by using a freeze fine-tuning method, and the metallographic particle training images have label information of each coating of the metallographic particle.

[0096] The corrosion processing unit 1240 is configured to perform morphological corrosion processing on the entire particle image to obtain prompt point information of each coating in the entire particle image.

[0097] The optimization unit 1250 is configured to optimize the prompt point information of each coating to make the prompt point positions of each coating uniformly distributed to obtain optimized prompt point information of each coating.

[0098] The segmentation unit 1260 can predict each coating mask image in the metallographic particle original image based on the optimized prompt point information of each coating through the target visual large model to obtain the coating segmentation result of the particle image.

[0099] It should be pointed out that the acquisition unit 1210, the extraction unit 1220, the fine-tuning unit 1230, the corrosion processing unit 1240, the optimization unit 1250, and the segmentation unit 1260 are respectively used to perform the foregoing steps 210-260. Here, the specific execution logic of each module can be referred to the description of the steps 210-260 in the foregoing method 200, which will not be described here again.

[0100] According to the metallographic particle image segmentation method 200 based on a visual large model in the embodiment of the present application, first, the saliency segmentation image of the metallographic particle original image is obtained by pre-training the visual large model, and the entire particle image is extracted from the saliency segmentation image, then, the pre-trained visual large model is fine-tuned to obtain the target visual large model based on the metallographic particle training images of multiple configurations by using the freeze fine-tuning method, and the entire particle image is subjected to morphological corrosion processing to obtain the prompt point information of each coating, the prompt point information of each coating is optimized, finally, each coating mask image in the metallographic particle original image can be predicted based on the optimized prompt point information of each coating through the target visual large model to obtain the coating segmentation result of the particle image. Thus, the precise segmentation of each coating in the metallographic particle image can be realized. Moreover, according to the fine-tuned target visual large model of the present application, the influence of data bias on the precision of the visual large model can be eliminated, which is suitable for coating segmentation of metallographic particle images of multiple different configurations, and has high segmentation precision.

[0101] In addition, the embodiments of the present application also disclose: A8, the method of any one of A1-A7, wherein the entire particle image is extracted from the saliency segmentation image, comprising: extracting the entire particle image from the saliency segmentation image based on the entire particle mask. A9, the method of any one of A1-A8, wherein the pre-trained visual large model is fine-tuned by using the freeze fine-tuning method, comprising: freezing the parameters of the image encoder in the pre-trained visual large model by using the freeze fine-tuning method, and fine-tuning the parameters of the prompt encoder and the mask decoder in the pre-trained visual large model. A10, the method of any one of A1-A9, wherein the visual large model is a SAM visual large model. B14, a readable storage medium storing program instructions, when the program instructions are read and processed by a computing device, the computing device processes the method of any one of A1-A10.

[0102] The various techniques described herein can be implemented in connection with hardware or software or, where appropriate, with a combination of both. Thus, the methods and apparatus of the present application, or certain aspects or portions thereof, can take the form of program code (i.e., instructions) embodied in tangible media, such as removable hard disks, USB flash drives, floppy diskettes, CD-ROMs, ROM or

[0103] In the case of program code execution on programmable computers, mobile terminals generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The storage media are configured to store program code, which is executed by the processor for implementing this application's methods for metallographic particle image segmentation based on visual large models.

[0104] In the context of this document, a "storage medium" can be any physical medium that stores information including computer-readable and communication-readable media. Such media include, but are not limited to, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage and / or transmission of information, such as computer readable instructions, data structures, program code, or other data. The term "computer-readable medium" includes, but is not limited to, portable and non-portable computer storage media, optical storage media, and any other storage and / or transmission medium known currently or developed in the future.

[0105] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order to not obscure the understanding of this description.

[0106] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order to not obscure the understanding of this description.

[0107] Similarly, it is to be understood that the above description is intended to be illustrative and not restrictive. Many embodiments of the present application will be apparent to those of skill in the art upon reviewing the above description, and it is therefore contemplated that the claims should be construed in light of the full scope of the disclosure and the equivalents thereof.

[0108] Those skilled in the art will understand that the modules, or units, or components of the devices in the examples disclosed herein can be arranged in the devices as described in the examples, or alternatively can be located in one or more devices different from the devices in the examples. The modules in the foregoing examples can be combined as a module or further divided into multiple sub-modules.

[0109] Unless otherwise defined, use of the ordinal adjectives "first," "second," "third," etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects, referred to necessarily have to appear in a given order.

Claims

1. A method for metallographic particle image segmentation based on a large visual model, executed in a computing device, wherein the metallographic particles comprise one or more coatings, the method comprising: Obtain the original image of the metallographic particles to be segmented; The original image of the metallographic particles is processed by a pre-trained large visual model to obtain a salient segmentation image of the original image of the metallographic particles, and the entire particle image is extracted from the salient segmentation image. Based on training images of metallographic particles with various configurations, the pre-trained visual large model is fine-tuned using the freeze-fine-tuning method to obtain the target visual large model. The training images of metallographic particles contain annotation information of each coating of the metallographic particles. The entire particle image is subjected to morphological erosion processing to obtain cue point information for each coating in the entire particle image, including: for each coating, based on the disk structural element with the radius corresponding to the coating, the entire particle image is subjected to morphological erosion processing to obtain the eroded particle image corresponding to the coating; multiple boundary points of the eroded particles are extracted from the eroded particle image, and the multiple boundary points are downsampled to obtain less than a predetermined number of boundary points as cue point information for the coating. The prompt point information of each coating is optimized to ensure that the prompt point positions of each coating are evenly distributed, resulting in the optimized prompt point information of each coating. Using the target visual large model, the mask images of each coating in the original image of the metallographic particles are predicted based on the optimized cue point information of each coating, so as to obtain the segmentation results of each coating in the particle image.

2. The method as described in claim 1, wherein, Using the target visual large model, based on the optimized cue point information of each coating, the mask images of each coating in the original image of the metallographic particles are predicted to obtain the segmentation results of each coating in the particle image, including: The original image of the metallographic particles, the optimized cue point information of each coating, and the cue point type are input into the target visual large model. The cue point type is used to indicate whether the region is segmented or not. Using the target visual big model, based on the optimized cue point information and cue point type of each coating, the mask image of each coating in the original image of the metallographic particles is predicted to obtain the segmentation result of each coating in the particle image.

3. The method as described in claim 1, wherein, Using the target visual large model, based on the optimized cue point information of each coating, the mask images of each coating in the original image of the metallographic particles are predicted to obtain the segmentation results of each coating in the particle image, including: Using the target visual big model, based on the optimized cue point information of each coating, the label value of each pixel in the original image of the metallographic particles is determined, and the coating corresponding to each pixel is determined based on the label value of each pixel to obtain each coating mask image. The label value of the pixel is used to represent the probability that the pixel belongs to each coating. The hole-filling operation is performed on each of the coating mask images and then added together to obtain the segmentation results of each coating in the particle image.

4. The method according to any one of claims 1-3, wherein, The target visual large model includes a target image encoder, a target cue encoder, and a target mask decoder. Using this target visual large model, based on the optimized cue point information and cue point type for each coating layer, the mask images of each coating layer in the original image of the metallographic particles are predicted to obtain the segmentation results for each coating layer of the particle image, including: The original image of the metallographic particles is encoded using the target image encoder to obtain image block embedding features; The target cue encoder encodes the optimized cue point information and cue point type of each coating to obtain the cue embedding vector. The target mask decoder predicts each coating mask image based on the image block embedding features and the cue embedding vector, and obtains the coating segmentation results of each particle image based on each coating mask image.

5. The method of claim 4, wherein, The segmentation results of each coating layer in the particle image are obtained based on the mask images of each coating layer, including: The hole-filling operation is performed on each of the coating mask images and then added together to obtain the segmentation results of each coating in the particle image.

6. The method according to any one of claims 1-3, wherein, The cue point information for each coating is optimized to ensure a uniform distribution of cue point positions across all coatings, resulting in optimized cue point information for each coating, including: For each coating, based on the cue point information of the coating, the mean and standard deviation of each cue point in the coating are calculated, and a deviation threshold is determined based on the standard deviation; One or more warning points in the coating whose deviation from the mean exceeds the deviation threshold are removed to obtain the optimized warning point information of the coating.

7. The method according to any one of claims 1-3, wherein, Extracting the entire grain image from the salient segmentation image includes: The entire particle image is extracted from the saliency segmentation image based on the entire particle mask.

8. The method according to any one of claims 1-3, wherein, The pre-trained large visual model is fine-tuned using the freeze-tuning method, including: The parameters of the image encoder in the pre-trained large visual model are frozen using the freeze-fine-tuning method, and the parameters of the cue encoder and mask decoder in the pre-trained large visual model are fine-tuned.

9. The method according to any one of claims 1-3, wherein, The large visual model mentioned is the SAM large visual model.

10. A metallographic grain image segmentation apparatus, deployed in a computing device, wherein the metallographic grains comprise one or more coatings, the apparatus comprising: The acquisition unit is suitable for acquiring the original image of the metallographic particles to be segmented; The extraction unit is adapted to process the original image of the metallographic particles using a pre-trained large visual model to obtain a salient segmentation image of the original image of the metallographic particles, and to extract the entire particle image from the salient segmentation image. The fine-tuning unit is suitable for training images of metallographic particles with various configurations. The pre-trained visual large model is fine-tuned using the freeze-fine-tuning method to obtain the target visual large model. The metallographic particle training image contains annotation information of each coating of the metallographic particles. The corrosion processing unit is adapted to perform morphological corrosion processing on the entire particle image to obtain cue point information for each coating in the entire particle image, and is further adapted to: for each coating, perform morphological corrosion processing on the entire particle image based on a disk structural element with a radius corresponding to the coating to obtain a corrosion particle image corresponding to the coating; extract multiple boundary points of the corrosion particles from the corrosion particle image, and perform a downsampling operation on the multiple boundary points to obtain less than a predetermined number of boundary points as cue point information for the coating. The optimization unit is suitable for optimizing the prompt point information of each coating to make the prompt point positions of each coating evenly distributed, and to obtain the optimized prompt point information of each coating. The segmentation unit is adapted to predict the mask images of each coating in the original image of the metallographic particles based on the optimized cue point information of each coating using the target visual large model, so as to obtain the segmentation results of each coating in the particle image.

11. A computing device, comprising: At least one processor; and A memory storing program instructions, wherein the program instructions are configured to be processed by the at least one processor, the program instructions including instructions for processing the method as described in any one of claims 1-9.

12. A computer program product comprising computer program instructions, wherein, When the computer program instructions are executed by the processor, they implement the method as described in any one of claims 1-9.

13. A readable storage medium storing program instructions that, when read and processed by a computing device, cause the computing device to perform the method as described in any one of claims 1-9.

Images