Metallographic particle image segmentation method and device based on visual large model

Abstract

The invention discloses a metallographic particle image segmentation method and device based on a visual large model, and the method comprises the steps: obtaining a saliency segmentation image of a metallographic particle original image through a pre-training visual large model, and extracting a whole particle image from the saliency segmentation image; based on the metallographic particle training images of various configurations, performing fine tuning on the pre-trained visual large model by using a freezing fine tuning method to obtain a target visual large model; performing morphological corrosion processing on the whole particle image to obtain prompt point information of each coating in the whole particle image; the information of the prompt points of all the coatings is optimized, so that the positions of the prompt points of all the coatings are evenly distributed; and predicting a mask image of each coating based on the optimized prompt point information of each coating through the target visual large model to obtain a segmentation result of each coating of the particle image. According to the method, each coating in the metallographic particle original images of various configurations can be accurately segmented.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a metallographic particle image segmentation method and a metallographic particle image segmentation device based on a large visual model. Background Art

[0002] As the basic unit of energy, energy particles are composed of an inner core and an annular layer attached to the outer core. The main function of the annular layer is to prevent the leakage of fission products. The thickness of the annular layer directly affects the performance and safety of energy. Therefore, accurate measurement of the thickness of the annular layer (coating) is crucial to the effectiveness and safety of energy.

[0003] According to the existing deep learning models, there is a problem of insufficient applicability when segmenting multi-configuration metallographic particle coatings. For example, when processing metallographic particles of different configurations, the deep convolutional network CNN often over-relies on image annotation information, and the model performance will drop sharply when the detection content changes.

[0004] The visual big model is a machine learning method based on neural networks, which has shown excellent generalization and migration capabilities in the field of image segmentation. The pre-trained visual big model focuses on natural image segmentation. The significant data offset between particle metallographic images and natural images will affect the generalization performance of the big model. Therefore, the visual big model needs to be fine-tuned to eliminate the impact of data offset on accuracy.

[0005] Therefore, a metallographic particle image segmentation method based on a large visual model is needed to solve the problems existing in the above technical solutions. Summary of the invention

[0006] To this end, the present invention provides a metallographic particle image segmentation method and device based on a visual large model to solve or at least alleviate the above problems.

[0007] According to one aspect of the present invention, there is provided a method for metallographic particle image segmentation based on a visual macromodel, which is executed in a computing device, wherein the metallographic particles include one or more coatings, and the method includes: obtaining an original image of the metallographic particles to be segmented; processing the original image of the metallographic particles through a pre-trained visual macromodel to obtain a saliency segmentation image of the original image of the metallographic particles, and extracting the entire particle image from the saliency segmentation image; fine-tuning the pre-trained visual macromodel based on metallographic particle training images of various configurations using a frozen fine-tuning method to obtain a target visual macromodel, wherein the metallographic particle training image carries annotation information of each coating of the metallographic particles; performing morphological corrosion processing on the entire particle image to obtain prompt point information of each coating in the entire particle image; optimizing the prompt point information of each coating so that the prompt point positions of each coating are evenly distributed to obtain optimized prompt point information of each coating; predicting each coating mask image in the original image of the metallographic particles based on the optimized prompt point information of each coating through the target visual macromodel to obtain the segmentation result of each coating of the particle image.

[0008] Optionally, in the metallographic particle image segmentation method based on the visual big model according to the present invention, each coating mask image in the metallographic particle original image is predicted based on the optimized cue point information of each coating through the target visual big model to obtain the segmentation results of each coating of the particle image, including: inputting the metallographic particle original image, the optimized cue point information of each coating, and the cue point type into the target visual big model, the cue point type is used to indicate the segmented area or the non-segmented area; through the target visual big model, based on the optimized cue point information and the cue point type of each coating, each coating mask image in the metallographic particle original image is predicted to obtain the segmentation results of each coating of the particle image.

[0009] Optionally, in the metallographic particle image segmentation method based on the visual big model according to the present invention, each coating mask image in the metallographic particle original image is predicted based on the optimized prompt point information of each coating through the target visual big model to obtain the segmentation results of each coating of the particle image, including: determining the label value of each pixel in the metallographic particle original image based on the optimized prompt point information of each coating through the target visual big model, determining the coating corresponding to each pixel based on the label value of each pixel to obtain each coating mask image, wherein the label value of the pixel is used to represent the probability that the pixel belongs to each coating; performing hole filling operation on each coating mask image and adding them to obtain the segmentation results of each coating of the particle image.

[0010] Optionally, in the metallographic particle image segmentation method based on the visual big model according to the present invention, the target visual big model includes a target image encoder, a target hint encoder, and a target mask decoder; through the target visual big model, based on the optimized hint point information and hint point type of each coating, each coating mask image in the metallographic particle original image is predicted to obtain the segmentation results of each coating layer of the particle image, including: through the target image encoder, the metallographic particle original image is encoded to obtain image block embedding features; through the target hint encoder, the optimized hint point information and hint point type of each coating are encoded to obtain a hint embedding vector; through the target mask decoder, each coating mask image is predicted based on the image block embedding features and the hint embedding vector, and the segmentation results of each coating layer of the particle image are obtained based on each coating mask image.

[0011] Optionally, in the metallographic particle image segmentation method based on the visual large model according to the present invention, the segmentation results of each coating layer of the particle image are obtained based on each coating mask image, including: performing a hole filling operation on each coating mask image and adding them to obtain the segmentation results of each coating layer of the particle image.

[0012] Optionally, in the metallographic particle image segmentation method based on the visual large model according to the present invention, the entire particle image is subjected to morphological corrosion processing to obtain the prompt point information of each coating in the entire particle image, including: for each coating, based on the disk structure element with a radius corresponding to the coating, the entire particle image is subjected to morphological corrosion processing to obtain the corroded particle image corresponding to the coating; multiple boundary points of the corroded particles are extracted from the corroded particle image, and the multiple boundary points are down-sampled to obtain boundary points less than a predetermined number as the prompt point information of the coating.

[0013] Optionally, in the metallographic particle image segmentation method based on the visual large model according to the present invention, the cue point information of each coating is optimized so that the cue point positions of each coating are evenly distributed, and the optimized cue point information of each coating is obtained, including: for each coating, based on the cue point information of the coating, calculating the mean and standard deviation of each cue point in the coating, and determining the deviation threshold based on the standard deviation; eliminating one or more cue points in the coating whose deviation values ​​from the mean exceed the deviation threshold, to obtain the optimized cue point information of the coating.

[0014] Optionally, in the metallographic particle image segmentation method based on the visual macromodel according to the present invention, extracting the entire particle image from the significant segmentation image includes: extracting the entire particle image from the significant segmentation image based on the entire particle mask.

[0015] Optionally, in the metallographic particle image segmentation method based on the visual big model according to the present invention, the pre-trained visual big model is fine-tuned using a frozen fine-tuning method, including: freezing the parameters of the image encoder in the pre-trained visual big model using the frozen fine-tuning method, and fine-tuning the parameters of the prompt encoder and mask decoder in the pre-trained visual big model.

[0016] Optionally, in the metallographic particle image segmentation method based on the visual large model according to the present invention, the visual large model is a SAM visual large model.

[0017] According to one aspect of the present invention, there is provided a metallographic particle image segmentation device, which is deployed in a computing device, wherein the metallographic particles include one or more coatings, and the device includes: an acquisition unit, adapted to acquire an original image of the metallographic particles to be segmented; an extraction unit, adapted to process the original image of the metallographic particles through a pre-trained visual large model to obtain a significant segmentation image of the original image of the metallographic particles, and to extract the entire particle image from the significant segmentation image; a fine-tuning unit, adapted to fine-tune the pre-trained visual large model based on metallographic particle training images of various configurations using a frozen fine-tuning method to obtain a target visual large model. The metallographic particle training image carries the annotation information of each coating of the metallographic particles; the corrosion processing unit is suitable for performing morphological corrosion processing on the entire particle image to obtain the prompt point information of each coating in the entire particle image; the optimization unit is suitable for optimizing the prompt point information of each coating so that the prompt point positions of each coating are evenly distributed to obtain the optimized prompt point information of each coating; the segmentation unit is suitable for predicting the mask images of each coating in the original image of the metallographic particles based on the optimized prompt point information of each coating through the target visual large model to obtain the segmentation results of each coating in the particle image.

[0018] According to one aspect of the present invention, there is provided a computing device, comprising: at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be suitable for execution by the at least one processor, and the program instructions include instructions for executing the metallographic particle image segmentation method based on the visual large model as described above.

[0019] According to one aspect of the present invention, a computer program product is provided, comprising a computer program / instruction, wherein the computer program / instruction implements the method as described above when executed by a processor.

[0020] According to one aspect of the present invention, a readable storage medium storing program instructions is provided. When the program instructions are read and executed by a computing device, the computing device executes the metallographic particle image segmentation method based on the visual large model as described above.

[0021] According to the technical solution of the present invention, a metallographic particle image segmentation method based on a visual large model is provided. First, a significant segmentation image of the original image of the metallographic particles is obtained by pre-training the visual large model, and the entire particle image is extracted from the significant segmentation image. Subsequently, based on the metallographic particle training images of various configurations, the pre-trained visual large model is fine-tuned by using the frozen fine-tuning method to obtain the target visual large model, and the entire particle image is morphologically corroded to obtain the prompt point information of each coating, and the prompt point information of each coating is optimized. Finally, the target visual large model can be used to predict the mask images of each coating in the original image of the metallographic particles based on the optimized prompt point information of each coating to obtain the segmentation results of each coating of the particle image. Thus, accurate segmentation of each coating in the metallographic particle image can be achieved. Moreover, according to the fine-tuned target visual large model of the present invention, the influence of data offset on the accuracy of the visual large model can be eliminated, and it is suitable for coating segmentation of metallographic particle images of various different configurations, and the segmentation accuracy is high.

[0022] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to achieve the above and related purposes, some illustrative aspects are described herein in conjunction with the following description and accompanying drawings, which indicate various ways in which the principles disclosed herein can be practiced, and all aspects and their equivalents are intended to fall within the scope of the claimed subject matter. By reading the following detailed description in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present invention will become more apparent. Throughout the present invention, the same reference numerals generally refer to the same parts or elements.

[0024] Figure 1 A schematic diagram of a computing device 100 provided according to an embodiment of the present invention is shown;

[0025] Figure 2 A schematic flow chart of a metallographic particle image segmentation method 200 based on a visual large model provided according to an embodiment of the present invention is shown;

[0026] Figure 3 Schematic diagrams showing metallographic particle images of various configurations provided according to embodiments of the present invention;

[0027] Figure 4 A schematic diagram showing a method of extracting an entire particle image from a saliency segmented image according to an embodiment of the present invention is shown;

[0028] Figure 5 A schematic diagram showing the principle of fine-tuning a pre-trained visual large model using the frozen fine-tuning method; Figure 6 A schematic diagram showing a prompt information image according to an embodiment of the present invention;

[0029] Figure 7 A schematic diagram showing an optimized prompt point information image according to an embodiment of the present invention is shown;

[0030] Figure 8 A schematic diagram showing the segmentation results of each coating layer of particle images corresponding to the original images of metallographic particles of three configurations according to an embodiment of the present invention is shown;

[0031] Fig. 9 It shows a schematic diagram of the principle of segmenting the original image of metallographic particles by the target visual large model 900 according to an embodiment of the present invention;

[0032] Fig.10 A schematic diagram showing an image of corrosion particles corresponding to a silicon carbide layer according to an embodiment of the present invention is shown;

[0033] Fig.11 A schematic diagram showing a prompt point information image of a silicon carbide layer according to an embodiment of the present invention is shown;

[0034] Fig.12 A schematic diagram of a metallographic particle image segmentation device 1200 provided according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0035] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0036] In view of the problem that the existing deep learning models cannot segment metallographic particle coatings of different configurations and have insufficient applicability, the present invention proposes a metallographic particle image segmentation method based on a visual large model. The visual large model is fine-tuned based on metallographic particle images of various configurations, which can eliminate the influence of data offset on the accuracy of the visual large model and can accurately segment each coating in the original images of metallographic particles of various configurations, thereby improving applicability.

[0037] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0038] Figure 1 FIG. 1 is a schematic diagram of a computing device 100 provided according to an embodiment of the present invention. Figure 1 As shown, in a basic configuration, the computing device 100 includes at least one processing unit 102 and a system memory 104. According to one aspect, depending on the configuration and type of the computing device, the processing unit 102 can be implemented as a processor. The system memory 104 includes, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memories. According to one aspect, the system memory 104 includes an operating system 105.

[0039] According to one aspect, operating system 105 is suitable for controlling the operation of computing device 100, for example. Furthermore, examples are practiced in conjunction with graphics libraries, other operating systems, or any other application programs, and are not limited to any particular application or system. Figure 1 The basic configuration is shown in FIG. 1 by those components within the dashed lines. According to one aspect, computing device 100 has additional features or functionality. For example, according to one aspect, computing device 100 includes additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage Figure 1 1 is illustrated by a removable storage device 109 and a non-removable storage device 110.

[0040] As stated above, according to one aspect, a program module 103 is stored in the system memory 104. According to one aspect, the program module 103 may include one or more application programs, and the present invention does not limit the type of application program, for example, the application program may include: email and contact application programs, word processing application programs, spreadsheet application programs, database application programs, slide show application programs, drawing or computer-aided application programs, web browser application programs, etc.

[0041] According to one aspect, the program module 103 may include multiple program instructions suitable for executing the metallographic particle image segmentation method 200 based on visual large model of the present invention, so that the computing device 100 is configured to execute the metallographic particle image segmentation method 200 based on visual large model of the present invention.

[0042] According to one aspect, the program module 103 may include a metallographic particle image segmentation device 1200 , and the metallographic particle image segmentation device 1200 may be configured to execute the metallographic particle image segmentation method 200 based on a visual macro model of the present invention.

[0043] According to one aspect, the examples may be practiced on a circuit comprising discrete electronic components, a packaged or integrated electronic chip containing logic gates, a circuit utilizing a microprocessor, or a single chip containing electronic components or a microprocessor. Figure 1Each or many components shown in can be integrated into a system on a chip (SOC) on a single integrated circuit to practice examples. According to one aspect, such a SOC device may include one or more processing units, a graphics unit, a communication unit, a system virtualization unit, and various application functions, all of which are integrated (or "burned") into a chip substrate as a single integrated circuit. When operated via SOC, the functions described in this article can be operated via a dedicated logic integrated with other components of the computing device 100 on a single integrated circuit (chip). Embodiments of the present invention can also be practiced using other technologies capable of performing logical operations (such as AND, OR, and NOT), including but not limited to mechanical, optical, fluid, and quantum technologies. In addition, embodiments of the present invention can be practiced in a general-purpose computer or in any other circuit or system.

[0044] According to one aspect, the computing device 100 may also have one or more input devices 112, such as a keyboard, a mouse, a pen, a voice input device, a touch input device, etc. Output devices 114 may also be included, such as a display, a speaker, a printer, etc. The aforementioned devices are examples and other devices may also be used. The computing device 100 may include one or more communication connections 116 that allow communication with other computing devices 118. Examples of suitable communication connections 116 include, but are not limited to: RF transmitter, receiver and / or transceiver circuits; Universal Serial Bus (USB), parallel and / or serial ports.

[0045] The term computer-readable medium as used herein includes computer storage media. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (e.g., computer-readable instructions, data structures, or program modules 103). System memory 104, removable storage device 109, and non-removable storage device 110 are all examples of computer storage media (i.e., memory storage). Computer storage media may include random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices, or any other products that can be used to store information and can be accessed by computing device 100. According to one aspect, any such computer storage medium can be a part of computing device 100. Computer storage media do not include carrier waves or other propagated data signals.

[0046] According to one aspect, communication media is implemented by computer readable instructions, data structures, program modules 103, or other data in a modulated data signal (e.g., a carrier wave or other transport mechanism), and includes any information delivery media. According to one aspect, the term "modulated data signal" describes a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0047] In an embodiment according to the present invention, the computing device 100 is configured to execute the metallographic particle image segmentation method 200 based on the visual large model of the present invention. The computing device 100 includes one or more processors and one or more readable storage media storing program instructions. When the program instructions are configured to be executed by one or more processors, the computing device executes the metallographic particle image segmentation method 200 based on the visual large model in the embodiment of the present invention.

[0048] Figure 2 A schematic flow chart of a metallographic particle image segmentation method 200 based on a visual large model provided according to an embodiment of the present invention is shown. The metallographic particle image segmentation method 200 based on a visual large model can be executed in a computing device (eg, the aforementioned computing device 100).

[0049] In an embodiment of the present invention, the computing device 100 for executing the metallographic particle image segmentation method 200 based on a visual large model of the present invention may be a terminal or a server.

[0050] In an embodiment of the present invention, the metallographic particles include one or more coatings. The metallographic particle image is a metallographic image of the metallographic particles.

[0051] According to the metallographic particle image segmentation method 200 based on a visual large model in the embodiment of the present invention, each coating in metallographic particle images of various configurations can be accurately segmented.

[0052] Figure 3 Schematic diagrams of metallographic particle images of various configurations provided according to embodiments of the present invention are shown. Figure 3 As shown, each configuration of metallographic particles may include an inner core and one or more coatings attached to the outer core. For example, the multiple coatings of the metallographic particles of the first configuration include a porous loose layer, an inner dense pyrolytic carbon layer, a silicon carbide layer, and an outer dense pyrolytic carbon layer. The multiple coatings of the metallographic particles of the second configuration include a porous loose layer and a silicon carbide layer. The metallographic particles of the third configuration only include one layer, namely, a porous loose layer. It should be noted that Figure 3Only three configurations of metallographic particle images are shown as examples, but in the present invention, both the fine-tuning process of the pre-trained visual large model and the segmentation process of the metallographic particle image based on the fine-tuned visual large model are not limited to the above three configurations of metallographic particle images.

[0053] In the following embodiments, in order to distinguish the metallographic particle images used for training from the metallographic particle images to be segmented, the metallographic particle images to be segmented may be referred to as "metallographic particle original images" and the metallographic particle images used for training may be referred to as "metallographic particle sample images". Figure 3 Metallographic grain image shown.

[0054] In some embodiments of the present invention, before executing method 200, a visual big model can be constructed and trained in advance based on the PyTorch deep learning framework to obtain a pre-trained visual big model.

[0055] like Figure 2 As shown, the metallographic particle image segmentation method 200 based on the visual macro model includes steps 210-260.

[0056] Step 210: The computing device 100 obtains an original image of metallographic particles to be segmented.

[0057] It should be noted that the reference Figure 3 The original image of the metallographic particles to be segmented can be any one of metallographic particle images of various configurations.

[0058] Step 220: The computing device 100 may process the original image of the metallographic particles by using the pre-trained visual large model to obtain a significant segmentation image of the original image of the metallographic particles, and extract the entire particle image from the significant segmentation image.

[0059] Figure 4 FIG. 2 shows a schematic diagram of extracting the entire particle image from the saliency segmentation image according to an embodiment of the present invention. Figure 4 As shown, after obtaining the salient segmentation image of the original image of the metallographic particles by pre-training the visual macro model, the entire particle image can be extracted from the salient segmentation image based on the entire particle mask.

[0060] It should be understood that in the salient segmentation image, except for the background area, the entire particle (a single complete particle) is the largest mask of the connected area. Therefore, by controlling the size of the connected area, the entire particle mask for extracting the entire particle image can be obtained. Furthermore, the entire particle image can be extracted from the salient segmentation image based on the entire particle mask.

[0061] Step 230: The computing device 100 may use a freeze-tuning method to fine-tune the pre-trained visual macromodel based on the metallographic particle training images of various configurations to obtain a target visual macromodel (i.e., a fine-tuned visual macromodel). The metallographic particle training images carry the annotation information of each coating of the metallographic particles.

[0062] It should be noted that the metallographic particle training image carries the annotation information of each coating layer of the metallographic particles. Specifically, the metallographic particle training image includes the metallographic particle sample image used for training (fine-tuning) (see Figure 3 The metallographic particle image in the sample image) and the corresponding annotated image, wherein the annotated image contains the annotation information of each coating in the metallographic particle sample image.

[0063] In an embodiment of the present invention, the visual big model may be a SAM (Segment Anything Model) visual big model. The pre-trained visual big model is the pre-trained SAM visual big model. The target visual big model is the fine-tuned SAM visual big model.

[0064] Figure 5 The schematic diagram of the principle of fine-tuning a pre-trained visual large model using a frozen fine-tuning method is shown. The pre-trained visual large model includes an image encoder, a prompt encoder and a mask encoder. The image encoder and the prompt encoder are coupled to the mask encoder respectively.

[0065] According to an embodiment of the present invention, Figure 5 As shown, in the process of fine-tuning the pre-trained visual model using the freeze-tuning method, the parameters of the image encoder in the pre-trained visual model can be frozen, and only the parameters of the prompt encoder and the mask encoder in the pre-trained visual model can be fine-tuned. Specifically, metallographic particle training images of various configurations can be input into the pre-trained visual model, and the pre-trained visual model can be fine-tuned based on the metallographic particle training images of various configurations using the freeze-tuning method. Among them, the image encoder of the pre-trained visual model can encode the metallographic particle sample image to obtain an image embedding; the prompt encoder can encode the annotated image (annotation information of each coating) to obtain a prompt vector. The mask encoder can predict the predicted mask images of each coating in the metallographic particle sample image based on the image embedding and the prompt vector, and then, the parameters of the mask encoder and the prompt encoder (parameters of some layers) are fine-tuned by calculating the loss between each coating prediction mask image and each coating annotation information.

[0066] In some embodiments, before executing step 230, a metallographic particle image dataset of multiple configurations may be pre-generated, and the metallographic particle image dataset may include metallographic particle training images of multiple configurations, wherein the metallographic particle training images of any configuration may include multiple metallographic particle training images belonging to the configuration. For example, the metallographic particle image dataset includes 500 metallographic particle training images with four layers, 600 metallographic particle training images with two layers, and 400 metallographic particle training images with one layer. Subsequently, the metallographic particle image dataset may be divided into a training set (e.g., including 1,200 metallographic particle training images involving multiple configurations) and a test set (e.g., including 300 metallographic particle training images involving multiple configurations). Furthermore, each metallographic particle training image in the training set may be input into a pre-trained visual large model, and the pre-trained visual large model may be fine-tuned using a frozen fine-tuning method.

[0067] Step 240: The computing device 100 may perform morphological corrosion processing on the entire particle image to obtain prompt point information of each coating layer in the entire particle image.

[0068] It should be noted that the cue point information includes a plurality of cue points (coordinates of points). Figure 6 FIG. 2 shows a schematic diagram of a prompt information image according to an embodiment of the present invention. Figure 6 As shown, each coating layer has a plurality of prompt points (green points).

[0069] Step 250: The computing device 100 may optimize the prompt point information of each coating layer obtained in step 240 so that the prompt point positions of each coating layer are evenly distributed, thereby obtaining the optimized prompt point information of each coating layer.

[0070] In the embodiment of the present invention, the optimized hint point information of each coating layer is used to indicate segmentation of each coating layer.

[0071] Figure 7 FIG. 2 shows a schematic diagram of an optimized prompt point information image according to an embodiment of the present invention. Figure 7 As shown, by optimizing the cue point information of each coating layer, the cue point positions of each coating layer can be evenly distributed.

[0072] In some embodiments, the computing device 100 may remove one or more prompt points that deviate too much from the mean from each coating, and obtain the optimized prompt point information of each coating (including multiple prompt points that are evenly distributed and not removed). Specifically, for each coating, the computing device 100 may calculate the mean and standard deviation of each prompt point in the coating based on the prompt point information of the coating, and determine the deviation threshold based on the standard deviation. Furthermore, one or more prompt points in the coating whose deviation from the mean exceeds the deviation threshold may be removed to obtain the optimized prompt point information of the coating. Specifically, the deviation value of each prompt point in the coating from the mean may be calculated to determine whether the deviation value of the prompt point from the mean exceeds the deviation threshold. If it exceeds the deviation threshold, the prompt point is removed; if it does not exceed the deviation threshold, the prompt point is retained; the optimized prompt point information of the coating may be obtained based on the multiple prompt points that are finally retained in the coating.

[0073] Step 260, the computing device 100 can input the original image of the metallographic particles (which can be any one of the metallographic particle original images of multiple configurations) and the optimized prompt point information of each coating into the target visual large model, and then, the target visual large model can be used to predict each coating mask image in the original image of the metallographic particles based on the optimized prompt point information of each coating, and then, the segmentation results of each coating of the particle image can be obtained from each coating mask image.

[0074] Therefore, each coating in the metallographic particle image can be accurately segmented. Moreover, the target visual macromodel fine-tuned according to the present invention can eliminate the influence of data offset on the accuracy of the visual macromodel, and is suitable for coating segmentation of metallographic particle images of various configurations with high segmentation accuracy.

[0075] Here, it is worth noting that the target visual large model can be used to perform mask prediction for the original image of metallographic particles layer by layer. Specifically, the mask image of the corresponding coating in the original image of metallographic particles can be predicted based on the optimized prompt point information of each coating, thereby realizing layer-by-layer segmentation of each coating.

[0076] In some embodiments, the target visual macromodel can be used to determine the label value of each pixel in the original image of the metallographic particles based on the optimized cue point information of each coating layer, and the coating layer corresponding to each pixel can be determined based on the label value of each pixel to obtain each coating layer mask image. Here, the label value of the pixel is used to indicate the probability that the pixel belongs to each coating layer.

[0077] Furthermore, by performing hole filling operations on each coating mask image and adding them together, the segmentation results of each coating layer of the particle image (i.e., a comprehensive mask image containing multiple coating label values) can be obtained. Here, Figure 8A schematic diagram showing the segmentation results of each coating layer of particle images corresponding to original images of metallographic particles of three configurations according to an embodiment of the present invention is shown.

[0078] Fig. 9 The schematic diagram shows the principle of segmenting the original image of metallographic particles according to the target visual large model 900 in the embodiment of the present invention. Fig. 9 To describe the process of segmenting the original image of metallographic particles by the target visual large model 900.

[0079] like Fig. 9 As shown, in step 260, first, the original image of the metallographic particles (which can be any of a variety of configurations of the original image of the metallographic particles), the optimized prompt point information of each coating, and the prompt point type can be input into the target visual large model 900. It should be noted that the prompt point type is used to indicate the segmented area or the unsegmented area. Furthermore, the target visual large model 900 can be used to predict the mask images of each coating in the original image of the metallographic particles based on the optimized prompt point information and prompt point type of each coating, and then the segmentation results of each coating of the particle image can be obtained based on each coating mask image. Based on this, accurate segmentation of the selected particle coating can be achieved.

[0080] In some embodiments, Fig. 9 As shown, the target visual model 900 includes a target image encoder, a target hint encoder, and a target mask decoder. The target image encoder and the target hint encoder are respectively coupled to the target mask decoder.

[0081] The specific process of predicting each coating mask image in the original image of the metallographic particles by using the target visual large model 900 based on the optimized cue point information and cue point type of each coating is as follows:

[0082] First, the original image of the metallographic particles can be input into the target image encoder of the target visual large model 900, and the optimized cue point information and cue point type can be input into the target cue encoder of the target visual large model 900.

[0083] Subsequently, the original image of the metallographic particles can be encoded by the target image encoder to obtain the image block embedding features. The optimized cue point information and cue point type of each coating can be encoded by the target cue encoder to obtain the cue embedding vector. Furthermore, the target mask decoder can be used to predict each coating mask image based on the image block embedding features and the cue embedding vector. Finally, the segmentation results of each coating layer of the particle image can be obtained based on each coating mask image.

[0084] Specifically, after predicting each coating mask image, each coating segmentation result (comprehensive mask image) of the particle image can be obtained by performing a hole filling operation on each coating mask image and adding them.

[0085] In addition, in some embodiments, in step 220, the computing device 100 may also extract a kernel image from the saliency segmentation image based on the kernel mask. In step 240, the computing device 100 may perform morphological corrosion processing on the entire particle image, or perform dilation processing on the kernel image to obtain cue point information of each coating layer in the entire particle image.

[0086] In some embodiments, in step 240, considering the nearly circular shape of the metallographic particles, a disk structure element may be used to erode the entire particle image (or dilate the kernel image) to reduce the appearance of jagged edges.

[0087] Fig.10 A schematic diagram showing an image of corrosion particles corresponding to a silicon carbide layer according to an embodiment of the present invention is shown. Fig.11 A schematic diagram showing a prompt point information image of a silicon carbide layer according to an embodiment of the present invention is shown.

[0088] Specifically, the following method can be used to perform morphological corrosion processing on the entire particle image: First, for each coating, the entire particle image can be morphologically eroded based on the disk structure element with a radius corresponding to the coating to obtain the eroded particle image corresponding to the coating. For example, for the silicon carbide layer, the radius corresponding to the silicon carbide layer is 59 pixels. Therefore, the disk structure element with a radius of 59 pixels can be used to perform morphological corrosion processing on the entire particle image to obtain the eroded particle image corresponding to the silicon carbide layer. The eroded particle image corresponding to the silicon carbide layer is shown in FIG. Fig.10 shown.

[0089] Secondly, multiple boundary points of the corrosion particles can be extracted from the corrosion particle image, and then the multiple boundary points can be downsampled to reduce the number of boundary points to obtain boundary points less than a predetermined number, and the boundary points less than the predetermined number are used as the prompt point information of the coating. Fig.11 The prompt point information image of the silicon carbide layer is shown. In a specific embodiment, when performing a downsampling operation on multiple boundary points, one boundary point can be extracted from every two boundary points in a loop, and this process is continued until the number of extracted boundary points is less than the predetermined number and converges. It should be understood that the predetermined number can be reasonably set according to actual needs, for example, the predetermined number can be set to 100.

[0090] Fig.12A schematic diagram of a metallographic particle image segmentation device 1200 provided according to an embodiment of the present invention is shown. The metallographic particle image segmentation device 1200 can be deployed in a computing device 100, and the metallographic particle image segmentation device 1200 is configured to execute the metallographic particle image segmentation method 200 based on a visual large model of the present invention.

[0091] like Fig.12 As shown, in an embodiment of the present invention, the metallographic particle image segmentation device 1200 includes an acquisition unit 1210, an extraction unit 1220, a fine-tuning unit 1230, a corrosion processing unit 1240, an optimization unit 1250 and a segmentation unit 1260 which are sequentially communicated with each other.

[0092] The acquisition unit 1210 is used to acquire the original image of the metallographic particles to be segmented.

[0093] The extraction unit 1220 can process the original image of the metallographic particles through the pre-trained visual large model to obtain a significant segmentation image of the original image of the metallographic particles, and extract the entire particle image from the significant segmentation image.

[0094] The fine-tuning unit 1230 can fine-tune the pre-trained visual large model based on metallographic particle training images of various configurations using a frozen fine-tuning method to obtain a target visual large model. The metallographic particle training images carry the annotation information of each coating of the metallographic particles.

[0095] The corrosion processing unit 1240 is used to perform morphological corrosion processing on the entire particle image to obtain the prompt point information of each coating layer in the entire particle image.

[0096] The optimization unit 1250 is used to optimize the cue point information of each coating layer so that the cue point positions of each coating layer are evenly distributed, thereby obtaining the optimized cue point information of each coating layer.

[0097] The segmentation unit 1260 can predict the mask images of each coating layer in the original image of the metallographic particles through the target visual macro model based on the optimized prompt point information of each coating layer to obtain the segmentation results of each coating layer of the particle image.

[0098] It should be noted that the acquisition unit 1210, the extraction unit 1220, the fine-tuning unit 1230, the corrosion processing unit 1240, the optimization unit 1250 and the segmentation unit 1260 are respectively used to execute the aforementioned steps 210 to 260. Here, the specific execution logic of each module can be found in the description of steps 210 to 260 in the method 200 above, which will not be repeated here.

[0099] According to the metallographic particle image segmentation method 200 based on the visual macro model in the embodiment of the present invention, first, the significant segmentation image of the original image of the metallographic particles is obtained by pre-training the visual macro model, and the entire particle image is extracted from the significant segmentation image. Subsequently, based on the metallographic particle training images of various configurations, the pre-trained visual macro model is fine-tuned by using the frozen fine-tuning method to obtain the target visual macro model, and the entire particle image is morphologically eroded to obtain the prompt point information of each coating, and the prompt point information of each coating is optimized. Finally, the target visual macro model can be used to predict the mask images of each coating in the original image of the metallographic particles based on the optimized prompt point information of each coating to obtain the segmentation results of each coating of the particle image. Thus, accurate segmentation of each coating in the metallographic particle image can be achieved. Moreover, according to the fine-tuned target visual macro model of the present invention, the influence of data offset on the accuracy of the visual macro model can be eliminated, and it is suitable for coating segmentation of metallographic particle images of various different configurations, and the segmentation accuracy is high.

[0100] In addition, an embodiment of the present invention also discloses: A8. A method as described in any one of A1-A7, wherein extracting the entire particle image from the saliency segmentation image comprises: extracting the entire particle image from the saliency segmentation image based on the entire particle mask. A9. A method as described in any one of A1-A8, wherein fine-tuning the pre-trained visual large model using a frozen fine-tuning method comprises: freezing the parameters of the image encoder in the pre-trained visual large model using a frozen fine-tuning method, and fine-tuning the parameters of the prompt encoder and mask decoder in the pre-trained visual large model. A10. A method as described in any one of A1-A9, wherein the visual large model is a SAM visual large model. B14. A readable storage medium storing program instructions, which, when the program instructions are read and processed by a computing device, enables the computing device to process the method as described in any one of A1-A10.

[0101] The various techniques described herein may be implemented in combination with hardware or software, or a combination thereof. Thus, the method and apparatus of the present invention, or some aspects or parts of the method and apparatus of the present invention may be in the form of program codes (i.e., instructions) embedded in a tangible medium, such as a removable hard disk, a USB flash drive, a floppy disk, a CD-ROM, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for practicing the present invention.

[0102] In the case where the program code is executed on a programmable computer, the mobile terminal generally includes a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store the program code; the processor is configured to execute the metallographic particle image segmentation method based on the visual large model of the present invention according to the instructions in the program code stored in the memory.

[0103] By way of example and not limitation, readable media include readable storage media and communication media. Readable storage media stores information such as computer readable instructions, data structures, program modules or other data. Communication media generally embody computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and include any information transfer medium. Any combination of the above is also included within the scope of readable media.

[0104] In the description provided herein, algorithms and displays are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the examples of the present invention. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to implement the content of the present invention described herein, and the above description of specific languages ​​is for the purpose of disclosing the best mode of the present invention.

[0105] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0106] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof.

[0107] Those skilled in the art will appreciate that the modules or units or components of the devices in the examples disclosed herein may be arranged in the devices described in the embodiment, or alternatively may be located in one or more devices different from the devices in the examples. The modules in the foregoing examples may be combined into one module or may be divided into multiple submodules.

[0108] Unless otherwise specified, the use of ordinal numbers "first", "second", "third", etc. to describe common objects merely indicates that different instances of similar objects are involved and is not intended to imply that the objects so described must have a given order in time, space, order or in any other manner.

Claims

1. A method for metallographic particle image segmentation based on a visual macromodel, executed in a computing device, wherein the metallographic particles include one or more coatings, the method comprising: Obtaining the original image of the metallographic particles to be segmented; The original image of the metallographic particles is processed by a pre-trained visual large model to obtain a significant segmentation image of the original image of the metallographic particles, and the entire particle image is extracted from the significant segmentation image; Based on metallographic particle training images of various configurations, the pre-trained visual large model is fine-tuned by using a frozen fine-tuning method to obtain a target visual large model, wherein the metallographic particle training images carry annotation information of each coating of the metallographic particles; Performing morphological corrosion processing on the entire particle image to obtain prompt point information of each coating layer in the entire particle image; Optimizing the prompt point information of each coating layer so that the prompt point positions of each coating layer are evenly distributed, thereby obtaining optimized prompt point information of each coating layer; Through the target visual large model, each coating mask image in the metallographic particle original image is predicted based on the optimized prompt point information of each coating, so as to obtain the segmentation results of each coating of the particle image.

2. The method of claim 1, wherein: Through the target visual large model, based on the optimized prompt point information of each coating, each coating mask image in the metallographic particle original image is predicted to obtain the segmentation results of each coating of the particle image, including: Inputting the original image of the metallographic particles, the optimized prompt point information of each coating layer, and the prompt point type into the target visual macro model, wherein the prompt point type is used to indicate a segmented area or an unsegmented area; Through the target visual large model, based on the optimized prompt point information and prompt point type of each coating, each coating mask image in the metallographic particle original image is predicted to obtain the segmentation results of each coating layer of the particle image.

3. The method according to claim 1 or 2, wherein: Through the target visual large model, based on the optimized prompt point information of each coating, each coating mask image in the metallographic particle original image is predicted to obtain the segmentation results of each coating of the particle image, including: Determine the label value of each pixel in the original image of the metallographic particles based on the optimized prompt point information of each coating through the target visual macro model, and determine the coating corresponding to each pixel based on the label value of each pixel to obtain each coating mask image, wherein the label value of the pixel is used to indicate the probability that the pixel belongs to each coating; A hole filling operation is performed on each of the coating mask images and the images are added to obtain the segmentation results of each coating layer of the particle image.

4. The method according to any one of claims 1 to 3, wherein: The target visual large model includes a target image encoder, a target prompt encoder, and a target mask decoder; through the target visual large model, based on the optimized prompt point information and prompt point type of each coating, each coating mask image in the metallographic particle original image is predicted to obtain the segmentation results of each coating of the particle image, including: The target image encoder is used to encode the original image of the metallographic particles to obtain image block embedding features; By means of the target cue encoder, the optimized cue point information and cue point type of each coating are encoded to obtain a cue embedding vector; Through the target mask decoder, each coating mask image is predicted based on the image block embedding feature and the hint embedding vector, and each coating segmentation result of the particle image is obtained based on each coating mask image.

5. The method of claim 4, wherein: Based on each coating mask image, the segmentation results of each coating layer of the particle image are obtained, including: A hole filling operation is performed on each of the coating mask images and the images are added to obtain the segmentation results of each coating layer of the particle image.

6. The method according to any one of claims 1 to 5, wherein: Performing morphological corrosion processing on the entire particle image to obtain prompt point information of each coating layer in the entire particle image includes: For each coating, based on a disk structure element with a radius corresponding to the coating, the entire particle image is subjected to morphological corrosion processing to obtain a corrosion particle image corresponding to the coating; A plurality of boundary points of the corrosion particles are extracted from the corrosion particle image, and a downsampling operation is performed on the plurality of boundary points to obtain boundary points less than a predetermined number as prompt point information of the coating.

7. The method according to any one of claims 1 to 6, wherein: The cue point information of each coating layer is optimized so that the cue point positions of each coating layer are evenly distributed, and the optimized cue point information of each coating layer is obtained, including: For each coating layer, based on the cue point information of the coating layer, calculating the mean and standard deviation of each cue point in the coating layer, and determining a deviation threshold value based on the standard deviation; One or more prompt points in the coating whose deviation values ​​from the mean value exceed the deviation threshold are removed to obtain optimized prompt point information of the coating.

8. A metallographic particle image segmentation device, deployed in a computing device, wherein the metallographic particles include one or more coatings, and the device comprises: An acquisition unit, adapted to acquire an original image of metallographic particles to be segmented; An extraction unit, adapted to process the original metallographic particle image by using a pre-trained visual macro model to obtain a significant segmentation image of the original metallographic particle image, and extract the entire particle image from the significant segmentation image; A fine-tuning unit, adapted to fine-tune the pre-trained visual macromodel based on metallographic particle training images of various configurations by using a frozen fine-tuning method to obtain a target visual macromodel, wherein the metallographic particle training images carry annotation information of each coating of the metallographic particles; an erosion processing unit, adapted to perform morphological erosion processing on the entire particle image to obtain prompt point information of each coating layer in the entire particle image; An optimization unit, adapted to optimize the prompt point information of each coating layer so that the prompt point positions of each coating layer are evenly distributed, thereby obtaining optimized prompt point information of each coating layer; The segmentation unit is adapted to predict each coating mask image in the metallographic particle original image based on the optimized prompt point information of each coating through the target visual macro model to obtain the segmentation results of each coating of the particle image.

9. A computing device comprising: at least one processor; and A memory storing program instructions, wherein the program instructions are configured to be processed by the at least one processor, and the program instructions include instructions for processing the method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, wherein: When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images