Metal carbide detection grading method and device, electronic equipment and storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 河钢数字技术股份有限公司
- Filing Date
- 2023-09-20
- Publication Date
- 2026-06-16
Smart Images
Figure CN117218092B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of metallographic data recognition technology, specifically to a method, apparatus, electronic device, and storage medium for the detection and grading of metal carbides. Background Technology
[0002] During the steel production process, metal carbides of different grades will exhibit various shapes and structural variations. Data identification of metal carbides is of great significance for ensuring metallurgical quality. For example, identifying the inhomogeneity of metal carbides is an important indicator for measuring metallurgical quality.
[0003] In related technologies, the detection method for metal carbides is usually manual metallographic examination. This involves technicians observing the metallurgical sample under a microscope and then determining the classification result of the metal carbides by referring to metallographic atlases, relevant testing standards, and their own professional knowledge. However, this method relies on human eyesight and experience, resulting in problems such as large detection and classification errors and low efficiency. It also requires a high level of expertise from technicians and can easily cause visual fatigue. Summary of the Invention
[0004] In view of this, embodiments of this application provide a method, apparatus, electronic device and storage medium for the detection and grading of metal carbides, in order to solve the technical problems of large detection and grading errors and low efficiency that may exist in manual metallographic detection in related technologies.
[0005] In a first aspect, embodiments of this application provide a method for detecting and classifying metal carbides, comprising: acquiring multiple images to be detected corresponding to a sample to be detected; wherein the multiple images to be detected are multiple metallographic images taken at different locations of the sample to be detected; inputting each image to be detected into a preliminary screening model to obtain a multi-channel feature map of each image to be detected, and determining whether each image to be detected is a metal carbide image based on the multi-channel feature map of each image to be detected, wherein the preliminary screening model inputs a metallographic image and outputs a multi-channel feature map;
[0006] Each metal carbide image from a set of images to be detected is input into a target detection model to obtain a target image. Each target image is then input into a grading model to obtain the corresponding carbide grade and confidence level. The target images are marked with bounding boxes used to locate carbides within the images. The target detection model inputs carbide images and outputs target images marked with bounding boxes. The grading model inputs target images marked with bounding boxes and outputs the carbide grade and confidence level. Based on the carbide grade and confidence level, a preset number of target images are selected and input into an ensemble model to obtain the final carbide grade of the sample to be detected. The ensemble model inputs multiple target images marked with bounding boxes and outputs the carbide grade.
[0007] In one possible implementation of the first aspect, the initial screening model includes multiple feature extraction networks; each of the multiple images to be detected is input into the initial screening model to obtain a multi-channel feature map of each image to be detected, and based on the multi-channel feature map of each image to be detected, it is determined whether each image to be detected is a metal carbide image, including:
[0008] Each of the multiple images to be detected is input into multiple feature extraction networks to obtain multiple multi-channel feature maps for each image to be detected. Each multi-channel feature map of each image to be detected corresponds one-to-one with one of the multiple feature extraction networks. For each multi-channel feature map of an image to be detected, the features at each position in each multi-channel feature map are determined. The features at each position in each multi-channel feature map correspond to each pixel in the corresponding multi-channel feature map. Based on the features at each position in each multi-channel feature map of an image to be detected and a preset feature library, an outlier map corresponding to each image to be detected is determined. Based on the outlier map corresponding to each image to be detected, it is determined whether each image to be detected is a metal carbide image.
[0009] In one possible implementation of the first aspect, for each multi-channel feature map of an image to be detected, determining the features at each position in each multi-channel feature map includes: for each multi-channel feature map of an image to be detected, determining the position region corresponding to each position in each multi-channel feature map; for each multi-channel feature map, calculating the value of an aggregation function based on the multi-channel features corresponding to the position region at each position, and using the value of the aggregation function as the feature at the corresponding position in the corresponding multi-channel feature map.
[0010] In one possible implementation of the first aspect, the preset feature library includes multiple sub-feature libraries, each sub-feature library corresponding one-to-one with each feature extraction network, and each sub-feature library corresponding to each multi-channel feature of each image to be detected. Figure 1 One-to-one correspondence; each sub-feature library includes standard features at each position in the multi-channel feature maps corresponding to multiple positive sample images; positive sample images are sample images that do not include metal carbides; based on the features at each position in each multi-channel feature map of each image to be detected and the preset feature library, the outlier map corresponding to each image to be detected is determined, including:
[0011] For each image to be detected, the distance between the feature at each position in each multi-channel feature map and each standard feature in the corresponding sub-feature library of the multi-channel feature map is determined, resulting in multiple feature distances. The feature distance with the smallest value among these multiple feature distances is taken as the outlier at each position in the corresponding multi-channel feature map. Based on the outliers at each position in the obtained multi-channel feature maps, a sub-outlier map corresponding to each image to be detected is constructed. Based on each sub-outlier map corresponding to each image to be detected, an outlier map corresponding to each image to be detected is determined. Each sub-outlier map corresponding to each image to be detected and each multi-channel feature map of each image to be detected are compared with each feature in the multi-channel feature library of the image to be detected. Figure 1 One-to-one correspondence.
[0012] In one possible implementation of the first aspect, determining whether each image to be detected is a metal carbide image based on the outlier map corresponding to each image to be detected includes: determining whether there are outliers greater than a preset threshold in the outlier map corresponding to each image to be detected; if there are outliers greater than the preset threshold in the outlier map, then the image to be detected corresponding to the outlier map with outliers greater than the preset threshold is taken as a metal carbide image.
[0013] In one possible implementation of the first aspect, after inputting each metal carbide image from a plurality of determined images to be detected into a target detection model to obtain a target image, the method further includes: inputting the target image into a thinning model to obtain a new target image with segmentation information; the segmentation information is used to indicate the carbides and background in the new target image; the thinning model inputs the target image marked with a detection box and outputs the target image with segmentation information; correspondingly, inputting each target image into a grading model to obtain the carbide grade and confidence level corresponding to each target image includes: inputting each new target image into the grading model to obtain the carbide grade and confidence level corresponding to each new target image.
[0014] In one possible implementation of the first aspect, a preset number of target images are selected and input into the ensemble model according to the carbide grade and confidence level to obtain the final carbide grade of the sample to be tested, including: sorting the target images according to the carbide grade from high to low and the confidence level from large to small to obtain a sorting result; determining the first preset number of target images according to the sorting result; and inputting the first preset number of target images into the ensemble model to obtain the final carbide grade of the sample to be tested.
[0015] Secondly, embodiments of this application provide a metal carbide detection and grading device, comprising:
[0016] The image acquisition module is used to acquire multiple images of the sample to be tested; wherein, the multiple images of the sample to be tested are multiple metallographic images taken at different positions of the sample to be tested.
[0017] The image determination module is used to input each of the multiple images to be detected into the initial screening model to obtain a multi-channel feature map of each image to be detected, and to determine whether each image to be detected is a metal carbide image based on the multi-channel feature map of each image to be detected. The initial screening model inputs a metallographic image and outputs a multi-channel feature map.
[0018] The grading prediction module is used to input each metal carbide image from multiple determined images to be detected into the target detection model to obtain the target image, and to input each target image into the grading model to obtain the carbide grade and confidence level corresponding to each target image. The target image is marked with a detection box, which is used to locate the carbide in the image. The target detection model inputs the carbide image and outputs the target image marked with the detection box, and the grading model inputs the target image marked with the detection box and outputs the carbide grade and confidence level.
[0019] The grade determination module is used to select a preset number of target images and input them into the ensemble model based on the carbide grade and confidence level to obtain the final carbide grade of the sample to be tested. The ensemble model inputs multiple target images marked with detection boxes and outputs the carbide grade.
[0020] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the metal carbide detection and grading method as described in any of the first aspects.
[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the metal carbide detection and grading method as described in any of the first aspects.
[0022] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute the metal carbide detection and grading method described in any one of the first aspects.
[0023] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.
[0024] The metal carbide detection and grading method, apparatus, electronic device, and storage medium provided in this application embodiment input each image of the sample to be tested into a preliminary screening model to obtain multi-channel feature maps of each image to be tested. Based on each multi-channel feature map, the metal carbide image in the image to be tested is determined. Then, the metal carbide image is sequentially input into a target detection model, a grading model, and an integrated model to obtain the final carbide grade of the sample to be tested. This can improve the accuracy and efficiency of metal carbide detection and grading in the sample to be tested, while eliminating the need for manual inspection and avoiding visual fatigue for technicians.
[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description
[0026] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0027] Figure 1 This is a schematic flowchart of a metal carbide detection and grading method provided in an embodiment of this application;
[0028] Figure 2 This is a flowchart of a metal carbide detection and grading method provided in an embodiment of this application;
[0029] Figure 3 This is a schematic flowchart of a metal carbide detection and grading method provided in another embodiment of this application;
[0030] Figure 4 This is a schematic diagram of the structure of a metal carbide detection and grading device provided in an embodiment of this application;
[0031] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0032] The present application will be described more clearly below with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the function of the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.
[0033] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.
[0034] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0035] In the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0036] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.
[0037] Furthermore, the term "multiple" mentioned in the embodiments of this application should be interpreted as two or more.
[0038] In the steel production process, metal carbides of different grades exhibit various shapes and structures. Data identification of metal carbides is crucial for ensuring metallurgical quality; for example, identifying the inhomogeneity of metal carbides is an important indicator of metallurgical quality. Currently, the detection method for metal carbides is typically manual metallographic examination. This involves technicians visually observing the metallurgical sample under a microscope and then determining the carbide classification based on metallographic images, relevant testing standards, and their own professional knowledge. However, this method relies on human observation and experience, resulting in large errors in classification and low efficiency. Furthermore, it demands a high level of expertise from technicians and can easily cause visual fatigue.
[0039] Based on the above problems, the inventors discovered that the metal carbide image in the image to be detected can be determined according to the multi-channel feature map corresponding to the image to be detected. Then, based on the metal carbide image, the final carbide grade of the sample to be detected can be obtained, thereby improving the accuracy and efficiency of metal carbide detection and grading in the sample to be detected.
[0040] Figure 1 This is a schematic flowchart of a metal carbide detection and classification method provided in one embodiment of this application. Figure 1 As shown, the method in the embodiments of this application may include:
[0041] Step 101: Obtain multiple images of the sample to be tested.
[0042] In this embodiment, the multiple images to be tested are multiple metallographic images taken at different locations of the sample to be tested. Optionally, the multiple images to be tested may include images of the entire metallographic structure of the sample to be tested, thereby improving the accuracy of the processing results by subsequently determining the carbide grade based on the images of the entire metallographic structure.
[0043] Step 102: Input each of the multiple images to be detected into the initial screening model to obtain the multi-channel feature map of each image to be detected, and determine whether each image to be detected is a metal carbide image based on the multi-channel feature map of each image to be detected.
[0044] In this embodiment, the initial screening model inputs a metallographic image and outputs a multi-channel feature map.
[0045] For example, refer to Figure 2 In this embodiment, the initial screening model may include multiple deep learning networks, such as at least one of the ResNet series, Vgg series, and Swim Transformer series networks. Each deep learning network operates independently, independently extracting features from the image to be detected to obtain multi-channel feature maps corresponding to the image. In this embodiment, each image to be detected is input into each deep learning network included in the initial screening model to obtain multiple multi-channel feature maps. Then, based on the obtained multi-channel feature maps, it is determined whether each image to be detected is a metal carbide image. The aforementioned metal carbide image is an image containing metal carbides.
[0046] Optionally, before using the initial screening model, this embodiment first trains the model to obtain a trained initial screening model. For example, this embodiment can train multiple deep learning networks, such as ResNet, Vgg, and Swim Transformer. This embodiment uses the real image dataset ImageNet to train the ResNet, Vgg, and Swim Transformer networks respectively, and uses the second and third layers of the trained ResNet, the feature extraction layer of the Vgg network, and the encoder layer of the Swim Transformer network as the feature extraction layer of the initial screening model, thus using the obtained trained initial screening model as the initial screening model.
[0047] Step 103: Input each metal carbide image from the determined multiple images to be detected into the target detection model to obtain the target image, and input each target image into the grading model to obtain the carbide grade and confidence level corresponding to each target image.
[0048] In this embodiment, the target image is marked with a detection box, which is used to locate carbides in the image. The target detection model takes the carbide image as input and outputs the target image marked with the detection box. The grading model takes the target image marked with the detection box as input and outputs the carbide grade and confidence level.
[0049] For example, in this embodiment, the object detection model can be based on the YOLO series network as the backbone network. Similarly, before using the object detection model, it is first trained to obtain a trained object detection model. For instance, in this embodiment, the initial object detection model can be pre-trained using the COCO dataset, and then trained using a metallographic image dataset with labeled detection boxes, thus using the resulting trained initial object detection model as the object detection model. During training, the training sample images in the metallographic image dataset can be rotated clockwise by 90°, 180°, and 270° to augment the metallographic image dataset and improve training performance. Furthermore, to further improve training performance, the training sample images can be scaled during training. For example, a training sample image originally 1463×1463 pixels can be scaled to 640×640 pixels to improve model training speed.
[0050] Optionally, in this embodiment, the grading model can be a multi-classification network based on ResNet. Before using the grading model, this embodiment first trains the model to obtain a trained grading model. For example, this embodiment can pre-train the initial grading model using the ImageNet dataset, and then train the pre-trained initial grading model using a metallographic image dataset labeled with carbide levels, thereby using the obtained trained initial grading model as the grading model.
[0051] For example, during training, this embodiment first moves the metal carbide regions in the training sample images of the metallographic image dataset to the center of the training sample images. Furthermore, if a metal carbide region in a training sample image is located at the edge of the image, meaning the image only contains a portion of the metal carbide region (a fragmented metal carbide region), then the fragmented metal carbide region is stitched together based on each training sample image in the metallographic image dataset to obtain a complete metal carbide region. This complete metal carbide region is then moved to the center of the training sample image. The training sample image with the complete metal carbide region in the center is then used to train the pre-trained initial grading model.
[0052] Step 104: Based on the carbide grade and confidence level, select a preset number of target images and input them into the ensemble model to obtain the final carbide grade of the sample to be tested.
[0053] In this embodiment, the integrated model inputs multiple target images marked with detection boxes and outputs carbide levels.
[0054] In one possible implementation, this embodiment may include A1 to A3 when obtaining the final carbide grade of the sample to be tested.
[0055] A1. Sort the target images according to the carbide grade from high to low and the confidence level from high to low to obtain the sorting results.
[0056] A2. Determine the preset number of target images based on the sorting results.
[0057] A3. Input the preset number of target images into the integrated model to obtain the final carbide grade of the sample to be tested.
[0058] For example, in this embodiment, the carbide grade can be divided into multiple levels, such as Level 1, Level 2, and Level 3, with Level 3 being the highest. Of course, the carbide grade can also be set to other levels as needed. Furthermore, the target images are sorted according to the carbide grade from high to low and the confidence level from high to low. For example, Level 3 target images can be sorted first according to confidence level from high to low, then Level 2 target images can be sorted according to confidence level from high to low, and finally Level 1 target images can be sorted according to confidence level from high to low to obtain the sorted result of all target images. The aforementioned preset number can be set as needed; for example, the preset number can be 20.
[0059] Optionally, in this embodiment, the ensemble model can be an encoder-decoder network. Before using the ensemble model, this embodiment first trains the model to obtain a trained ensemble model. For example, this embodiment can use a metallographic image dataset labeled with carbide grades, confidence levels, and final carbide grades to train the initial ensemble model, thereby obtaining the trained initial ensemble model as the ensemble model.
[0060] In one possible implementation, after inputting each metal carbide image from a plurality of determined images to be detected into a target detection model to obtain a target image, this embodiment may further input the target image into a thinning model to obtain a new target image with segmentation information.
[0061] In this embodiment, the segmentation information is used to indicate the carbides and background in the new target image. The segmentation information includes a matrix of the same size as the new target image, and the pixel values at the carbides in the matrix are different from the pixel values at the background. For example, the pixel values at the carbides can be 1, and the pixel values at the background can be 0. The thinning model takes the target image marked with detection boxes as input and outputs a target image with segmentation information.
[0062] For example, this embodiment utilizes a thinning model to adjust the detection box region in the target image, thereby enabling more accurate identification of metallic carbides during subsequent carbide detection based on the adjusted image. Here, the thinning model includes a hierarchical feature extraction network and a hierarchical thinning network. The hierarchical feature extraction network extracts features of different scales from the detection box region in the target image in a progressively larger order. The hierarchical thinning network then adjusts the detection box of the target image layer by layer according to the obtained features, in ascending order. Specifically, this embodiment can first detect the global edges of the detection box, and then further refine the enlarged edge features to finally obtain a new target image with segmentation information. Before using the thinning model, this embodiment first trains the model to obtain a trained thinning model. For example, this embodiment can use a pre-set salient target detection dataset and a camouflaged target detection dataset to pre-train the hierarchical thinning network.
[0063] Accordingly, in this embodiment, when inputting each target image into the grading model to obtain the carbide grade and confidence level corresponding to each target image, each new target image can be input into the grading model to obtain the carbide grade and confidence level corresponding to each new target image.
[0064] In this embodiment, by setting a refinement model, the range of metal carbides in the new target image can be further determined, thereby obtaining a more accurate carbide grade and confidence level based on the new target image.
[0065] The metal carbide detection and grading method provided in this application involves inputting each image of the sample to be tested into a preliminary screening model to obtain multi-channel feature maps of each image. Based on these multi-channel feature maps, the metal carbide images in the images to be tested are determined. The metal carbide images are then sequentially input into a target detection model, a grading model, and an integrated model to obtain the final carbide grade of the sample. This method improves the accuracy and efficiency of metal carbide detection and grading in the sample, while eliminating the need for manual inspection and preventing visual fatigue for technicians.
[0066] To more accurately detect and classify metal carbides in the sample to be tested, based on the above embodiments, the features of each position in the multi-channel feature map of the image to be tested can be determined, and then the image to be tested can be accurately determined as a metal carbide image based on the features of each position.
[0067] Figure 3 This is a schematic flowchart of a metal carbide detection and grading method provided in another embodiment of this application. Figure 3 As shown, the method in the embodiments of this application may include:
[0068] Step 101: Obtain multiple images of the sample to be tested.
[0069] Step 201: Input each of the multiple images to be detected into multiple feature extraction networks to obtain multiple multi-channel feature maps for each image to be detected.
[0070] In this embodiment, each multi-channel feature map of each image to be detected corresponds one-to-one with multiple feature extraction networks.
[0071] For example, as described above, the multiple feature extraction networks in this embodiment can be ResNet, Vgg, and Swim Transformer networks. Each feature extraction network operates independently, independently extracting block-level features from the image to be detected, resulting in multiple multi-channel feature maps of different styles and scales for each image to be detected.
[0072] Step 202: For each multi-channel feature map of the image to be detected, determine the features at each position in each multi-channel feature map.
[0073] In this context, the feature at each position in each multi-channel feature map corresponds to each pixel in the corresponding multi-channel feature map.
[0074] In one possible implementation, determining the features at each location may include: for each multi-channel feature map of the image to be detected, determining the location region corresponding to each location in each multi-channel feature map; for each multi-channel feature map, calculating the value of an aggregation function based on the multi-channel features corresponding to the location region at each location, and using the value of the aggregation function as the feature of the corresponding location in the corresponding multi-channel feature map.
[0075] For example, in this embodiment, each pixel in the multi-channel feature map represents a position in the multi-channel feature map. For each position X in the multi-channel feature map, a region of a preset size centered at position X is taken as the position region corresponding to position X. Then, based on the multi-channel features corresponding to the position region of position X in the multi-channel feature map, an aggregation function is calculated and its value is determined. The obtained aggregation function value is used as the feature corresponding to position X.
[0076] In this embodiment, by determining the location region corresponding to each location X, and using the aggregated value of the multi-channel features corresponding to the location region as the feature of that location X, the features of each location X can be obtained more accurately.
[0077] Step 203: Determine the outlier map corresponding to each image to be detected based on the features of each position in each multi-channel feature map of each image to be detected and the preset feature library.
[0078] In this embodiment, the preset feature library includes multiple sub-feature libraries, each of which corresponds one-to-one with a feature extraction network. Each sub-feature library in the multiple sub-feature libraries also corresponds to a multi-channel feature of each image to be detected. Figure 1 One-to-one correspondence; each sub-feature library includes standard features at each position in the multi-channel feature map corresponding to multiple positive sample images; positive sample images are sample images that do not include metal carbides.
[0079] In one possible implementation, this embodiment may include B1 to B2 when determining the outlier map corresponding to each image to be detected.
[0080] B1. For each image to be detected, determine the distance between the feature at each position in each multi-channel feature map and each standard feature in the corresponding sub-feature library of the multi-channel feature map, obtain multiple feature distances, and take the feature distance with the smallest value among the multiple feature distances as the outlier at each position in the corresponding multi-channel feature map. Based on the outlier at each position in the obtained multi-channel feature map, construct the sub-outlier map corresponding to each image to be detected.
[0081] B2. Based on the sub-outlier maps corresponding to each image to be detected, determine the outlier map corresponding to each image to be detected.
[0082] In this embodiment, each sub-outlier map corresponding to each image to be detected and each multi-channel feature of each image to be detected are... Figure 1 One-to-one correspondence.
[0083] It should be noted that, before determining the outlier map corresponding to each image to be detected, this embodiment may also include constructing a preset feature library. In constructing the preset feature library, this embodiment may include C1 to C3.
[0084] C1. Input each positive sample image from the multiple positive sample images into the multiple feature extraction networks of the initial screening model to obtain multiple standard multi-channel feature maps for each positive sample image.
[0085] C2. For each positive sample image, determine the standard features at each standard position in the standard multi-channel feature map.
[0086] C3. Based on the positional features of each standard position in the corresponding standard multi-channel feature map of all positive sample images, construct each sub-feature library, and construct a preset feature library based on each sub-feature library.
[0087] In this embodiment, the specific implementation process and principle of determining the standard features at each standard position in each standard multi-channel feature map can be referred to in step 202 for determining the features at each position in each multi-channel feature map, and will not be repeated here.
[0088] For example, as mentioned above, multiple feature extraction networks can be ResNet, Vgg, and SwimTransformer. Then, the standard multi-channel feature maps obtained after each positive sample image passes through the above three feature extraction networks are defined as standard multi-channel feature map R, standard multi-channel feature map V, and standard multi-channel feature map S, respectively.
[0089] For example, in this embodiment, the positional features of each standard position in the standard multi-channel feature map R of all positive sample images are placed into the first sub-feature library without distinguishing position. The first sub-feature library is used as the sub-feature library corresponding to the ResNet network. The positional features of each standard position in the standard multi-channel feature map V of all positive sample images are placed into the second sub-feature library without distinguishing position. The second sub-feature library is used as the sub-feature library corresponding to the Vgg network. And the positional features of each standard position in the standard multi-channel feature map S of all positive sample images are placed into the third sub-feature library without distinguishing position. The third sub-feature library is used as the sub-feature library corresponding to the Swim Transformer network. Thus, each sub-feature library corresponding one-to-one with each feature extraction network is obtained.
[0090] For example, as mentioned above, multiple feature extraction networks can be ResNet, Vgg, and SwimTransformer. Then, the corresponding multi-channel feature maps obtained after each image to be detected passes through the above three feature extraction networks are defined as multi-channel feature map r, multi-channel feature map v, and multi-channel feature map s, respectively.
[0091] For example, for each image Y to be detected, this embodiment determines the distance between the feature of each position P in the multi-channel feature map r corresponding to the image Y and the feature of each position in the first sub-feature library, and takes the feature distance with the smallest value among the multiple feature distances as the outlier of position P in the multi-channel feature map r corresponding to the image Y. Then, based on the set of all outliers of position P in the multi-channel feature map r corresponding to the image Y, a sub-outlier map Tr of the image Y to be detected is constructed.
[0092] Similarly, for each image Y to be detected, this embodiment determines the distance between the feature at each position Z in the multi-channel feature map v corresponding to the image Y and the feature at each position in the second sub-feature library, and takes the feature distance with the smallest value among the multiple feature distances as the outlier at position Z in the multi-channel feature map v corresponding to the image Y. Based on the set of all outliers at position Z in the multi-channel feature map v corresponding to the image Y, a sub-outlier map Tv for the image Y is constructed.
[0093] Furthermore, for each image Y to be detected, this embodiment determines the distance between the feature at each position M in the multi-channel feature map s corresponding to the image Y and the feature at each position in the third sub-feature library, and takes the feature distance with the smallest value among the multiple feature distances as the outlier at position M in the multi-channel feature map s corresponding to the image Y. Based on the set of all outliers at position M in the multi-channel feature map s corresponding to the image Y, a sub-outlier map Ts of the image Y to be detected is constructed.
[0094] For example, in this embodiment, the outliers in the sub-outlier maps Tr, Tv, and Ts of the image to be detected are averaged to obtain the outlier map of the image to be detected, wherein the outlier map includes the final outlier corresponding to each position.
[0095] Step 204: Determine whether each image to be detected is a metal carbide image based on the outlier map corresponding to each image to be detected.
[0096] In one possible implementation, when determining whether each image to be detected is a metal carbide image, this embodiment can first determine whether there are outliers in the outlier map corresponding to each image to be detected that are greater than a preset threshold; if there are outliers in the outlier map that are greater than the preset threshold, then the image to be detected corresponding to the outlier map with outliers greater than the preset threshold is taken as a metal carbide image.
[0097] For example, in this embodiment, if there is an outlier in the outlier map of the image Y to be detected that is greater than a preset threshold, then the image Y to be detected is determined to be a metal carbide image.
[0098] Step 103: Input each metal carbide image from the determined multiple images to be detected into the target detection model to obtain the target image, and input each target image into the grading model to obtain the carbide grade and confidence level corresponding to each target image.
[0099] Step 104: Based on the carbide grade and confidence level, select a preset number of target images and input them into the ensemble model to obtain the final carbide grade of the sample to be tested.
[0100] The metal carbide detection and grading method provided in this application involves inputting each image of the sample to be tested into a preliminary screening model to obtain a multi-channel feature map of each image, determining the features at each position in the multi-channel feature map, and then determining the metal carbide image in the image to be tested based on the features at each position. The metal carbide image is then sequentially input into a target detection model, a grading model, and an integrated model to obtain the final carbide grade of the sample to be tested. This method can accurately determine the metal carbide image, thereby improving the accuracy and efficiency of metal carbide detection and grading in the sample to be tested.
[0101] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0102] Figure 4 This is a schematic diagram of the structure of a metal carbide detection and grading device provided in one embodiment of this application. Figure 4As shown, the metal carbide detection and grading device provided in this embodiment may include: an image acquisition module 301, an image determination module 302, a grade prediction module 303, and a grade determination module 304.
[0103] The image acquisition module 301 is used to acquire multiple images to be detected corresponding to the sample to be detected; wherein the multiple images to be detected are multiple metallographic images taken at different positions of the sample to be detected.
[0104] The image determination module 302 is used to input each of the multiple images to be detected into the preliminary screening model to obtain a multi-channel feature map of each image to be detected, and to determine whether each image to be detected is a metal carbide image based on the multi-channel feature map of each image to be detected. The preliminary screening model inputs a metallographic image and outputs a multi-channel feature map.
[0105] The grading prediction module 303 is used to input each metal carbide image from a plurality of determined images to be detected into the target detection model to obtain a target image, and to input each target image into the grading model to obtain the carbide grade and confidence level corresponding to each target image; wherein, the target image is marked with a detection box, which is used to locate the carbide in the image; the target detection model inputs the carbide image and outputs the target image marked with the detection box; the grading model inputs the target image marked with the detection box and outputs the carbide grade and confidence level.
[0106] The grade determination module 304 is used to select a preset number of target images and input them into the ensemble model according to the carbide grade and confidence level to obtain the final carbide grade of the sample to be tested. The ensemble model inputs multiple target images marked with detection boxes and outputs the carbide grade.
[0107] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.
[0108] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 5 As shown, the electronic device 400 of this embodiment includes a processor 410 and a memory 420, wherein the memory 420 stores a computer program 421 that can run on the processor 410. When the processor 410 executes the computer program 421, it implements the steps in any of the above-described method embodiments, for example... Figure 1 Steps 101 to 104 are shown. Alternatively, when processor 410 executes computer program 421, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 4 The functions of modules 301 to 304 are shown.
[0109] For example, computer program 421 may be divided into one or more modules / units, one or more of which are stored in memory 420 and executed by processor 410 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 421 in electronic device 400.
[0110] Those skilled in the art will understand that Figure 5 This is merely an example of an electronic device and does not constitute a limitation on the electronic device. It may include more or fewer components than shown, or combinations of certain components, or different components, such as input / output devices, network access devices, buses, etc.
[0111] The processor 410 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.
[0112] The memory 420 can be an internal storage unit of the electronic device, such as a hard drive or memory, or an external storage device, such as a plug-in hard drive, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card. The memory 420 can also include both internal and external storage units. The memory 420 is used to store computer programs and other programs and data required by the electronic device. The memory 420 can also be used to temporarily store data that has been output or will be output.
[0113] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0114] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0115] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0116] In the embodiments provided by this invention, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0117] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0118] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0119] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0120] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A method for detecting and classifying metal carbides, characterized in that, include: Acquire multiple images corresponding to the sample to be tested; wherein, the multiple images to be tested are multiple metallographic images taken at different positions of the sample to be tested; Each of the plurality of images to be detected is input into the initial screening model to obtain a multi-channel feature map of each image to be detected. The initial screening model inputs a metallographic image and outputs a multi-channel feature map. Based on the multi-channel feature map of each image to be detected, it is determined whether each image to be detected is a metal carbide image. Each metal carbide image from the determined plurality of images to be detected is input into the target detection model to obtain a target image. Each target image is then input into a grading model to obtain the carbide grade and confidence level corresponding to each target image. The target image is marked with a detection box, which is used to locate the carbides in the image. The target detection model inputs the carbide image and outputs the target image marked with the detection box. The grading model inputs the target image marked with the detection box and outputs the carbide grade and confidence level. Based on the carbide grade and the confidence level, a preset number of target images are selected and input into the ensemble model to obtain the final carbide grade of the sample to be detected. The ensemble model inputs multiple target images marked with detection boxes and outputs the carbide grade. The initial screening model includes multiple feature extraction networks; Each of the plurality of images to be detected is input into the initial screening model to obtain a multi-channel feature map of each image to be detected. Based on the multi-channel feature map of each image to be detected, it is determined whether each image to be detected is a metal carbide image, including: Each of the plurality of images to be detected is input into the plurality of feature extraction networks to obtain a plurality of multi-channel feature maps for each image to be detected; wherein, each multi-channel feature map of each image to be detected corresponds one-to-one with the plurality of feature extraction networks; For each multi-channel feature map of the image to be detected, the features at each position in each multi-channel feature map are determined; wherein, the features at each position in each multi-channel feature map correspond to each pixel in the corresponding multi-channel feature map. Based on the features at each position in each multi-channel feature map of each image to be detected and a preset feature library, the outlier map corresponding to each image to be detected is determined. Based on the outlier map corresponding to each image to be detected, determine whether each image to be detected is a metal carbide image; The step of selecting a preset number of target images and inputting them into the ensemble model based on the carbide grade and the confidence level to obtain the final carbide grade of the sample to be detected includes: The target images are sorted according to the carbide level from high to low and the confidence level from large to small to obtain the sorting result; The preset number of target images is determined based on the sorting results; The predetermined number of target images are input into the integrated model to obtain the final carbide grade of the sample to be tested.
2. The method for detecting and classifying metal carbides according to claim 1, characterized in that, For each multi-channel feature map of the image to be detected, determining the features at each position in each multi-channel feature map includes: For each multi-channel feature map of the image to be detected, determine the location region corresponding to each position in each multi-channel feature map; For each multi-channel feature map, the value of the aggregation function is calculated based on the multi-channel features corresponding to the location region at each position, and the value of the aggregation function is used as the feature at the corresponding position in the corresponding multi-channel feature map.
3. The method for detecting and classifying metal carbides according to claim 1, characterized in that, The preset feature library includes multiple sub-feature libraries, each of which corresponds one-to-one with a feature extraction network and each of which corresponds one-to-one with a multi-channel feature map of each image to be detected. Each sub-feature library includes standard features at each position in the multi-channel feature map of multiple positive sample images. The positive sample images are sample images that do not contain metal carbides. The step of determining the outlier map corresponding to each image to be detected based on the features at each position in each multi-channel feature map of each image to be detected and a preset feature library includes: For each image to be detected, the distance between the feature at each position in each multi-channel feature map and each standard feature in the corresponding sub-feature library of the multi-channel feature map is determined to obtain multiple feature distances. The feature distance with the smallest value among the multiple feature distances is taken as the outlier at each position in the corresponding multi-channel feature map. Based on the outlier at each position in the obtained multi-channel feature map, a sub-outlier map corresponding to each image to be detected is constructed. Based on each sub-outlier map corresponding to each image to be detected, an outlier map corresponding to each image to be detected is determined; each sub-outlier map corresponding to each image to be detected corresponds one-to-one with each multi-channel feature map of each image to be detected.
4. The method for detecting and classifying metal carbides according to claim 1, characterized in that, The step of determining whether each image to be detected is a metal carbide image based on the outlier map corresponding to each image to be detected includes: Determine whether there are outliers greater than a preset threshold in the outlier map corresponding to each image to be detected; If there is an outlier in the outlier map that is greater than the preset threshold, then the image to be detected corresponding to the outlier map where the outlier is greater than the preset threshold is taken as the metal carbide image.
5. The method for detecting and classifying metal carbides according to claim 1, characterized in that, After inputting each of the determined metal carbide images from the plurality of images to be detected into the target detection model to obtain the target image, the method further includes: The target image is input into a thinning model to obtain a new target image with segmentation information; the segmentation information is used to indicate the carbides and background in the new target image; the thinning model inputs a target image marked with detection boxes and outputs a target image with segmentation information. Accordingly, the step of inputting each target image into the grading model to obtain the carbide grade and confidence level corresponding to each target image includes: Each new target image is input into the grading model to obtain the carbide grade and confidence level corresponding to each new target image.
6. A metal carbide detection and grading device, characterized in that, include: The image acquisition module is used to acquire multiple images to be detected corresponding to the sample to be detected; wherein, the multiple images to be detected are multiple metallographic images taken at different positions of the sample to be detected; The image determination module is used to input each of the plurality of images to be detected into a preliminary screening model to obtain a multi-channel feature map of each image to be detected, wherein the preliminary screening model inputs a metallographic image and outputs a multi-channel feature map; and determines whether each image to be detected is a metal carbide image based on the multi-channel feature map of each image to be detected. The grade prediction module is used to input each metal carbide image from the determined plurality of images to be detected into the target detection model to obtain a target image, and to input each target image into the grading model to obtain the carbide grade and confidence level corresponding to each target image; wherein, the target image is marked with a detection box, the detection box is used to locate the carbide in the image, the target detection model inputs the carbide image and outputs the target image marked with the detection box, and the grading model inputs the target image marked with the detection box and outputs the carbide grade and confidence level; The grade determination module is used to select a preset number of target images to input into the ensemble model based on the carbide grade and the confidence level, so as to obtain the final carbide grade of the sample to be detected. The ensemble model inputs multiple target images marked with detection boxes and outputs the carbide grade. The initial screening model includes multiple feature extraction networks; the image determination module is further configured to input each of the multiple images to be detected into the multiple feature extraction networks respectively to obtain multiple multi-channel feature maps of each image to be detected; wherein each multi-channel feature map of each image to be detected corresponds one-to-one with the multiple feature extraction networks. For each multi-channel feature map of the image to be detected, the features at each position in each multi-channel feature map are determined; wherein, the features at each position in each multi-channel feature map correspond to each pixel in the corresponding multi-channel feature map. Based on the features at each position in each multi-channel feature map of each image to be detected and a preset feature library, the outlier map corresponding to each image to be detected is determined. Based on the outlier map corresponding to each image to be detected, determine whether each image to be detected is a metal carbide image; The grade determination module is further configured to sort the target images according to the carbide grade from high to low and the confidence level from large to small, and obtain a sorting result; The preset number of target images is determined based on the sorting results; The predetermined number of target images are input into the integrated model to obtain the final carbide grade of the sample to be tested.
7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the metal carbide detection and grading method as described in any one of claims 1 to 5.
8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the metal carbide detection and grading method as described in any one of claims 1 to 5.