Training sample picture preprocessing method and device, electronic equipment and neural network model
By preprocessing the metallographic images of gear steel using sliding window cropping and data augmentation techniques, pseudo-labels are generated and unlabeled images are used to train a neural network. This solves the problem of insufficient training samples, improves the generalization performance and recognition accuracy of the model, and enables precise quantitative analysis of the residual austenite content.
Patent Information
- Application Number
- CN202511460425.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-13
- Publication Date
- 2026-01-09
AI Technical Summary
Existing technologies are insufficient to effectively solve the quantitative analysis of the residual austenite content in metallographic images of gear steel. Traditional methods are inefficient, costly, and difficult to achieve visualized quantitative analysis of microstructure distribution. Insufficient training samples for neural network models result in poor model generalization and low recognition accuracy.
The sample images are preprocessed using sliding window cropping and data augmentation techniques. High-resolution sample images are cropped into multiple low-pixel cropped images, and pseudo-labels are generated through a semantic segmentation neural network. An enhanced hybrid matching algorithm is then used to train the algorithm on unlabeled images.
The number of training samples was significantly increased, the detailed features of the original images were preserved, the generalization performance and recognition accuracy of the neural network model were improved, and intelligent and accurate quantitative analysis of the residual austenite content was achieved.
Smart Images
Figure CN121305017A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and more specifically to a method, apparatus, and electronic device for preprocessing training sample images. Background Technology
[0002] Metallographic images are obtained by observing the internal microstructure of metallic materials using a metallographic microscope or other metallographic analysis techniques. They can visually display key information such as the crystal structure, phase composition, grain size, morphology, and distribution of the metal, and are an important basis for evaluating the performance, quality, and processing effects of metallic materials. Metallographic images of gear steel can reveal the microstructure characteristics of gear steel, mainly including acicular or lath martensite formed after quenching (which determines hardness), retained austenite (white blocky), and possibly bainite (feathery or acicular). Retained austenite usually appears as white blocky or stripy areas (not corroded by nitric acid or alcohol) distributed on a martensitic matrix. Identifying the content of retained austenite is of great significance: an appropriate amount of retained austenite (usually controlled between 5% and 20%) can improve the toughness and impact resistance of gears and reduce the tendency for quenching cracks; however, excessive content will reduce surface hardness and wear resistance, and during service, due to stress or temperature changes, it will transform into martensite, leading to dimensional instability or microcracks. By quantitatively analyzing the residual austenite in metallographic images (such as by combining X-ray diffraction), heat treatment processes can be optimized (such as adjusting the tempering temperature or adding deep cryogenic treatment) to ensure that gears have both high strength and dimensional stability under load conditions.
[0003] Traditional methods for determining the content of retained austenite (such as X-ray diffraction) rely on expensive equipment and are difficult to visualize and quantitatively analyze the distribution of microstructures. As a result, there are few metallographic images with retained austenite calibration, and the training samples for neural network models used to analyze retained austenite in metallographic images are severely insufficient. Summary of the Invention
[0004] In view of this, embodiments of the present invention provide a method, apparatus, electronic device, and neural network model for preprocessing training sample images to address the problem of insufficient training samples for neural network models.
[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:
[0006] A training sample image preprocessing method includes:
[0007] Extract sample images;
[0008] A sliding window is used to perform cropping preprocessing on the sample image to crop the sample image into multiple low-pixel cropped images;
[0009] The cropped image undergoes data augmentation preprocessing to achieve a target resolution, which is the resolution of the input image for the neural network model to be trained.
[0010] Optionally, in the above training sample image preprocessing method, a sliding window is used to perform cropping preprocessing on the sample images, including:
[0011] N horizontal coordinate sampling points are uniformly set along the horizontal coordinate of the sample image, where N is a positive integer not less than 1;
[0012] M vertical coordinate sampling points are uniformly set along the vertical coordinate of the sample image, where M is a positive integer not less than 1;
[0013] Based on the horizontal and vertical sampling points, N×M starting points are determined. The sample image is cropped based on each starting point to obtain N×M cropped images. The difference between the resolution of the cropped images and the target resolution is within an allowable range.
[0014] Optionally, in the above training sample image preprocessing method, before performing cropping preprocessing on the sample images using a sliding window, the method further includes:
[0015] Obtain the resolution of the input image for the neural network model to be trained.
[0016] Optionally, the above training sample image preprocessing method further includes:
[0017] Extract the cropped images corresponding to the unlabeled images from the sample images, and denote them as unlabeled cropped images;
[0018] Add random noise to unlabeled cropped images;
[0019] A semantic segmentation neural network model is used to predict unlabeled cropped images with added random noise, and the pseudo-labels and confidence scores of the unlabeled cropped images are obtained.
[0020] Unlabeled cropped images with confidence levels lower than the target confidence level are identified as the first images to be mixed and cropped.
[0021] The cropped image corresponding to at least a portion of the labeled image in the sample image is denoted as the second image to be mixed and cropped.
[0022] The first image to be mixed and cropped is then subjected to a blending enhancement process using the second image to be mixed and cropped.
[0023] Obtain the pseudo-tags of the first image to be blended and cropped after the blending enhancement process.
[0024] Optionally, in the above training sample image preprocessing method, after extracting the cropped images corresponding to the unlabeled images in the sample images and denoting them as unlabeled cropped images, and before adding random noise to each unlabeled cropped image, the method further includes:
[0025] Each of the unlabeled cropped images is copied to obtain K copies of the unlabeled cropped images, where K is a positive integer not less than 1;
[0026] The step of adding random noise to each unlabeled cropped image includes adding random noise to K unlabeled cropped images respectively;
[0027] The semantic segmentation neural network model is used to predict the pseudo-label and confidence score of each unlabeled cropped image after adding random noise, including:
[0028] A semantic segmentation neural network model is used to predict K unlabeled cropped images after random noise is added, so as to obtain the confidence score of each unlabeled cropped image and the pseudo-label of the unlabeled cropped image.
[0029] Optionally, in the above training sample image preprocessing method, determining the unlabeled cropped images with a confidence level lower than the target confidence level includes: obtaining K unlabeled cropped images with a confidence level lower than the target confidence level;
[0030] The first image to be mixed and cropped is subjected to a blending enhancement process using the second image to be mixed and cropped, including:
[0031] A second image to be cropped and blended is cropped and blended with the first image to be cropped and blended for enhancement.
[0032] Perform pixel blending enhancement on another second image to be blended and cropped, and the first image to be blended and enhanced.
[0033] Optionally, in the above training sample image preprocessing method, the sample image is a metallographic image of gear steel, and the labeled target in the sample image is retained austenite.
[0034] A neural network model training sample image preprocessing device, comprising:
[0035] The sample extraction unit is used to extract sample images;
[0036] The cropping unit is used to perform cropping preprocessing on the sample image using a sliding window, so as to crop the sample image into multiple low-pixel cropped images;
[0037] The first preprocessing unit is used to perform data augmentation preprocessing on the cropped image so that the resolution of the cropped image is the target resolution.
[0038] An electronic device includes at least one processing device and a storage device connected to the processing device, wherein:
[0039] The storage device is used to store computer programs;
[0040] The processing device is used to execute the computer program so that the electronic device can implement any of the training sample image preprocessing methods described above.
[0041] A neural network model, comprising:
[0042] The neural network model is trained using any of the above-mentioned training sample image preprocessing methods.
[0043] Based on the above technical solution, the solution provided by the embodiments of the present invention preprocesses the sample image by using a sliding window to crop the sample image into multiple low-pixel cropped images. Each cropped image can be used as the input image of the neural network model without large-scale image scaling, which greatly increases the number of sample images used to train the neural network model. Furthermore, it avoids the problem of losing a large number of image detail features caused by large-scale scaling of sample images, thus preserving most of the detail features in the sample image. Attached Figure Description
[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0045] Figure 1 This is a flowchart illustrating the training sample image preprocessing method disclosed in this application;
[0046] Figure 2 This is a comparison image showing an original image and an annotated image from the dataset;
[0047] Figure 3 This is a schematic diagram of the cropped image obtained by using a sliding window cropping technique on an original metallographic image of gear steel.
[0048] Figure 4 This is a schematic flowchart of a training sample image preprocessing method disclosed in another embodiment of this application;
[0049] Figure 5 This is a flowchart illustrating the enhanced hybrid matching algorithm disclosed in an embodiment of this application;
[0050] Figure 6 This is a schematic diagram of the structure of the neural network model training sample image preprocessing device disclosed in this application;
[0051] Figures 7-9 Example images comparing the prediction results of three neural network models trained using the scheme disclosed in this application with the original and labeled images are presented. Detailed Implementation
[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0053] Traditional methods for determining retained austenite content (such as X-ray diffraction or manual microscopy) suffer from low efficiency, high cost, and insufficient local characterization, making them unsuitable for rapid industrial testing. While deep learning-based image semantic segmentation technology has enabled automated analysis in recent years, the small data volume, high annotation accuracy requirements, and difficulty in annotating gear steel metallographic images lead to insufficient training data and inadequate neural network learning in fully supervised learning schemes. This results in poor model generalization, low recognition accuracy, and an inability to stably apply the model to real-world industrial scenarios. Furthermore, the high resolution and fine-grained distribution of retained austenite in gear steel metallographic images mean that directly compressing the entire image and inputting it as a single sample into a neural network model (such as a semi-supervised learning model) reduces the utilization of metallographic image information.
[0054] To address the aforementioned issues, this application provides a preprocessing method for training sample images suitable for determining residual austenite in the metallographic structure of gear steel, thereby providing sufficient training data for the neural network model. (See [link to relevant documentation]). Figure 1 The preprocessing method includes:
[0055] Step S101: Extract sample images.
[0056] The sample images are used to train the neural network model. The pixel size of these sample images is much larger than the target pixel size that the neural network model can input. For example, in a scenario provided in this application, the sample image can be a metallographic image of gear steel with a resolution of up to 1944x2580, while the target pixel size that the neural network model can input is only 320x320.
[0057] The sample images constitute a semantic segmentation dataset, which includes a training set and a test set. The training set contains not only precisely labeled images but also a large number of unlabeled images to support semi-supervised learning strategies. Taking the metallographic images of gear steel as an example, if the labeled object is retained austenite, the labeled images are rigorously annotated by metallographic experts with pixel-level accuracy, accurately identifying the boundary features of retained austenite and other microstructures in the metallographic image. These labeled images contain two semantic segmentation categories (retained austenite category and other microstructure categories). The test set can use independent samples to ensure the objectivity of model evaluation. To make the model training results more reliable, the dataset construction fully considers the diversity of gear steel metallographic structures, covering different material compositions, heat treatment processes, and microstructural features. That is, the dataset includes gear steel metallographic images made of different materials and with different processes, providing a reliable data foundation for subsequent deep learning model training and optimization. Ultimately, the training set of this dataset can contain 352 training sample images (image resolution 1944x2580), of which 102 are labeled training sample images and 250 are unlabeled training sample images. The test set of this dataset can include 21 labeled test sample images. This dataset can effectively solve the problems of low efficiency and strong subjectivity of manual annotation in traditional gear steel metallographic analysis, and realize intelligent and accurate quantitative analysis of residual austenite content. Figure 2 The image shows a comparison between an original image from the dataset and an annotated image (in the annotated image, the red area represents the residual austenite region, and the green area represents other microstructure regions).
[0058] Step S102: Use a sliding window to perform cropping preprocessing on the sample image to crop the sample image into multiple low-pixel cropped images.
[0059] Before inputting sample images into the neural network model to be trained, the sample images need to undergo data augmentation preprocessing to ensure that the resolution of the processed image matches the target resolution of the images that the neural network can input. Different neural network models have different target resolutions. Before using a sliding window to crop the sample images, the resolution of the input image to the neural network model to be trained can be obtained by data scraping from the model or through user input, and this resolution can be used as the target resolution. Traditional data augmentation techniques include randomly flipping, randomly cropping, randomly scaling, randomly adding color noise, and randomly converting the original image to grayscale. Traditional techniques only work when the difference between the pixel count of the sample image and the target resolution of the neural network model is small. However, the gear steel metallographic image dataset used in this invention has a large difference between the resolution of the sample images and the target resolution. For example, the resolution of the sample images in the gear steel metallographic image dataset is 1944x2580, while the input resolution (target resolution) of the neural network is 320x320. If traditional data augmentation techniques are used to directly scale the entire sample image to a resolution of 320x320, or to randomly crop it to a resolution of 320x320, a significant amount of information about the residual austenite distribution in the original image will be lost. To address this problem, this invention proposes a data augmentation technique based on sliding window cropping in the sample preprocessing stage.
[0060] Specifically, in this step, a sliding window is used to crop the high-resolution sample images in the dataset, dividing the sample images into multiple uniformly sized and consecutively cropped images. The resolution of these cropped images is within an acceptable range compared to the target resolution. Each cropped image can then be individually input into the neural network model for training.
[0061] During cropping, N horizontal coordinate sampling points can be uniformly set along the horizontal axis of the sample image, where N is a positive integer not less than 1; M vertical coordinate sampling points can be uniformly set along the vertical axis of the sample image, where M is a positive integer not less than 1; N×M starting points are determined based on the horizontal and vertical coordinate sampling points, and the sample image is cropped based on each starting point to obtain N×M cropped images. The difference between the resolution of the cropped images and the target pixels is within an allowable range. That is, the difference between the number of pixels on the horizontal axis of the cropped image and the number of pixels on the horizontal axis of the target pixels is within an allowable range, and the difference between the number of pixels on the vertical axis of the cropped image and the number of pixels on the vertical axis of the target pixels is within an allowable range. In this scheme, in order to ensure the training results of the model, it is necessary to ensure that the number of pixels on the horizontal axis of the cropped image is greater than the number of pixels on the horizontal axis of the target resolution, and the number of pixels on the vertical axis of the cropped image is greater than the number of pixels on the vertical axis of the target resolution.
[0062] For example, for a sample image with a resolution of 1944x2580, during the cropping process, a sampling point is set every 162 pixels on the 1944-pixel height, for a total of 11 sampling points along the vertical axis. At this point, the pixels on the vertical axis can be divided into 12 parts, and each sampling point on the vertical axis serves as the starting point of the cropped image's vertical coordinate, for a total of 11 starting points. Then, a sampling point is set every 215 pixels on the 2580-pixel width, for a total of 11 sampling points. At this point, the pixels on the horizontal axis can be divided into 12 parts, and the origin of the horizontal axis and each sampling point on the horizontal axis serve as the starting point of the cropped image's vertical coordinate, for a total of 11 starting points. Combining the 11 horizontal coordinate starting points and the 11 vertical coordinate starting points yields 121 starting points. These starting points correspond to the starting points of each cropped image. During cropping, the original image is cropped using these starting points as the origin (the midpoint of the cropped image), resulting in 121 cropped images. Each cropped image has a resolution of 324x430, thus completing the preprocessing of the sample image cropping. Figure 3 This shows a schematic diagram of the cropped image obtained by using a sliding window cropping technique on an original metallographic image of gear steel.
[0063] Furthermore, the cropped image, when unfolded, can cover the entire sample image. Adjacent cropped images will also overlap; by setting these overlapping areas, the sensitivity of the neural network to pixel location can be reduced.
[0064] Step S103: Perform data augmentation preprocessing on the cropped image to make the resolution of the cropped image the target resolution, which is the resolution of the input image of the neural network model to be trained.
[0065] After obtaining these cropped images, traditional data augmentation preprocessing is performed on each image. This involves random flipping, random cropping, random scaling, random addition of color noise, and random conversion to grayscale to normalize the images to the target resolution before inputting them into the neural network. This image preprocessing operation based on sliding window cropping and traditional data augmentation not only increases the number of image samples by N×M times, significantly increasing the dataset size and enhancing the generalization performance of the neural network, but also ensures that the cropped images before inputting into the neural network have not undergone significant image scaling, thus preserving the detailed information of the original images and enabling the model to better segment and extract features from the cropped images.
[0066] Therefore, this invention employs a sliding window to preprocess the sample images by cropping them into multiple low-pixel cropped images. Each cropped image can be used as the input image for the neural network model without requiring large-scale image scaling, which greatly increases the number of sample images available for training the neural network model. Furthermore, it avoids the problem of losing a large amount of image detail features caused by large-scale scaling of sample images, ensuring that most of the detail features in the sample images are preserved.
[0067] For certain sample images that require a large amount of annotation work and are difficult to annotate, the number of annotated images in the dataset is relatively small. For example, taking gear steel metallographic images as an example, gear steel metallographic images are difficult to collect, and the number of gear steel metallographic images that can be collected is relatively small. In addition, the fine-grained distribution of retained austenite in gear steel metallographic images leads to high requirements for annotation accuracy. Manual data annotation is costly and inefficient, resulting in a small number of gear steel metallographic images with retained austenite annotated in the dataset. During the model training process, there may be problems such as insufficient training data for neural network models and insufficient learning of neural networks, which will lead to poor model generalization, low recognition accuracy, and inability to be stably applied to real industrial scenarios.
[0068] Based on this, this application discloses a technical solution that uses an enhanced hybrid matching algorithm to help the neural network generate pseudo-labels from unlabeled image samples, based on the aforementioned image cropping. By using unlabeled images carrying pseudo-labels, the neural network model can learn on its own to improve the performance of semantic segmentation.
[0069] See Figure 4 and Figure 5The training sample image preprocessing method disclosed in this application, after performing data augmentation preprocessing on the cropped image, further includes:
[0070] Step S401: Extract the cropped image corresponding to the unlabeled image from the sample images, and denot it as the unlabeled cropped image. .
[0071] Step S402: Add random noise to the unlabeled cropped image.
[0072] In this step, each unlabeled image is cropped. Add a uniformly distributed random noise to ensure that each image is unique.
[0073] Step S403: Use a semantic segmentation neural network model to predict the unlabeled cropped image after adding random noise, and obtain the pseudo-label and confidence score of the unlabeled cropped image;
[0074] Then, a semantic segmentation neural network model (which can be the neural network model to be trained in this application or other neural network models) is used to predict these unlabeled cropped images with added random noise. Based on the prediction results, the pseudo-labels and confidence scores of the unlabeled cropped images are obtained.
[0075] In this step, to improve the confidence level and the reliability of pseudo-labels in the prediction results of unlabeled cropped images, each unlabeled cropped image can be pre-processed. The process involves copying the images to obtain K (K is a positive integer not less than 2) unlabeled cropped images. Then, uniformly distributed random noise is added to each of the K unlabeled cropped images to obtain K noisy unlabeled cropped images. Then, a semantic segmentation neural network model (such as...) is used. Figure 5 The Trans-UNet backbone network in the image (with added random noise) is used to process K unlabeled cropped images. Make predictions and obtain network prediction results. The network prediction results are processed to obtain these K unlabeled cropped images. Corresponding confidence level and unlabeled cropped images pseudo-tags :
[0076]
[0077] in A three-dimensional matrix exist The value of the position, For the sample exist The predicted entropy of location information can be extracted from the network prediction results. Let be the summation function of a matrix along the m-th dimension. , and These are the mean function, the maximum function, and the logarithmic function, respectively.
[0078] Step S404: Identify unlabeled cropped images with a confidence level lower than the target confidence level, and denot them as the first images to be mixed and cropped.
[0079] In this step, unlabeled cropped images with a confidence level lower than the target confidence level need to be identified as the first images to be mixed and cropped, and they need to be mixed and enhanced in subsequent steps.
[0080] If each unlabeled cropped image is... When copying to obtain K unlabeled cropped images, the first image to be mixed and cropped is an unlabeled cropped image among these K unlabeled cropped images whose confidence level is lower than the target confidence level. That is, obtain K unlabeled cropped images among the unlabeled cropped images whose confidence level is lower than the target confidence level.
[0081] Step S405: The cropped image corresponding to at least a portion of the labeled image in the sample image is denoted as the second image to be mixed and cropped;
[0082] Step S406: Use the second image to be mixed and cropped to perform a mixing enhancement process on the first image to be mixed and cropped;
[0083] Step S407: Obtain the pseudo-tags of the first image to be blended and cropped after the blending enhancement process.
[0084] In steps S405 and S406 above, a hybrid cropping technique is used to enhance the first image to be hybrid cropped. During the enhancement process, such as... Figure 5 As shown, you can choose between CutMix (crop blending enhancement) and MixUp (pixel blending enhancement) schemes to combine a second image to be cropped and blended. The image is cropped, blended, and enhanced with the first image to be cropped; then another image is cropped, blended, and enhanced with the second image. Perform pixel blending enhancement on the first image to be blended and enhanced, after cropping and blending.
[0085] For each unlabeled cropped image In the case of copying and obtaining K unlabeled cropped images, in steps S405 and S406, as follows: Figure 5 As shown, select the corresponding confidence level. Less than the target confidence level Unlabeled cropped images Perform blending enhancement processing with any two cropped images corresponding to labeled images. The One-Hot encoding corresponding to the real label is represented as follows: , The One-Hot encoding corresponding to the real label is as follows: , First of all, will be with Perform the CutMix operation to obtain and its corresponding pseudo-tag Specifically, The middle will randomly sample a box-shaped pixel region, and use The corresponding box-shaped pixel area Perform pixel replacement. pseudo-tags It will also be used in the corresponding frame-shaped pixel area. Authentic Labels The pixel replacement operation described above can be expressed by the following formula:
[0086] ;
[0087] in, express The pixels corresponding to the (i, j) region, for The pseudo-label of the pixel corresponding to (i, j) express The region (i, j) in the middle, express The (i, j) region in the equation. express The label corresponding to the (i, j) region in the text. express The pseudo-labels corresponding to the (i, j) region in the diagram.
[0088] Then Will make peace The MixUp operation produces a blended image with no labels and a cropped image. Unlabeled cropped images Corresponding pseudo tags During MixUp, this invention randomly generates a probability value. After the MixUp operation and its corresponding pseudo-tag It can be obtained through the following formula:
[0089] ;
[0090] express The pixels corresponding to the (i, j) region, express The pixels corresponding to the (i, j) region, express The pseudo-label corresponding to the region (i, j) express The label corresponding to the region (i, j) in the middle.
[0091] Final confidence level Greater than the target confidence level Unlabeled cropped images (Corresponding pseudo-tags are) ) and confidence level Less than the target confidence level Unlabeled cropped images (Corresponding pseudo-tags) The data, along with labeled cropped images and their corresponding ground truth labels, is input into the Trans-UNet backbone network of the neural network model to be trained for segmentation supervision. This enhanced hybrid matching algorithm, based on the original hybrid matching algorithm, utilizes labeled data to increase the confidence of pseudo-labels for unlabeled cropped images with low pseudo-label confidence. In this invention, a CutMix+MixUp fusion data augmentation mechanism is used to add data to unlabeled cropped images with low confidence, further increasing the confidence of their corresponding pseudo-labels.
[0092] This embodiment discloses a neural network model training sample image preprocessing device. For the specific working content of each unit in the device, please refer to the above method embodiment.
[0093] The neural network model training sample image preprocessing device provided in the embodiments of the present invention will be described below. The neural network model training sample image preprocessing device described below can be referred to in correspondence with the neural network model training sample image preprocessing method described above.
[0094] See Figure 6 The neural network model training sample image preprocessing apparatus disclosed in this application includes:
[0095] The sample extraction unit 10, which corresponds to step S101 in the above method, is used to extract sample images;
[0096] The cropping unit 20, which corresponds to step S102 in the above method, is used to perform cropping preprocessing on the sample image using a sliding window to crop the sample image into multiple low-pixel cropped images.
[0097] The first preprocessing unit 30, corresponding to step S103 in the above method, is used to perform data augmentation preprocessing on the cropped image so that the resolution of the cropped image is the target resolution.
[0098] This embodiment discloses an electronic device, which includes at least one processing device and a storage device connected to the processing device, wherein the storage device is used to store computer programs;
[0099] The processing device is used to execute the computer program so that the electronic device can implement any of the above-described training sample image preprocessing methods.
[0100] This embodiment discloses a neural network model, including: a prediction model trained by any of the above-mentioned training sample image preprocessing methods.
[0101] In this embodiment, the neural network model training sample image preprocessing device can also be integrated into the neural network model. In this case, the aforementioned dataset can be directly input into the neural network model, and the neural network model can train and optimize itself.
[0102] To verify the effectiveness of this solution, the applicant trained the neural network model on the training dataset provided in the previous example (containing 352 training images (image resolution 1944x2580), of which 102 were labeled and 250 were unlabeled), and tested it on the test dataset (containing 21 labeled test images). All images are metallographic images of gear steel. The evaluation metrics used in the tests included IoU (Intersection over Union), mIoU (mean Intersection over Union), and OA (Overall Accuracy) for the retained austenite category. The batch size used in this invention was 5, the base learning rate was 0.01, and the maximum number of training iterations was 20.
[0103] This experiment first trained and tested the dataset using the baseline method (labeled sample preprocessing (data augmentation) + Trans-UNet + DiceLoss). Then, it trained and tested using Method 1 (labeled sample preprocessing (sliding window cropping + data augmentation) + Trans-UNet + DiceLoss), Method 2 (labeled sample preprocessing (sliding window cropping + data augmentation) + unlabeled sample preprocessing (sliding window cropping + data augmentation) + Trans-UNet + DiceLoss), and Method 3 (labeled sample preprocessing (sliding window cropping + data augmentation) + unlabeled sample preprocessing (sliding window cropping + data augmentation + enhanced hybrid matching algorithm) + Trans-UNet + DiceLoss). The sliding window cropping refers to processing the samples in step S102, the data augmentation refers to processing the samples in step S103, and the enhanced hybrid matching algorithm refers to processing the samples in steps S401-S407. The evaluation metrics during testing are shown in Table 1.
[0104] Table 1. Evaluation metrics of the baseline method and methods 1-3 on the test set.
[0105] Evaluation indicators Benchmark Method Method 1 Method 2 Method 3 Residual austenite IoU 67.58 69.46 70.08 70.96 mIoU 79.03 80.75 82.01 82.57 OA 92.29 93.75 94.20 94.90
[0106] As shown in Table 1, Method 1's evaluation metrics are higher than the benchmark method, indicating that sliding window cropping can enhance the generalization performance of neural networks by increasing the dataset size and preserving the details of the original images. Method 2's evaluation metrics are higher than Method 1, indicating that neural networks can improve their performance by learning from unlabeled data and their corresponding pseudo-labels. Method 3's evaluation metrics are higher than Method 2, indicating that the enhanced hybrid matching algorithm can further improve the learning ability of neural networks by increasing the confidence of pseudo-labels for unlabeled samples. Overall, Method 3 improves mIoU by 3.54% and OA by 2.61% compared to the benchmark method, demonstrating the effectiveness of the proposed enhanced hybrid matching semi-supervised semantic segmentation algorithm based on sliding window cropping in handling the problem of extracting residual austenite content from the metallographic structure of gear steel. Figures 7-9 The original images, labeled images, and predicted images of samples from three test sets are shown. These images were trained using a neural network model trained with the scheme disclosed in this application. Figure 7 It can be observed that the algorithm proposed in this invention can accurately identify the specific location of retained austenite from the metallographic image of gear steel, thereby obtaining the content ratio of retained austenite in the image.
[0107] For ease of description, the above system is described by dividing it into various modules based on their functions. Of course, in implementing this invention, the functions of each module can be implemented in one or more software and / or hardware components.
[0108] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0109] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0110] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0111] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0112] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for preprocessing training sample images, characterized in that, include: Extract sample images; A sliding window is used to perform cropping preprocessing on the sample image to crop the sample image into multiple low-pixel cropped images; The cropped image undergoes data augmentation preprocessing to achieve a target resolution, which is the resolution of the input image for the neural network model to be trained.
2. The training sample image preprocessing method according to claim 1, characterized in that, The sample image is pre-processed by cropping using a sliding window, including: N horizontal coordinate sampling points are uniformly set along the horizontal coordinate of the sample image, where N is a positive integer not less than 1; M vertical coordinate sampling points are uniformly set along the vertical coordinate of the sample image, where M is a positive integer not less than 1; Based on the horizontal and vertical sampling points, N×M starting points are determined. The sample image is cropped based on each starting point to obtain N×M cropped images. The difference between the resolution of the cropped images and the target resolution is within an allowable range.
3. The training sample image preprocessing method according to any one of claims 1-2, characterized in that, Before performing cropping preprocessing on the sample image using a sliding window, the process also includes: Obtain the resolution of the input image for the neural network model to be trained.
4. The training sample image preprocessing method according to claim 1, characterized in that, Also includes: Extract the cropped images corresponding to the unlabeled images from the sample images, and denote them as unlabeled cropped images; Add random noise to unlabeled cropped images; A semantic segmentation neural network model is used to predict unlabeled cropped images with added random noise, and the pseudo-labels and confidence scores of the unlabeled cropped images are obtained. Unlabeled cropped images with confidence levels lower than the target confidence level are identified as the first images to be mixed and cropped. The cropped image corresponding to at least a portion of the labeled image in the sample image is denoted as the second image to be mixed and cropped. The first image to be mixed and cropped is then subjected to a blending enhancement process using the second image to be mixed and cropped. Obtain the pseudo-tags of the first image to be blended and cropped after the blending enhancement process.
5. The training sample image preprocessing method according to claim 4, characterized in that, After extracting the cropped images corresponding to the unlabeled images in the sample images and denoting them as unlabeled cropped images, before adding random noise to each unlabeled cropped image, the process also includes: Each of the unlabeled cropped images is copied to obtain K copies of the unlabeled cropped images, where K is a positive integer not less than 1; The step of adding random noise to each unlabeled cropped image includes adding random noise to K unlabeled cropped images respectively; The semantic segmentation neural network model is used to predict the pseudo-label and confidence score of each unlabeled cropped image after adding random noise, including: A semantic segmentation neural network model is used to predict K unlabeled cropped images after random noise is added, so as to obtain the confidence score of each unlabeled cropped image and the pseudo-label of the unlabeled cropped image.
6. The training sample image preprocessing method according to claim 5, characterized in that, The process of identifying unlabeled cropped images with a confidence level lower than the target confidence level includes: obtaining K unlabeled cropped images from which the confidence level is lower than the target confidence level; The first image to be mixed and cropped is subjected to a blending enhancement process using the second image to be mixed and cropped, including: A second image to be cropped and blended is cropped and blended with the first image to be cropped and blended for enhancement. Perform pixel blending enhancement on another second image to be blended and cropped, and the first image to be blended and enhanced.
7. The training sample image preprocessing method according to claim 1, characterized in that, The sample image is a metallographic image of gear steel, and the labeled target in the sample image is retained austenite.
8. A device for preprocessing training sample images for a neural network model, characterized in that, include: The sample extraction unit is used to extract sample images; The cropping unit is used to perform cropping preprocessing on the sample image using a sliding window, so as to crop the sample image into multiple low-pixel cropped images; The first preprocessing unit is used to perform data augmentation preprocessing on the cropped image so that the resolution of the cropped image is the target resolution.
9. An electronic device, characterized in that, It includes at least one processing device and a storage device connected to the processing device, wherein: The storage device is used to store computer programs; The processing device is used to execute the computer program so that the electronic device can implement the training sample image preprocessing method as described in any one of claims 1 to 7.
10. A neural network model, characterized in that, include: The neural network model is trained using the training sample image preprocessing method of any one of claims 1-7.