Extension and construction method of noise modified multi-scale machine vision task dataset

Abstract

The present invention provides a method for expanding and constructing a noise-modified multi-scale machine vision task dataset, comprising: randomly acquiring a plurality of images, performing image preprocessing and feature extraction to obtain feature images; constructing corresponding image expansion models based on the image scales and image types of the plurality of images, combining adaptive noise addition and a sliding window algorithm; performing multiple image construction processes on the feature images of the macro image using the image expansion model based on the image type to obtain an expanded macro image; and / or, according to the image expansion model, performing data enhancement on the feature images of the micro image and adding mechanical noise, Gaussian noise, and background noise in preset proportions to generate an expanded micro image; and constructing an expanded macro dataset and / or micro dataset based on the expanded macro image and / or the expanded micro image. The present invention can achieve the expansion and construction of a multi-scale image dataset, thereby improving the authenticity of multi-scale simulated images.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a method for expanding and constructing a noise-modified multi-scale machine vision task data set. Background Art

[0002] Deep learning has made significant progress in machine vision, covering a wide range of scales from macroscopic natural images to microscopic electron microscope images. However, the success of machine vision is highly dependent on the scale and quality of training data.

[0003] Currently, many specific tasks lack widely accepted standard datasets, resulting in limited training data and hindering effective training of deep learning in machine vision. Building sufficiently large datasets often requires collecting data from multiple sources, or even creating data independently. Furthermore, the quality of the data directly impacts the performance of deep learning models.

[0004] Existing methods such as data augmentation and image simulation synthesis increase data diversity and thus data volume through operations such as rotation, flipping, scaling, and cropping. However, these methods have obvious limitations when processing cross-scale data. For example, simple data augmentation techniques may not be able to fully simulate the complex relationships between images of different scales, resulting in a decrease in model performance when faced with images of different scales. At the same time, although image simulation synthesis can generate new samples, these samples may lack the complexity and diversity of real-world images, especially at the microscale, thereby limiting the model's ability to generalize to real situations. For microscopic images, if experimental images are used entirely, the cost is very expensive.

[0005] Therefore, there is an urgent need for a method to expand the multi-scale machine vision task dataset that can expand the data volume, improve data quality, simulate the complex relationships between images of multiple scales, and enhance the model's ability to generalize to real situations. Summary of the Invention

[0006] Based on this, it is necessary to provide a method for expanding and constructing a noise-modified multi-scale machine vision task dataset to address the above technical problems.

[0007] A method for expanding and constructing a noise-modified multi-scale machine vision task dataset comprises the following steps: randomly acquiring a number of images, performing image preprocessing and feature extraction on the images to obtain feature images, the images comprising macro images and / or micro images; constructing corresponding image expansion models based on the image scales and image types of the images in combination with adaptive noise addition and a sliding window algorithm; performing multiple image construction processes on the feature images of the macro images based on the image types of the macro images and according to the image expansion model to obtain expanded macro images; and / or performing data enhancement on the feature images of the micro images and adding mechanical noise, Gaussian noise, and background noise in preset proportions to generate expanded micro images according to the image expansion model; and constructing an expanded macro dataset and / or micro dataset based on the expanded macro images and / or the expanded micro images.

[0008] In one embodiment, the randomly acquiring a number of images includes: randomly collecting a number of outdoor image pairs in an outdoor data set, the outdoor image pairs including clear and fog-free outdoor images and foggy images corresponding to the outdoor images; randomly collecting a number of indoor image pairs in an indoor data set, the indoor image pairs including clear and fog-free indoor images and foggy images corresponding to the indoor images; randomly collecting a number of mixed images in a mixed data set, performing image pair recognition on the mixed images to obtain indoor image pairs, outdoor image pairs and single images, and adding noise to the single image to obtain indoor image pairs and / or outdoor image pairs, the indoor images and outdoor images in the mixed data set being a preset ratio; and obtaining a number of macro images based on the outdoor image pairs and indoor image pairs.

[0009] In one embodiment, the randomly acquiring a plurality of images further includes: obtaining a plurality of electron microscope images of the material and corresponding label images according to an electron microscope experiment; and / or obtaining a structure file of the material, and simulating and generating a plurality of electron microscope images and corresponding label images based on the structure file and electron microscope simulation parameters; wherein, by setting the pixel value of a single atom in the electron microscope image to a two-dimensional Gaussian distribution, the three-dimensional position coordinate information of the atom is read based on the structure file, and a circular convolution method is used to obtain a label image corresponding to the electron microscope image on a two-dimensional plane; based on the plurality of electron microscope images and the corresponding label images, a microscopic image pair is formed to obtain a plurality of microscopic images.

[0010] In one embodiment, based on the image type of the macro image, the feature image of the macro image is subjected to multiple image construction processing according to the image expansion model to obtain an expanded macro image, including: according to the image expansion model, using the fog image simulation method of the atmospheric scattering model, performing multiple image pair construction on the feature image of the outdoor image, and using random noise to perform noise perturbation to obtain an expanded outdoor image; according to the image expansion model, using the fog image simulation method of the FoHIS model, performing multiple image pair construction on the feature image of the indoor image, and using random noise to perform noise perturbation to obtain an expanded indoor image; combining the expanded outdoor image and the expanded indoor image to obtain an expanded macro image.

[0011] In one embodiment, the random noise is obtained based on isotropic Gaussian noise sampling of the haze-free image.

[0012] In one embodiment, the method performs data enhancement on the feature image of the microscopic image according to the image expansion model, and adds mechanical noise, Gaussian noise and background noise in preset proportions to generate an expanded microscopic image, including: performing data enhancement on the feature image of the microscopic image according to the image expansion model to obtain an enhanced image; adding mechanical noise to the enhanced image to obtain a first image, wherein the mechanical noise includes applying stress in the horizontal and vertical directions to simulate sample drift and performing numerical scrolling operations in the horizontal direction to simulate probe jitter; applying Gaussian noise to the first image based on a preset proportion to simulate global noise to obtain a second image; and extracting a low-frequency background signal from a real experimental image using a Fourier filtering method, and adding the low-frequency background signal to the second image based on a preset proportion to generate an expanded microscopic image.

[0013] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows: by randomly acquiring a number of images and performing image preprocessing and feature processing, a feature image is obtained, the number of images include macro images and / or micro images, and based on the image scales and image types of the number of images, a corresponding image expansion model is constructed in combination with an adaptive noise addition and sliding window algorithm, so as to realize customized construction of multi-scale and different types of image models to improve the performance of the model; when performing macro image expansion, based on the image type of the macro image, the feature image of the macro image is subjected to multiple image construction processes according to the image expansion model to obtain an expanded macro image, and a more realistic expanded macro image can be obtained through the division of image types and targeted processing; in the process of expanding the macro image, the image expansion model is used to perform image expansion processing on the feature image of the macro image multiple times to obtain the expanded macro image. When performing microscopic image expansion, data enhancement is performed on the feature images of the microscopic image according to the image expansion model to enhance data diversity, and mechanical noise, Gaussian noise and background noise in preset proportions are added to generate an expanded microscopic image of the microscopic image. Based on the multi-type noise model of the real electron microscope image, a more realistic expanded microscopic image is obtained, which improves the authenticity of the atomic-level simulated image; based on the expanded macroscopic image and / or expanded microscopic image, an expanded macroscopic data set and / or microscopic data set is constructed to achieve data set expansion of multi-scale images, obtain higher quality expanded images, and improve the effectiveness of the expanded images, thereby effectively improving the generalization ability and robustness of the model on images of different scales and meeting the needs of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 1 is a flowchart of a method for expanding and constructing a noise-modified multi-scale machine vision task dataset in one embodiment;

[0015] Figure 2 This is an example of a microscopic scale dataset in one embodiment, wherein (a) is a simulated atomic-resolution STEM image, and (b) is a label map of the STEM image;

[0016] Figure 3 Figure 1 shows a comparison of the noise addition results of an indoor image using the atmospheric scattering model and the FoHIS model in one embodiment, where (a) is a fog-free image; (b) is the noise addition result using the atmospheric scattering model; and (c) is the noise addition result using the FoHIS model.

[0017] Figure 4 The following are foggy images, fog-free images, defogging effects using the atmospheric scattering model and the FoHIS model, and defogging effects using the atmospheric scattering model alone for four cases in an embodiment;

[0018] Figure 51 is the input of the microscopic image model and the corresponding prediction results in one embodiment, wherein (a) and (d) are the model inputs; (b) and (e) are the output results of (a) and (d) after the residual U-type network; (c) and (f) are the results of classifying each atomic column in (b) and (e) using an unsupervised machine learning algorithm. DETAILED DESCRIPTION

[0019] Before describing the specific embodiments of the present invention, the overall concept of the present invention is described as follows:

[0020] This invention is mainly developed based on machine vision. Currently, the amount of available data is limited and the data quality is uneven. Traditional data enhancement methods and image simulation synthesis technologies have limitations in improving model performance, and the model's ability to generalize to real situations is limited.

[0021] Therefore, the present invention proposes a method for expanding and constructing a noise-modified multi-scale machine vision task data set, which obtains feature images by randomly acquiring a number of images and performing image preprocessing and feature processing. The images include macro images and / or micro images, and based on the image scales and image types of the images, a corresponding image expansion model is constructed in combination with an adaptive noise addition and sliding window algorithm to achieve customized construction of multi-scale and different types of image models to improve the performance of the model; when expanding the macro image, based on the image type of the macro image, the feature image of the macro image is subjected to multiple image construction processes according to the image expansion model to obtain an expanded macro image. By dividing the image types and performing targeted processing, a more realistic expansion can be obtained. Macro image; when performing micro image expansion, according to the image expansion model, the feature image of the micro image is enhanced to enhance data diversity, and mechanical noise, Gaussian noise and background noise in preset proportions are added to generate an expanded micro image of the micro image. Based on the multi-type noise model of the real electron microscope image, a more realistic expanded micro image is obtained, which improves the authenticity of the atomic-level simulated image; based on the expanded macro image and / or expanded micro image, an expanded macro data set and / or micro data set is constructed to achieve data set expansion of multi-scale images, obtain higher quality expanded images, and improve the effectiveness of the expanded images, thereby effectively improving the generalization ability and robustness of the model on images of different scales and meeting data processing requirements.

[0022] After introducing the overall concept of the present invention, in order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0023] In one embodiment, Figure 1 As shown in FIG, a method for expanding and constructing a noise-modified multi-scale machine vision task dataset is provided, comprising the following steps:

[0024] Step S101 : randomly acquiring a number of images, performing image preprocessing and feature extraction on the images to obtain feature images, wherein the images include macroscopic images and / or microscopic images.

[0025] Specifically, a number of images are randomly acquired, and the images may be microscopic images and / or macroscopic images, wherein the macroscopic images may be randomly acquired through an existing data set; the microscopic images may be measured images or simulated images. Since the acquisition cost of microscopic measured images is high and the conditions are harsh, an algorithm or third-party software may be used to generate a number of microscopic images through simulation.

[0026] After acquiring multiple images, due to potential inconsistencies in their dimensions and other characteristics, preprocessing is required to ensure image consistency and quality. This includes denoising, contrast adjustment, and size standardization. After preprocessing, the images undergo feature extraction. Macroscopic images can extract global features of the scene, such as the outline and shape of objects; microscopic images can extract local features, such as the microstructure and texture of materials. Feature extraction improves the performance and accuracy of subsequent models, reduces the computational complexity of image processing, and improves image processing efficiency.

[0027] Among them, a number of images are randomly acquired, including: randomly collecting a number of outdoor image pairs in an outdoor data set, the outdoor image pairs include clear and fog-free outdoor images and foggy images corresponding to the outdoor images; randomly collecting a number of indoor image pairs in an indoor data set, the indoor image pairs include clear and fog-free indoor images and foggy images corresponding to the indoor images; randomly collecting a number of mixed images in a mixed data set, performing image pair recognition on the number of mixed images to obtain indoor image pairs, outdoor image pairs and single images, and adding noise to the single image to obtain indoor image pairs and outdoor image pairs, and the indoor images and outdoor images in the mixed data set are in a preset ratio; based on the outdoor image pairs and the indoor image pairs, a number of macro images are obtained.

[0028] Specifically, when acquiring macroscopic images, these images are natural scene images, including both outdoor and indoor images. For outdoor images, several pairs of outdoor images can be randomly collected from a publicly available outdoor image dataset. These pairs consist of a clear, fog-free outdoor image and its corresponding foggy image. For the image dehazing task, for example, 300 pairs of image data, consisting of foggy and fog-free images, can be downloaded directly from an internet dataset as raw data. This makes image acquisition quick and simple.

[0029] When obtaining indoor images, several indoor image pairs are randomly collected from the public indoor image dataset. The indoor image pairs include clear and fog-free indoor images and corresponding foggy images.

[0030] If the dataset contains both outdoor and indoor images, several mixed images are randomly collected in the mixed dataset. The indoor images and outdoor images in the mixed dataset are in a preset ratio, and the ratios of the two are similar, such as 1:1, 1:2, 1:3, 2:1 and 3:1, etc., wherein the mixed images include outdoor image pairs, indoor image pairs and single images, and the single image is denoised to obtain indoor image pairs and / or outdoor image pairs; based on all the obtained indoor image pairs and outdoor image pairs, several macro images are obtained.

[0031] By collecting images from different data sets, we can obtain richer expanded images, which is more conducive to improving the performance of the training model.

[0032] Among them, randomly acquiring a number of images also includes: acquiring a structure file of the material, and simulating and generating a number of electron microscope images and corresponding label maps based on the structure file and electron microscope simulation parameters; and / or obtaining a number of electron microscope images of the material and corresponding label maps according to an electron microscope experiment; wherein, by setting the pixel value of a single atom in the electron microscope image to a two-dimensional Gaussian distribution, based on the structure file, the three-dimensional position coordinate information of the atom is read and obtained, and a circular convolution method is used to obtain a label map corresponding to the electron microscope image on a two-dimensional plane; based on the number of electron microscope images and the corresponding label maps, a microscopic image pair is formed to obtain a number of microscopic images.

[0033] Specifically, when obtaining a microscopic image, a number of electron microscope images of the material and corresponding label images can be obtained based on the electron microscope experiment. The label image is an image corresponding to the electron microscope image, which can reflect the type and content of the elements in the material, as well as the distribution of the elements. Since the acquisition cost of the measured electron microscope image is high, the electron microscope image can be obtained by simulation generation. When simulating the electron microscope image, the electron microscope image is an electron microscope image with atomic resolution. The structure file of the random material is obtained, such as a cif (Crystallographic Information File) file, which contains detailed information of each crystal, such as unit cell parameters, atomic coordinates and literature data, as well as necessary electron microscope simulation parameters; based on the structure file and the electron microscope simulation parameters, the electron microscope image simulation software is used to simulate the electron microscope image under the ideal state, and the electron microscope simulation parameters are adjusted to perform multiple simulations to obtain a number of electron microscope images.

[0034] When simulating the electron microscope image, the corresponding label map needs to be generated at the same time. The pixel value of a single atom in the atomic-level resolution transmission electron microscope image of the two-dimensional material is set to a two-dimensional Gaussian distribution (that is, it can be simplified to a circle). The three-dimensional coordinate information of the atom is read from the cif file. The circular convolution method is used to obtain the label map corresponding to the simulated electron microscope image on the two-dimensional plane, such as Figure 2 As shown, (a) is a simulated atomic-level STEM image, and (b) is a label image corresponding to the atomic-level STEM image, that is, the simulated microscopic image contains the microscopic image and its corresponding label image.

[0035] According to the several electron microscope images and label images obtained by the above two methods, microscopic image pairs are formed, and several microscopic images are obtained, thereby realizing the acquisition of multi-channel microscopic images.

[0036] Step S102 : Based on the image scales and image types of the plurality of images, a corresponding image expansion model is constructed in combination with an adaptive noise addition and a sliding window algorithm.

[0037] Specifically, during model construction, to better expand the dataset for multi-scale images, we built corresponding image expansion models based on the image scales and image types, combining adaptive noise addition and a sliding window algorithm. This allows for customized model construction, building specific models for each image scale and type, implementing targeted noise addition strategies, and achieving more realistic simulated images. Image scales include both macro and micro, and image types target macro images, including both indoor and outdoor.

[0038] When designing noise-addition strategies, for example, macroscopic indoor and outdoor images are affected by factors like atmospheric scattering and pipelines, so we can focus on simulating the effects of atmospheric scattering and light variations on noise. Microscopic images, on the other hand, are primarily affected by sample drift and the microscope's own noise. Therefore, we can design noise-addition strategies to simulate sample drift and probe jitter. By designing different noise-addition strategies for different images, the augmented images generated by the image augmentation model can be made more consistent with the real image, improving the quality of the augmented images.

[0039] Step S103 : Based on the image type of the macro image, the image expansion model is used to perform multiple image construction processes on the feature image of the macro image to obtain an expanded macro image.

[0040] Specifically, after obtaining several macro images, based on the image types of the macro images, which include indoor and outdoor, the macro images are divided into indoor images and outdoor images. According to the corresponding image expansion model, the feature images of the two types of images are subjected to multiple image construction processes to obtain multiple expanded macro images. Since outdoor images may be foggy, targeted processing can be achieved through the division of image types, thereby obtaining higher quality images.

[0041] Among them, step S103 includes: according to the image expansion model, using the fog image simulation method of the atmospheric scattering model, constructing multiple image pairs of the feature image of the outdoor image, and using random noise to add noise disturbance to obtain the expanded outdoor image; according to the image expansion model, using the fog image simulation method of the FoHIS model, constructing multiple image pairs of the feature image of the indoor image, and using random noise to add noise disturbance to obtain the expanded indoor image; combining the expanded outdoor image and the expanded indoor image to obtain the expanded macro image.

[0042] Specifically, since the collected macro images include two types of images, indoor and outdoor, in order to better expand the data set of macro images and obtain higher quality expanded images, models are constructed for indoor images and outdoor images respectively, so as to facilitate the processing of the two images separately and obtain higher quality expanded images.

[0043] For outdoor images, due to the presence of fog, directly adding noise to the image augmentation results in poor results, or even no corresponding augmented image. Therefore, based on the constructed image augmentation model, a fog image simulation method based on the atmospheric scattering model can be used to construct multiple image pairs of the outdoor image's feature images. For example, for each feature image of the original outdoor image, three image pair constructions are performed. During the noise addition, random numbers can be added to generate random noise to achieve noise perturbation, ensuring the diversity of the noise addition results. Through multiple constructions and noise perturbations, multiple augmented outdoor images are obtained.

[0044] For indoor images, the fog simulation effect is poor due to the atmospheric scattering model used. Figure 3As shown, the FoHIS model achieves better indoor image simulation results than the atmospheric scattering model. Therefore, based on the constructed image augmentation model, the FoHIS model's fog image simulation method is used to construct multiple image pairs for the feature images of indoor images, improving the realism of the simulated fog images. For example, for each feature image of the original indoor image, three image pairs are constructed. During noise addition, random numbers are added to generate random noise, achieving noise perturbation and ensuring diversity in the noise addition results. Through multiple image pair constructions and noise perturbation, multiple expanded indoor images are obtained.

[0045] By combining the expanded outdoor image and the expanded indoor image, an expanded macro image that is several times larger than the original image pair is obtained, achieving high-quality expansion of the macro image and obtaining a more effective expanded macro image.

[0046] The random noise is obtained by sampling the isotropic Gaussian noise of the haze-free image.

[0047] Specifically, within the model, the original haze-free image signal will be corrupted by multi-step, small amounts of Gaussian noise during the diffusion process, eventually becoming isotropic Gaussian noise. Therefore, during the dataset generation process, the model will sample a random noise from the above isotropic Gaussian noise to generate diverse augmented images.

[0048] And / or, step S104 , performing data enhancement on the feature image of the microscopic image according to the image expansion model, and adding mechanical noise, Gaussian noise and background noise in preset proportions to generate an expanded microscopic image.

[0049] Specifically, since traditional data enhancement methods, such as rotation, flipping, scaling and cropping, have limited ability to enhance data sets, after simulating and generating several microscopic images, the aforementioned methods can be used to perform data enhancement on the microscopic images to increase the diversity of the images; and according to the image expansion model, mechanical noise, Gaussian noise and background noise in preset proportions are added to the feature images of several microscopic images. The preset proportions can be adjusted accordingly according to actual needs to generate expanded microscopic images of several microscopic images, thereby achieving a more realistic simulation of the microscopic images and optimizing the quality of the expanded images.

[0050] Among them, since atomic-level microscopic images, such as electron microscope samples in experimental electron microscope images, are contaminated by spontaneous adsorption of hydrocarbons on the sample surface indoors, resulting in drastic changes in the contrast of some areas of the microscopic image, such as some areas are extremely bright and some areas are extremely dark, pollution simulation can be performed by adding background noise to make the obtained expanded microscopic image more in line with the actual situation, thereby achieving more realistic image expansion of the microscopic image and improving image quality.

[0051] Among them, step S104 includes: according to the image expansion model, data enhancement is performed on the characteristic image of the microscopic image to obtain an enhanced image; mechanical noise is added to the enhanced image to obtain a first image, and the mechanical noise includes applying stress in the horizontal and vertical directions to simulate sample drift and performing numerical scrolling operations in the horizontal direction to simulate probe jitter; based on a preset ratio, Gaussian noise is applied to the first image to simulate global noise to obtain a second image; a low-frequency background signal is extracted from the real experimental image using a Fourier filtering method, and is added to the second image based on a preset ratio to generate an expanded microscopic image.

[0052] Specifically, the feature image of the microscopic image is input into the corresponding image expansion model, and data enhancement operations such as rotation, flipping, and scaling are performed on it to obtain an enhanced image; in order to simulate the electron microscope image to better represent the real electron microscope image, it is also necessary to add three types of noise to several microscopic images according to the image expansion model, including mechanical noise, Gaussian noise and background noise of the real electron microscope.

[0053] When adding mechanical noise to the enhanced image, stress is applied in the horizontal and vertical directions to simulate sample drift in electron microscopy experiments, and the image is numerically scrolled in the horizontal direction to realize the influence of probe jitter under experimental conditions on the final imaging results, thereby obtaining the first image.

[0054] Due to the background noise of the electron source and detector, the statistical noise of the detector, the instability of the electron beam, the interference of the sample and the environment, and the randomness of the interaction between the electron beam and the sample, Gaussian noise will appear in the microscopic image. Therefore, after the mechanical noise is added, Gaussian noise needs to be applied to the first image according to a preset ratio to obtain a second image. The degree of added noise is controlled by adjusting the size of the standard deviation of the Gaussian distribution. The larger the standard deviation, the more noise is added, thereby achieving the simulation of global noise.

[0055] After adding Gaussian noise, background noise needs to be applied to the second image based on a preset ratio to generate an expanded microscopic image. When adding background noise, the random brightness contrast interface can be used to manipulate contrast to simulate contrast changes caused by hydrocarbon adsorption. Alternatively, a Fourier filter method can be used to extract a low-frequency background signal from a real experimental image and add it to the microscopic image to achieve a more realistic simulation of sample contamination.

[0056] By adding the above three types of noise, we can obtain an expanded microscopic image that is closer to reality and achieve accurate simulation of atomic-level microscopic images, so as to improve the model's generalization ability to real situations.

[0057] Step S105 : constructing an expanded macroscopic data set and / or microscopic data set based on the expanded macroscopic image and / or the expanded microscopic image.

[0058] Specifically, an expanded macro dataset is constructed based on the expanded macro image, and an expanded micro dataset is constructed based on the expanded micro image. If the acquired image contains both macro and micro images, a multi-scale dataset can be constructed based on the expanded images at multiple scales, thereby realizing the expansion of image datasets at multiple scales from macro to atomic level and micro, as well as the realistic simulation of the complex relationships between images at different scales, improving the image quality in the dataset, thereby helping to improve model performance and meet data processing requirements.

[0059] In this embodiment, a number of images are randomly acquired and image preprocessing and feature processing are performed to obtain feature images. The number of images include macro images and / or micro images. Based on the image scales and image types of the number of images, a corresponding image expansion model is constructed in combination with an adaptive noise addition and sliding window algorithm to achieve customized construction of multi-scale and different type image models to improve the performance of the model. When expanding a macro image, based on the image type of the macro image, the feature image of the macro image is subjected to multiple image construction processes according to the image expansion model to obtain an expanded macro image. By dividing the image types and performing targeted processing, a more realistic expanded macro image can be obtained. When expanding a micro image, According to the image expansion model, the feature image of the microscopic image is enhanced to enhance the data diversity, and mechanical noise, Gaussian noise and background noise in preset proportions are added to generate an expanded microscopic image of the microscopic image. Based on the multi-type noise model of the real electron microscope image, a more realistic expanded microscopic image is obtained, which improves the authenticity of the atomic-level simulation image; based on the expanded macroscopic image and / or expanded microscopic image, an expanded macroscopic data set and / or microscopic data set is constructed to achieve data set expansion of multi-scale images, obtain higher quality expanded images, and improve the effectiveness of the expanded images, thereby effectively improving the generalization ability and robustness of the model on images of different scales and meeting the needs of data processing.

[0060] The above method is also applicable to the processing of mesoscopic images and the construction of datasets, thereby expanding the dataset of multi-scale images.

[0061] In one embodiment, to verify the effectiveness of the expanded macro dataset, an open-source deep learning model for image dehazing based on a diffusion model and frequency compensation was selected as a test model. To unify the training image scale and improve training speed and time, the corresponding hazy, fog-free, and depth maps in the prepared dataset were resized to 128 pixels wide and 128 pixels wide using image processing software. The order of all data points in the dataset was randomly shuffled to mitigate overfitting in the deep learning model.

[0062] Within the model, the original haze-free image signal is corrupted by multiple, small amounts of Gaussian noise during the diffusion process, ultimately transforming it into isotropic Gaussian noise. During the generation process, the model samples a random noise from the aforementioned isotropic Gaussian noise. Then, using the haze-containing image from the data pair as the generation condition, the model simultaneously feeds the conditional haze image and the sampled noise into the model. This process is repeated to ultimately produce a dehazed image.

[0063] like Figure 4 As shown in the figure, using the expanded macro dataset, the image augmentation model combines the atmospheric scattering model (ASM) and the FoHIS model to process indoor and outdoor images separately. Cases 1 and 2 are outdoor images processed using the ASM model, while cases 3 and 4 are indoor images processed using the FoHIS model. As a result, the trained deep learning model achieves better dehazing results in both dense and light fog, and in both indoor and outdoor scenes, compared to data without indoor and outdoor classification and using the atmospheric scattering model alone.

[0064] In one embodiment, in order to test the expanded microscopic dataset, a residual U-shaped network model with 4 times maximum pooling downsampling is used. The spatial dimension of the feature map is gradually reduced by 4 times of downsampling while keeping the depth of the feature increased, which helps the network capture feature information of different scales. In the downsampling process, a pooling layer is used to reduce the resolution of the feature map while maintaining the transmission of important information. At the same time, in order to ensure the gradient backpropagation in the deep neural network, a skip connection is introduced between the encoder output features and the decoder input features of the same size. The model takes minimizing the binary cross entropy loss (BCE Loss) as the optimization goal, and continuously adjusts and optimizes the network weights by introducing a learning rate scheduler and combining it with the Adam optimizer to adaptively adjust the learning rate.

[0065] After model training, performance was evaluated on the validation set, using the Dice coefficient and BCE Loss (binary cross entropy loss) as evaluation criteria to monitor model training progress. Finally, the model trained on the constructed dataset achieved Dice coefficients of 0.978 and 0.976 on the training and validation sets, respectively, and BCE Loss of 0.013 and 0.012.

[0066] In order to verify the effect of the model trained with the expanded microscopic dataset, zeolite ZSM-5 (see Figure 5 first row) and transition metal sulfide WS2 (see Figure 5 Scanning transmission electron microscopy (STEM) images of the second row) served as test cases for the model.

[0067] First, the original STEM image ( Figure 5 (a) and (d)) are used as the input of the model, and the output of the residual U-shaped network is obtained Figure 5 (b) and (e). Then, a spot detection algorithm is used to determine the atomic column coordinates, radius and other information from the model output; based on the above information, physical information such as the peak value, mean value, and local crystal structure of each atomic column are extracted from the input and output images of the model. Finally, using this physical information, an unsupervised machine learning algorithm is used to classify the elements of each atomic column. The results are shown in Figure 5 (c) and (f). Obviously, the model trained with the expanded microscopic dataset can achieve a realistic simulation of microscopic images.

[0068] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0069] Obviously, those skilled in the art should understand that the modules or steps of the present invention described above can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices. Alternatively, they can be implemented using program codes executable by the computing device, so that they can be stored in a computer storage medium (ROM / RAM, magnetic disk, optical disk) and executed by the computing device. In some cases, the steps shown or described can be performed in a different order than herein, or they can be made into individual integrated circuit modules, or multiple modules or steps can be made into a single integrated circuit module for implementation. Therefore, the present invention is not limited to any specific combination of hardware and software.

[0070] The above content is a further detailed description of the present invention in conjunction with specific embodiments, and the specific implementation of the present invention cannot be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for expanding and constructing a noise-modified multi-scale machine vision task dataset, characterized in that: The following steps are involved: Randomly acquiring a number of images, performing image preprocessing and feature extraction on the images to obtain feature images, the images including macro images and micro images, the macro images including outdoor images and indoor images; Based on the image scales and image types of the plurality of images, a corresponding image expansion model is constructed by combining an adaptive noise addition and a sliding window algorithm; Based on the image type of the macro image, and according to the image expansion model, performing multiple image construction processes on the feature image of the macro image to obtain an expanded macro image, including: according to the image expansion model, using a fog image simulation method of an atmospheric scattering model, performing multiple image pair constructions on the feature image of the outdoor image, and using random noise to perform noise perturbation to obtain an expanded outdoor image; according to the image expansion model, using a fog image simulation method of a FoHIS model, performing multiple image pair constructions on the feature image of the indoor image, and using random noise to perform noise perturbation to obtain an expanded indoor image, wherein the random noise is obtained by sampling isotropic Gaussian noise of a fog-free image; and combining the expanded outdoor image and the expanded indoor image to obtain an expanded macro image; According to the image expansion model, data enhancement is performed on the feature image of the microscopic image, and mechanical noise, Gaussian noise and background noise are added in preset proportions to generate an expanded microscopic image, including: according to the image expansion model, data enhancement is performed on the feature image of the microscopic image to obtain an enhanced image; mechanical noise is added to the enhanced image to obtain a first image, wherein the mechanical noise includes applying stress in the horizontal and vertical directions to simulate sample drift and performing a numerical scrolling operation in the horizontal direction to simulate probe jitter; Gaussian noise is applied to the first image based on a preset proportion to simulate global noise to obtain a second image; a low-frequency background signal is extracted from a real experimental image using a Fourier filtering method, and is added to the second image based on a preset proportion to generate an expanded microscopic image; Based on the expanded macroscopic image and the expanded microscopic image, an expanded macroscopic data set and an expanded microscopic data set are constructed.

2. The method for expanding and constructing a noise-modified multi-scale machine vision task dataset according to claim 1 is characterized in that: The randomly acquiring a plurality of images comprises: Randomly collect a number of outdoor image pairs from an outdoor dataset, where the outdoor image pairs include a clear, fog-free outdoor image and a foggy image corresponding to the outdoor image; Randomly collect a number of indoor image pairs in the indoor dataset, where the indoor image pairs include a clear, fog-free indoor image and a foggy image corresponding to the indoor image; Randomly collecting a number of mixed images from a mixed data set, performing image pair recognition on the number of mixed images to obtain indoor image pairs, outdoor image pairs, and single images, and adding noise to the single images to obtain indoor image pairs and / or outdoor image pairs, wherein the indoor images and outdoor images in the mixed data set are in a preset ratio; Based on the outdoor image pair and the indoor image pair, several macro images are obtained.

3. The method for expanding and constructing a noise-modified multi-scale machine vision task dataset according to claim 1, characterized in that: The randomly acquiring a plurality of images further comprises: Obtain several electron microscope images of the material and corresponding label images based on electron microscope experiments; and / or, obtaining a structural file of the material, and simulating and generating a plurality of electron microscope images and corresponding label images based on the structural file and electron microscope simulation parameters; The method comprises setting the pixel value of a single atom in the electron microscope image to a two-dimensional Gaussian distribution, reading the three-dimensional position coordinate information of the atom based on the structure file, and using a circular convolution method to obtain a label map corresponding to the electron microscope image on a two-dimensional plane; According to the plurality of electron microscope images and the corresponding label images, microscopic image pairs are formed to obtain a plurality of microscopic images.

Images