Expansion and construction method of noise-modified multi-scale machine vision task data set

Abstract

The invention provides a method for expanding and constructing a noise-modified multi-scale machine vision task data set, and the method comprises the steps: randomly obtaining a plurality of images, carrying out the image preprocessing and feature extraction, and obtaining a feature image; based on the image scales and the image types of the plurality of images, constructing a corresponding image expansion model in combination with an adaptive noise adding and sliding window algorithm; based on the image type, an image expansion model is adopted to carry out multiple times of image construction processing on the feature image of the macroscopic image, and an expanded macroscopic image is obtained; and / or, according to the image expansion model, performing data enhancement on the feature image of the microscopic image, and adding a preset proportion of mechanical noise, Gaussian noise and background noise to generate an expanded microscopic image; and based on the expanded macroscopic image and / or the expanded microscopic image, constructing an expanded macroscopic data set and / or microscopic data set. According to the method, the data set expansion and construction of the multi-scale image can be realized, and the authenticity of the multi-scale analog image is improved.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a method for expanding and constructing a noise-modified multi-scale machine vision task data set. Background Art

[0002] In the field of machine vision, deep learning technology has made remarkable progress, covering multiple scales from macroscopic natural images to microscopic electron microscope images. However, the success of machine vision is highly dependent on the scale and quality of training data.

[0003] Currently, many specific tasks lack widely recognized standard datasets, resulting in limited available training data and inability to effectively train machine vision deep learning. In order to build a dataset with a large enough data volume, it is usually necessary to collect datasets from multiple parties or even create data independently. In addition, the quality of the data will directly affect the performance of the deep learning model.

[0004] Existing methods such as data augmentation and image simulation synthesis increase data diversity and thus data volume through operations such as rotation, flipping, scaling, and cropping. However, these methods have obvious limitations when processing cross-scale data. For example, simple data augmentation techniques may not be able to fully simulate the complex relationship between images of different scales, resulting in a decrease in the performance of the model when faced with images of different scales. At the same time, although image simulation synthesis can generate new samples, these samples may lack the complexity and diversity of real-world images, especially at the microscopic scale, thus limiting the model's ability to generalize to real situations. For microscopic images, if experimental images are used entirely, the cost is very expensive.

[0005] Therefore, there is an urgent need for a method to expand the multi-scale machine vision task dataset that can expand the data volume, improve data quality, simulate the complex relationships of images of multiple scales, and enhance the model's ability to generalize to real situations. Summary of the invention

[0006] Based on this, it is necessary to provide a method for expanding and constructing a noise-modified multi-scale machine vision task dataset to address the above technical problems.

[0007] A method for expanding and constructing a noise-modified multi-scale machine vision task data set comprises the following steps: randomly acquiring a number of images, performing image preprocessing and feature extraction on the images to obtain feature images, wherein the images include macro images and / or micro images; constructing a corresponding image expansion model based on the image scales and image types of the images in combination with an adaptive noise addition and a sliding window algorithm; performing multiple image construction processes on the feature images of the macro images based on the image types of the macro images according to the image expansion model to obtain an expanded macro image; and / or, performing data enhancement on the feature images of the micro images according to the image expansion model, and adding mechanical noise, Gaussian noise and background noise in preset proportions to generate an expanded micro image; and constructing an expanded macro data set and / or micro data set based on the expanded macro image and / or the expanded micro image.

[0008] In one embodiment, the randomly acquiring a number of images includes: randomly collecting a number of outdoor image pairs in an outdoor data set, the outdoor image pairs including clear and fog-free outdoor images and foggy images corresponding to the outdoor images; randomly collecting a number of indoor image pairs in an indoor data set, the indoor image pairs including clear and fog-free indoor images and foggy images corresponding to the indoor images; randomly collecting a number of mixed images in a mixed data set, performing image pair recognition on the number of mixed images to obtain indoor image pairs, outdoor image pairs and single images, and adding noise to the single image to obtain indoor image pairs and / or outdoor image pairs, the indoor images and outdoor images in the mixed data set being in a preset ratio; and obtaining a number of macro images based on the outdoor image pairs and the indoor image pairs.

[0009] In one embodiment, the randomly acquiring a number of images also includes: obtaining a number of electron microscope images of the material and corresponding label images according to the electron microscope experiment; and / or obtaining the structure file of the material, and simulating and generating a number of electron microscope images and corresponding label images based on the structure file and electron microscope simulation parameters; wherein, by setting the pixel value of a single atom in the electron microscope image to a two-dimensional Gaussian distribution, based on the structure file, the three-dimensional position coordinate information of the atom is read and obtained, and a circular convolution method is used to obtain a label image corresponding to the electron microscope image on a two-dimensional plane; based on the several electron microscope images and the corresponding label images, a microscopic image pair is formed to obtain a number of microscopic images.

[0010] In one of the embodiments, based on the image type of the macro image, according to the image expansion model, the feature image of the macro image is subjected to multiple image construction processing to obtain an expanded macro image, including: according to the image expansion model, using the fog image simulation method of the atmospheric scattering model, the feature image of the outdoor image is subjected to multiple image pair construction, and random noise is used to perform noise perturbation to obtain an expanded outdoor image; according to the image expansion model, using the fog image simulation method of the FoHIS model, the feature image of the indoor image is subjected to multiple image pair construction, and random noise is used to perform noise perturbation to obtain an expanded indoor image; combining the expanded outdoor image and the expanded indoor image to obtain an expanded macro image.

[0011] In one of the embodiments, the random noise is obtained based on isotropic Gaussian noise sampling of the haze-free image.

[0012] In one of the embodiments, the feature image of the microscopic image is data enhanced according to the image expansion model, and mechanical noise, Gaussian noise and background noise are added in preset proportions to generate an expanded microscopic image, including: according to the image expansion model, the feature image of the microscopic image is data enhanced to obtain an enhanced image; mechanical noise is added to the enhanced image to obtain a first image, wherein the mechanical noise includes applying stress in the horizontal and vertical directions to simulate sample drift and performing numerical scrolling operations in the horizontal direction to simulate probe jitter; based on a preset proportion, Gaussian noise is applied to the first image to simulate global noise to obtain a second image; a low-frequency background signal is extracted from a real experimental image using a Fourier filtering method, and is added to the second image based on a preset proportion to generate an expanded microscopic image.

[0013] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows: by randomly acquiring a number of images and performing image preprocessing and feature processing to obtain feature images, the number of images include macro images and / or micro images, and based on the image scales and image types of the number of images, a corresponding image expansion model is constructed in combination with an adaptive noise addition and sliding window algorithm, so as to realize customized construction of multi-scale and different types of image models to improve the performance of the model; when performing macro image expansion, based on the image type of the macro image, the feature image of the macro image is subjected to multiple image construction processing according to the image expansion model to obtain an expanded macro image, and a more realistic expanded macro image can be obtained through the division of image types and targeted processing; in the process of When performing microscopic image expansion, data enhancement is performed on the feature images of the microscopic images according to the image expansion model to enhance data diversity, and mechanical noise, Gaussian noise and background noise in preset proportions are added to generate an expanded microscopic image of the microscopic image. Based on the multi-type noise model of the real electron microscope image, a more realistic expanded microscopic image is obtained, which improves the authenticity of the atomic-level simulated image. Based on the expanded macroscopic image and / or expanded microscopic image, an expanded macroscopic data set and / or microscopic data set is constructed to achieve data set expansion of multi-scale images, obtain higher quality expanded images, and improve the effectiveness of the expanded images, thereby effectively improving the generalization ability and robustness of the model on images of different scales to meet data processing needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A flowchart of a method for expanding and constructing a noise-modified multi-scale machine vision task data set in one embodiment;

[0015] Figure 2 This is an example of a microscopic scale data set in an embodiment, wherein (a) is a simulated atomic-resolution STEM image, and (b) is a label map of the STEM image;

[0016] Figure 3 The following is a comparison diagram of the noise addition of indoor images using the atmospheric scattering model and the FoHIS model in one embodiment, wherein (a) is a fog-free image; (b) is the noise addition result of the atmospheric scattering model; and (c) is the noise addition result of the FoHIS model.

[0017] Figure 4 The foggy images, fog-free images, defogging effects using the atmospheric scattering model and the FoHIS model, and defogging effects using the atmospheric scattering model alone are shown in four cases in an embodiment;

[0018] Figure 51 is the input of the microscopic image model in an embodiment and the corresponding prediction results, wherein (a) and (d) are the model inputs; (b) and (e) are the output results of (a) and (d) after the residual U-type network; (c) and (f) are the results of classifying each atomic column in (b) and (e) using an unsupervised machine learning algorithm. DETAILED DESCRIPTION

[0019] Before describing the specific embodiments of the present invention, the overall concept of the present invention is described as follows:

[0020] The present invention is mainly developed based on machine vision. Currently, the amount of available data is limited and the data quality is uneven. Traditional data enhancement methods and image simulation synthesis technologies have limitations in improving model performance, and the model's generalization ability to real situations is limited.

[0021] Therefore, the present invention proposes a method for expanding and constructing a noise-modified multi-scale machine vision task data set, by randomly acquiring a number of images, and performing image preprocessing and feature processing to obtain feature images, the number of images including macro images and / or micro images, and based on the image scales and image types of the number of images, combining adaptive noise addition and sliding window algorithms to construct corresponding image expansion models, to achieve customized construction of multi-scale and different types of image models, so as to improve the performance of the model; when performing macro image expansion, based on the image type of the macro image, according to the image expansion model, the feature image of the macro image is subjected to multiple image construction processes to obtain an expanded macro image, and through the division of image types and targeted processing, a more realistic expansion can be obtained. Macro image; when expanding the micro image, according to the image expansion model, the feature image of the micro image is enhanced to enhance the data diversity, and mechanical noise, Gaussian noise and background noise in preset proportions are added to generate an expanded micro image of the micro image. Based on the multi-type noise model of the real electron microscope image, a more realistic expanded micro image is obtained, which improves the authenticity of the atomic-level simulation image; based on the expanded macro image and / or the expanded micro image, an expanded macro data set and / or micro data set is constructed to achieve data set expansion of multi-scale images, obtain a higher quality expanded image, and improve the effectiveness of the expanded image, thereby effectively improving the generalization ability and robustness of the model on images of different scales to meet the needs of data processing.

[0022] After introducing the overall concept of the present invention, in order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail by specific implementation methods in combination with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0023] In one embodiment, Figure 1 As shown, a method for expanding and constructing a noise-modified multi-scale machine vision task dataset is provided, comprising the following steps:

[0024] Step S101, randomly acquiring a number of images, performing image preprocessing and feature extraction on the number of images to obtain feature images, the number of images including macroscopic images and / or microscopic images.

[0025] Specifically, a number of images are randomly acquired, and the images may be microscopic images and / or macroscopic images, wherein the macroscopic images may be randomly acquired through an existing data set; the microscopic images may be measured images or simulated images. Since the acquisition cost of microscopic measured images is high and the conditions are harsh, an algorithm or third-party software may be used to generate a number of microscopic images through simulation.

[0026] After acquiring several images, since the size and other characteristics of several images may be inconsistent, the acquired images need to be preprocessed, such as denoising, contrast adjustment, size standardization, etc., to ensure the consistency and quality of the images. After preprocessing, the images need to be subjected to feature extraction operations. Macro images can extract global features of the scene, such as the outline and shape of objects; micro images can extract local features, such as the microstructure and texture of materials. Feature extraction can improve the performance and accuracy of subsequent models, reduce the amount of computational effort in image processing, and improve image processing efficiency.

[0027] Among them, a number of images are randomly acquired, including: randomly collecting a number of outdoor image pairs in an outdoor data set, the outdoor image pairs include clear fog-free outdoor images and foggy images corresponding to the outdoor images; randomly collecting a number of indoor image pairs in an indoor data set, the indoor image pairs include clear fog-free indoor images and foggy images corresponding to the indoor images; randomly collecting a number of mixed images in a mixed data set, performing image pair recognition on the mixed images to obtain indoor image pairs, outdoor image pairs and single images, and adding noise to the single image to obtain indoor image pairs and outdoor image pairs, and the indoor images and outdoor images in the mixed data set are in a preset ratio; based on the outdoor image pairs and the indoor image pairs, a number of macro images are obtained.

[0028] Specifically, when acquiring macro images, the macro images are natural scene images, including outdoor images and indoor images; when acquiring outdoor images, several outdoor image pairs can be randomly collected from the public outdoor image dataset, and the outdoor image pairs include a clear and fog-free outdoor image and its corresponding foggy image. Taking the image defogging task as an example, 300 pairs of image data, including foggy images and corresponding fog-free images, can be directly downloaded from the Internet dataset as raw data, and the image acquisition method is simple and fast.

[0029] When obtaining indoor images, several indoor image pairs are randomly collected from a public indoor image dataset, and the indoor image pairs include clear and fog-free indoor images and corresponding foggy images.

[0030] If the data set contains both outdoor images and indoor images, a number of mixed images are randomly collected in the mixed data set. The indoor images and outdoor images in the mixed data set are in a preset ratio, and the ratios of the two are similar, such as 1:1, 1:2, 1:3, 2:1 and 3:1, etc., wherein the mixed images include outdoor image pairs, indoor image pairs and single images, and the single image is denoised to obtain indoor image pairs and / or outdoor image pairs; based on all the obtained indoor image pairs and outdoor image pairs, a number of macro images are obtained.

[0031] By collecting images from different data sets, we can obtain richer expanded images, which is more conducive to improving the performance of the training model.

[0032] Among them, randomly acquiring a number of images also includes: acquiring a structure file of the material, and based on the structure file and electron microscope simulation parameters, simulating and generating a number of electron microscope images and corresponding label images; and / or, obtaining a number of electron microscope images of the material and corresponding label images according to an electron microscope experiment; wherein, by setting the pixel value of a single atom in the electron microscope image to a two-dimensional Gaussian distribution, based on the structure file, reading and obtaining the three-dimensional position coordinate information of the atom, and using a circular convolution method to obtain a label image corresponding to the electron microscope image on a two-dimensional plane; based on the number of electron microscope images and the corresponding label images, forming a microscopic image pair to obtain a number of microscopic images.

[0033] Specifically, when obtaining a microscopic image, several electron microscope images of the material and corresponding label images can be obtained according to the electron microscope experiment. The label image is an image corresponding to the electron microscope image, which can reflect the element type and content of the material, as well as the distribution of the elements. Since the acquisition cost of the measured electron microscope image is high, the electron microscope image can be obtained by simulation generation. When simulating the electron microscope image, the electron microscope image is an electron microscope image with atomic resolution. The structure file of the random material is obtained, such as a cif (Crystallographic Information File) file, which contains detailed information of each crystal, such as unit cell parameters, atomic coordinates and literature data, etc., as well as necessary electron microscope simulation parameters; based on the structure file and the electron microscope simulation parameters, the electron microscope image simulation software is used to simulate the electron microscope image under the ideal state, and the electron microscope simulation parameters are adjusted for multiple simulations to obtain several electron microscope images.

[0034] When simulating the electron microscope image, the corresponding label map needs to be generated at the same time. The pixel value of a single atom in the atomic-resolution transmission electron microscope image of the two-dimensional material is set to a two-dimensional Gaussian distribution (that is, it can be simplified to a circle). The three-dimensional coordinate information of the atom is read from the cif file. The circular convolution method is used to obtain the label map corresponding to the simulated electron microscope image on the two-dimensional plane, such as Figure 2 As shown, (a) is a simulated atomic-level STEM image, and (b) is a label image corresponding to the atomic-level STEM image, that is, the simulated microscopic image contains the microscopic image and its corresponding label image.

[0035] According to the several electron microscope images and label images obtained by the above two methods, a microscopic image pair is formed, and several microscopic images are obtained, thereby realizing the acquisition of multi-channel microscopic images.

[0036] Step S102, based on the image scales and image types of the plurality of images, a corresponding image expansion model is constructed in combination with an adaptive noise addition and a sliding window algorithm.

[0037] Specifically, when constructing the model, in order to better expand the data set of multi-scale images, based on the image scales and image types of several images, the corresponding image expansion model is constructed in combination with the adaptive noise addition and sliding window algorithm to achieve customized construction of the model, that is, to build a specific model for each scale and type of image type, implement a targeted noise addition strategy, and obtain a more realistic simulated image. Among them, the image scale includes macro and micro, and the image type is for macro images, including indoor and outdoor.

[0038] When designing a noise addition strategy, for example, since macroscopic indoor and outdoor images are affected by factors such as atmospheric scattering and pipelines, we can focus on simulating the impact of atmospheric scattering and light changes on noise; while microscopic images are mainly affected by sample drift and the noise of the electron microscope itself, so we can design the noise impact of microscopic images from the direction of simulating sample drift and probe jitter. By designing different noise addition strategies for different images, the expanded images generated by the image expansion model can be more consistent with the real image, improving the quality of the expanded images.

[0039] Step S103 , based on the image type of the macro image, perform multiple image construction processes on the feature image of the macro image according to the image expansion model to obtain an expanded macro image.

[0040] Specifically, after obtaining several macro images, based on the image types of the macro images, which include indoor and outdoor, the several macro images are divided into indoor images and outdoor images. According to the corresponding image expansion model, the feature images of the two types of images are subjected to multiple image construction processes to obtain multiple expanded macro images. Since outdoor images may be foggy, targeted processing can be achieved through the division of image types, thereby obtaining higher quality images.

[0041] Among them, step S103 includes: according to the image expansion model, using the fog image simulation method of the atmospheric scattering model, constructing multiple image pairs of feature images of the outdoor image, and using random noise to add noise disturbance to obtain the expanded outdoor image; according to the image expansion model, using the fog image simulation method of the FoHIS model, constructing multiple image pairs of feature images of the indoor image, and using random noise to add noise disturbance to obtain the expanded indoor image; combining the expanded outdoor image and the expanded indoor image to obtain the expanded macro image.

[0042] Specifically, since the collected macro images include two types of images, indoor and outdoor, in order to better expand the data set of macro images and obtain higher quality expanded images, models are constructed for indoor images and outdoor images respectively, so as to process the two images separately and obtain higher quality expanded images.

[0043] For outdoor images, since there is fog in outdoor images, adding noise directly will result in poor image expansion effect, or even fail to obtain the corresponding expanded image. Therefore, the constructed image expansion model can be used to simulate the fog image of the atmospheric scattering model to construct multiple image pairs for the feature image of the outdoor image. For example, for each feature image of the original outdoor image, three image pairs are constructed. When adding noise, random noise can be obtained by adding random numbers to achieve noise disturbance, ensuring the diversity of the noise results. Multiple expanded outdoor images can be obtained through multiple constructions and noise disturbances.

[0044] For indoor images, the effect of fog simulation using atmospheric scattering model is poor. Figure 3As shown in the figure, the indoor image simulation using the FoHIS model has a better simulation effect than the atmospheric scattering model. Therefore, according to the constructed image expansion model, the fog image simulation method of the FoHIS model is used to construct multiple image pairs for the feature images of the indoor images, thereby improving the authenticity of the simulated fog image. For example, for each feature image of the original indoor image, three image pairs are constructed. When adding noise, random noise can be obtained by adding random numbers to achieve noise disturbance, thereby ensuring the diversity of the noise results. Multiple expanded indoor images are obtained through multiple image pair constructions and noise disturbances.

[0045] By combining the expanded outdoor image and the expanded indoor image, an expanded macro image several times larger than the original image pair is obtained, thereby achieving high-quality expansion of the macro image and obtaining a more effective expanded macro image.

[0046] The random noise is obtained by sampling the isotropic Gaussian noise of the haze-free image.

[0047] Specifically, within the model, the original haze-free image signal will be destroyed by multi-step, small amounts of Gaussian noise during the diffusion process and eventually become isotropic Gaussian noise. Therefore, during the generation of the dataset, the model will sample a random noise from the above isotropic Gaussian noise to generate a variety of expanded images.

[0048] And / or, step S104, performing data enhancement on the feature image of the microscopic image according to the image expansion model, and adding mechanical noise, Gaussian noise and background noise in preset proportions to generate an expanded microscopic image.

[0049] Specifically, since traditional data enhancement methods, such as rotation, flipping, scaling and cropping, have limited ability to enhance data sets, after simulating and generating several microscopic images, the microscopic images can be enhanced using the aforementioned method to increase the diversity of the images; and according to the image expansion model, mechanical noise, Gaussian noise and background noise in preset proportions are added to the feature images of several microscopic images. The preset proportions can be adjusted accordingly according to actual needs to generate expanded microscopic images of several microscopic images, thereby achieving a more realistic simulation of microscopic images and optimizing the quality of the expanded images.

[0050] Among them, since atomic-level microscopic images, such as electron microscope samples in experimental electron microscope images, are contaminated by spontaneous adsorption of hydrocarbons on the sample surface indoors, resulting in drastic changes in the contrast of some areas of the microscopic image, such as some areas are extremely bright and some areas are extremely dark. Therefore, pollution simulation can be performed by adding background noise to make the obtained expanded microscopic image more in line with the actual situation, thereby achieving more realistic image expansion of the microscopic image and improving image quality.

[0051] Among them, step S104 includes: according to the image expansion model, data enhancement is performed on the characteristic image of the microscopic image to obtain an enhanced image; mechanical noise is added to the enhanced image to obtain a first image, and the mechanical noise includes applying stress in the horizontal and vertical directions to simulate sample drift and performing numerical rolling operations in the horizontal direction to simulate probe jitter; based on a preset ratio, Gaussian noise is applied to the first image to simulate global noise to obtain a second image; a low-frequency background signal is extracted from the real experimental image using a Fourier filtering method, and is added to the second image based on a preset ratio to generate an expanded microscopic image.

[0052] Specifically, the feature image of the microscopic image is input into the corresponding image expansion model, and data enhancement operations such as rotation, flipping, and scaling are performed on it to obtain an enhanced image; in order to simulate the electron microscope image to better represent the real electron microscope image, it is also necessary to add three types of noise to several microscopic images according to the image expansion model, including mechanical noise, Gaussian noise and background noise of the real electron microscope.

[0053] When adding mechanical noise to the enhanced image, stress is applied in the horizontal and vertical directions to simulate sample drift in electron microscopy experiments, and the image is numerically scrolled in the horizontal direction to realize the influence of probe jitter under experimental conditions on the final imaging result, thereby obtaining the first image.

[0054] Due to the background noise of the electron source and detector, the statistical noise of the detector, the instability of the electron beam, the interference of the sample and the environment, and the randomness of the interaction between the electron beam and the sample, Gaussian noise will appear in the microscopic image. Therefore, after the mechanical noise is added, it is necessary to apply Gaussian noise to the first image according to a preset ratio to obtain the second image. The degree of added noise is controlled by adjusting the size of the standard deviation of the Gaussian distribution. The larger the standard deviation, the more noise is added, thereby achieving simulation of global noise.

[0055] After the Gaussian noise is added, background noise needs to be applied to the second image based on a preset ratio to generate an expanded microscopic image. When adding background noise, the random brightness contrast interface can be used to process the contrast to simulate the contrast changes caused by hydrocarbon adsorption, or the Fourier filtering method can be used to extract the low-frequency background signal from the real experimental image and add it to the microscopic image to achieve a more realistic sample contamination simulation.

[0056] By adding the above three types of noise, we can obtain an expanded microscopic image that is closer to reality and achieve accurate simulation of atomic-level microscopic images, so as to improve the model's generalization ability to real situations.

[0057] Step S105 , constructing an expanded macroscopic data set and / or microscopic data set based on the expanded macroscopic image and / or the expanded microscopic image.

[0058] Specifically, an expanded macroscopic data set is constructed based on the expanded macroscopic image, and an expanded microscopic data set is constructed based on the expanded microscopic image. If the acquired image contains both macroscopic and microscopic images, a multi-scale data set can be constructed based on the expanded images at multiple scales, thereby realizing the expansion of image data sets at multiple scales from macroscopic to atomic level and microscopic, as well as the realistic simulation of the complex relationships between images of different scales, thereby improving the image quality in the data set, thereby helping to improve model performance and meet data processing requirements.

[0059] In this embodiment, a feature image is obtained by randomly acquiring a number of images and performing image preprocessing and feature processing. The number of images include macro images and / or micro images. Based on the image scales and image types of the number of images, a corresponding image expansion model is constructed in combination with an adaptive noise addition and sliding window algorithm to achieve customized construction of multi-scale and different types of image models to improve the performance of the model. When expanding a macro image, based on the image type of the macro image, the feature image of the macro image is subjected to multiple image construction processes according to the image expansion model to obtain an expanded macro image. By dividing the image types and performing targeted processing, a more realistic expanded macro image can be obtained. When expanding a micro image, According to the image augmentation model, the feature image of the microscopic image is enhanced to enhance the data diversity, and mechanical noise, Gaussian noise and background noise in preset proportions are added to generate an augmented microscopic image of the microscopic image. Based on the multi-type noise model of the real electron microscope image, a more realistic augmented microscopic image is obtained, which improves the authenticity of the atomic-level simulated image; based on the augmented macroscopic image and / or the augmented microscopic image, an augmented macroscopic data set and / or microscopic data set is constructed to achieve data set expansion of multi-scale images, obtain higher quality augmented images, and improve the effectiveness of the augmented images, thereby effectively improving the generalization ability and robustness of the model on images of different scales and meeting the needs of data processing.

[0060] The above method is also applicable to the processing of mesoscopic images and the construction of data sets, thereby achieving the expansion of data sets for multi-scale images.

[0061] In one embodiment, in order to verify the effectiveness of the expanded macro data set, an open source image defogging deep learning model based on diffusion model and frequency compensation is selected as a test model. In order to unify the training image scale and consider the training speed and time, the one-to-one corresponding foggy images, fog-free images and depth images in the prepared data set are uniformly adjusted to 128 pixels and 128 pixels wide through image processing software, and the order of all data points in the data set is randomly disrupted to alleviate the overfitting problem of the deep learning model.

[0062] Inside the model, the image signal of the original fog-free image will be destroyed by multi-step, small amounts of Gaussian noise during the diffusion process, and eventually become isotropic Gaussian noise. During the generation process, the model will sample a random noise from the above isotropic Gaussian noise, and then use the foggy image in the data pair as the generation condition, input the foggy image as the condition and the sampled noise into the model at the same time, and then repeat the above steps to finally get the dehazed image.

[0063] like Figure 4 As shown in the figure, the image expansion model combines the atmospheric scattering model (ASM) and the FoHIS model to process indoor and outdoor images respectively. Cases 1 and 2 are outdoor images, which are processed by the ASM model, and cases 3 and 4 are indoor images, which are processed by the FoHIS model. Therefore, the deep learning model trained by it achieves better defogging effect in dense fog and light fog, indoor and outdoor scenes, compared with the data without indoor and outdoor classification and using the atmospheric scattering model alone.

[0064] In one embodiment, in order to test the expanded micro data set, a residual U-type network model with 4 times maximum pooling downsampling is used. The spatial dimension of the feature map is gradually reduced by 4 times of downsampling, while the depth of the feature is increased, which helps the network capture feature information of different scales. In the process of downsampling, a pooling layer is used to reduce the resolution of the feature map while maintaining the transmission of important information. At the same time, in order to ensure the gradient backpropagation in the deep neural network, a skip connection is introduced between the encoder output features and the decoder input features of the same size. The model takes minimizing the binary cross entropy loss (BCE Loss) as the optimization goal, and adjusts the learning rate adaptively by introducing a learning rate scheduler and combining the Adam optimizer to continuously adjust and optimize the weights of the network.

[0065] After model training, performance evaluation was performed on the validation set, using the Dice coefficient and BCE Loss (binary cross entropy loss) as evaluation criteria to monitor the model training progress. Finally, the Dice coefficient of the model trained using the constructed dataset reached 0.978 and 0.976 on the training set and validation set, respectively, and the BCE Loss was 0.013 and 0.012.

[0066] In order to verify the effect of the model trained with the expanded microscopic dataset, zeolite ZSM-5 (see Figure 5 first row) and transition metal sulfide WS2 (see Figure 5 Scanning transmission electron microscopy (STEM) images of the (second row) as a test case for the model.

[0067] First, the original STEM image ( Figure 5 (a) and (d)) are used as the input of the model, and the output of the residual U-shaped network is obtained Figure 5 (b) and (e). Then, the spot detection algorithm is used to determine the atomic column coordinates, radius and other information from the output of the model; based on the above information, the physical information such as the pixel peak value, mean value, local crystal structure of each atomic column is extracted from the input and output images of the model. Finally, using this physical information, an unsupervised machine learning algorithm is used to classify the elements of each atomic column. The results are shown in Figure 5 (c) and (f). Obviously, the model trained with the expanded microscopic dataset can achieve a realistic simulation of microscopic images.

[0068] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0069] Obviously, those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a computer storage medium (ROM / RAM, magnetic disk, optical disk) and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Therefore, the present invention is not limited to any specific combination of hardware and software.

[0070] The above contents are further detailed descriptions of the present invention in combination with specific implementation methods, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. A method for expanding and constructing a noise-modified multi-scale machine vision task dataset, characterized in that: The following steps are involved: Randomly acquiring a number of images, performing image preprocessing and feature extraction on the number of images to obtain feature images, the number of images including macroscopic images and / or microscopic images; Based on the image scales and image types of the plurality of images, a corresponding image expansion model is constructed in combination with an adaptive noise addition and a sliding window algorithm; Based on the image type of the macro image and according to the image expansion model, performing multiple image construction processes on the feature image of the macro image to obtain an expanded macro image; and / or, according to the image expansion model, performing data enhancement on the feature image of the microscopic image, and adding mechanical noise, Gaussian noise and background noise in preset proportions to generate an expanded microscopic image; Based on the expanded macroscopic image and / or the expanded microscopic image, an expanded macroscopic data set and / or microscopic data set is constructed.

2. The method for expanding and constructing a noise-modified multi-scale machine vision task dataset according to claim 1, characterized in that: The randomly acquiring a number of images comprises: A number of outdoor image pairs are randomly collected from an outdoor dataset, wherein the outdoor image pairs include a clear outdoor image without fog and a foggy image corresponding to the outdoor image; A number of indoor image pairs are randomly collected in an indoor dataset, wherein the indoor image pairs include a clear and fog-free indoor image and a foggy image corresponding to the indoor image; A number of mixed images are randomly collected from a mixed data set, image pair recognition is performed on the number of mixed images to obtain indoor image pairs, outdoor image pairs and single images, and noise is added to the single image to obtain indoor image pairs and / or outdoor image pairs, wherein the indoor images and outdoor images in the mixed data set are in a preset ratio; Based on the outdoor image pair and the indoor image pair, several macro images are obtained.

3. The method for expanding and constructing a noise-modified multi-scale machine vision task dataset according to claim 1, characterized in that: The randomly acquiring a number of images also includes: Obtain several electron microscope images of the material and corresponding label images according to electron microscope experiments; and / or, obtaining a structural file of the material, and simulating and generating a plurality of electron microscope images and corresponding label images based on the structural file and electron microscope simulation parameters; The method comprises setting the pixel value of a single atom in the electron microscope image to a two-dimensional Gaussian distribution, reading the three-dimensional position coordinate information of the atom based on the structure file, and using a circular convolution method to obtain a label map corresponding to the electron microscope image on a two-dimensional plane; According to the plurality of electron microscope images and the corresponding label images, a microscopic image pair is formed to obtain a plurality of microscopic images.

4. The method for expanding and constructing a noise-modified multi-scale machine vision task dataset according to claim 2, characterized in that: The method of performing multiple image construction processes on the feature image of the macro image based on the image type of the macro image and according to the image expansion model to obtain an expanded macro image includes: According to the image expansion model, a fog image simulation method of an atmospheric scattering model is used to construct multiple image pairs of feature images of the outdoor image, and random noise is used to perform noise disturbance to obtain an expanded outdoor image; According to the image expansion model, the fog image simulation method of the FoHIS model is used to construct multiple image pairs of the feature image of the indoor image, and random noise is used to perform noise disturbance to obtain an expanded indoor image; The expanded outdoor image and the expanded indoor image are combined to obtain an expanded macro image.

5. The method for expanding and constructing a noise-modified multi-scale machine vision task dataset according to claim 4, characterized in that: The random noise is obtained based on isotropic Gaussian noise sampling of the haze-free image.

6. The method for expanding and constructing a noise-modified multi-scale machine vision task dataset according to claim 1, characterized in that: The method of performing data enhancement on the feature image of the microscopic image according to the image expansion model and adding mechanical noise, Gaussian noise and background noise of preset proportions to generate an expanded microscopic image includes: According to the image expansion model, data enhancement is performed on the feature image of the microscopic image to obtain an enhanced image; Adding mechanical noise to the enhanced image to obtain a first image, wherein the mechanical noise includes applying stress in horizontal and vertical directions to simulate sample drift and performing a numerical scrolling operation in the horizontal direction to simulate probe jitter; Based on a preset ratio, applying Gaussian noise to the first image to simulate global noise to obtain a second image; A low-frequency background signal is extracted from a real experimental image using a Fourier filtering method, and is added to the second image based on a preset ratio to generate an expanded microscopic image.

Images