A processing method and apparatus for generating electron microscopy graph big data sets

Abstract

Embodiments of the present application relate to a processing method and device for generating electron microscope image big data set, the method comprising: constructing a first data set; taking a visual semantic segmentation large model as a first segmentation model, taking an SDXL model as a first generation model; and first fine-tuning the first segmentation model according to the first data set, and then fine-tuning the first generation model according to the first data set and the first segmentation model; constructing a first simulated semantic segmentation image library; and generating simulated electron microscope images by the first generation model according to the first simulated semantic segmentation image library, and each first simulated electron microscope image and the corresponding first simulated semantic segmentation image form a second data record, and all the obtained second data records form an electron microscope image big data set. Through the present application, a large number of electron microscope images can be automatically generated without the help of any manpower or external resources, and the data preparation difficulty of the electron microscope image big data set is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a processing method and device for generating a large data set of electron microscope images. BACKGROUND

[0002] Experimental characterization is a process of researching and quantifying the physical, chemical and structural properties of materials or substances through various analytical techniques and instruments. Through experimental characterization, the color, morphology, crystal structure, chemical composition and other characteristics of materials can be deeply understood, which helps quality control, performance optimization and new material development. Electron microscopy (EM) plays an important role in experimental characterization, which observes the structure and morphology of materials at different scales through imaging technology. For example, scanning electron microscope (SEM) can analyze the morphological characteristics of materials in the industrial field, which is of great significance for material and product standardization and quality control; in the field of scientific research, electron microscopy can reveal the microstructure of materials, which helps to understand and develop new materials.

[0003] Traditional electron microscope image characterization analysis relies on manual processing by software, such as ImageJ or DigitalMicrograph. The problems of this traditional analysis method are obvious: low analysis efficiency, poor analysis consistency, and inability to handle large number of analysis tasks. In order to improve analysis efficiency, reduce analysis error, improve analysis consistency and improve the processing capacity of large number of analysis tasks, some researchers begin to use deep learning models to process electron microscope image characterization analysis tasks. From the technical principle, the introduction of deep learning model can indeed achieve the above improvement effect, but the prerequisite is that the model used must be fully trained, that is, a large number of (at least more than one hundred) or even a large number of (at least more than one thousand) electron microscope images are needed to truly play the technical advantages of the model. However, it is very difficult to prepare a large number of electron microscope images in the real environment, and the main reasons are as follows: limited channels for obtaining electron microscope images, high cost of obtaining, and long period of pixel-level electron microscope image semantic annotation based on manual annotation mechanism. SUMMARY

[0004] The present application aims at the defects of the prior art, and provides a processing method and device for generating an electron microscope image big data set, an electronic equipment and a computer readable storage medium. The present application only collects a small amount of true value electron microscope images, and performs pixel-level semantic labeling on each true value electron microscope image to obtain a corresponding semantic segmentation image by means of artificial labeling or other machine labeling, and a first data record is composed of each true value electron microscope image and the corresponding semantic segmentation image, and all the obtained first data records form a corresponding small-scale data set, i.e., a first data set. A visual semantic segmentation large model that has completed model pre-training is used as a first segmentation model, and an SDXL (Stable Diffusion XL) model that has completed model pre-training is used as a first generation model. The first segmentation model is fine-tuned using the first data set, and the first generation model is fine-tuned based on the first data set and the first segmentation model. After fine-tuning, a random simulation semantic segmentation image is created using a basic particle information library to obtain a corresponding simulation semantic segmentation image library, and the first generation model generates a corresponding simulation electron microscope image based on each simulation semantic segmentation image in the image library. Each simulation electron microscope image and the corresponding simulation semantic segmentation image form a second data record, and all the obtained second data records form an electron microscope image big data set. The first segmentation model is used to fine-tune the first generation model, the basic particle information library is used to construct a diverse simulation semantic segmentation image library, and the first generation model is used to generate a diverse electron microscope image big data set based on the simulation semantic segmentation image library. The present application can automatically generate a large number of electron microscope images without the aid of any manpower or external resources, can improve the electron microscope image semantic labeling efficiency, reduce the electron microscope image acquisition cost, and reduce the data preparation difficulty of the electron microscope image big data set.

[0005] To achieve the above-mentioned object, the first aspect of the embodiment of the present application provides a processing method for generating an electron microscope image big data set, which comprises:

[0006] A first number of first true value electron microscope images are collected, and a corresponding first semantic segmentation image is obtained by performing pixel-level semantic labeling on each first true value electron microscope image by means of artificial labeling or other machine labeling. Each first true value electron microscope image and the corresponding first semantic segmentation image form a corresponding first data record, and all the obtained first data records form a corresponding first data set. The first number is a positive integer not more than 10;

[0007] a first segmentation model of a class of visual semantic segmentation large models that have completed model pre-training; and a first generation model of an SDXL model that has completed model pre-training; and first fine-tuning the first segmentation model according to the first data set for electron microscope image semantic segmentation performance; and then fine-tuning the first generation model according to the first data set and the first segmentation model for simulated electron microscope image generation performance;

[0008] According to the preset second number and the basic particle information library, a corresponding first simulated semantic segmentation map library is obtained by creating a simulated semantic segmentation map; and a corresponding first simulated electron microscope image is obtained by the first generation model according to each first simulated semantic segmentation map of the first simulated semantic segmentation map library; and each first simulated electron microscope image and the corresponding first simulated semantic segmentation map form a corresponding second data record; and all the obtained second data records form a corresponding electron microscope image big data set; the first simulated semantic segmentation map library is composed of the second number of first simulated semantic segmentation maps; the second number is a positive integer not less than 100.

[0009] Preferably, the first segmentation model at least includes a DeepLab series model, a DETR model, a Segformer model, and a SAM series model; the first segmentation model is used for semantic segmentation processing of the electron microscope image input by the model and outputs a corresponding semantic segmentation map;

[0010] The first generation model has processing capability for multiple image generation tasks after completing model pre-training, and the multiple image generation tasks at least include an image generation task of generating a corresponding simulated image according to a semantic segmentation map;

[0011] The first generation model is used for identifying the first input image type and the first generated image type corresponding to the prompt text input by the model; and when the first input image type is a semantic segmentation map and the first generated image type is an electron microscope image or a simulated electron microscope image, the first generation model is used for performing corresponding simulated electron microscope image generation processing according to the semantic segmentation map input by the model and outputting a corresponding generated image;

[0012] The basic particle information library includes a plurality of basic particle information; the basic particle information includes particle type, particle shape type, particle color, particle size constraint, and particle growth condition constraint; the particle shape type includes spherical shape and irregular polygon;

[0013] Preferably, the first segmentation model according to the first data set is fine-tuned for electron microscope image semantic segmentation performance, specifically including:

[0014] Step 31, randomly split the first data set into two sub-data sets based on a preset first split ratio, denoted as a corresponding first training set and a first evaluation set;

[0015] Wherein, the first training set and the first evaluation set are both composed of a plurality of first data records; the ratio of the total number of records of the first training set to the total number of records of the first evaluation set meets the first split ratio;

[0016] Step 32, extract the first first data record of the first training set as a corresponding current training record;

[0017] Step 33, input the first true value electron microscope image of the current training record into the first segmentation model to perform corresponding electron microscope image semantic segmentation processing to obtain a corresponding first predicted semantic segmentation map;

[0018] Step 34, bring the first predicted semantic segmentation map and the first semantic segmentation map of the current training record into a preset first model loss function; and based on a preset first model optimizer, a round of fine-tuning is performed on the model parameters of the first segmentation model in the direction of minimizing the first model loss function;

[0019] Wherein, the first model loss function includes L1 loss function, L2 loss function and cross-entropy loss function; the first model optimizer includes at least ADAM series optimizer, LoRA series optimizer, zero-order series optimizer;

[0020] Step 35, identify whether the current training record is the last first data record of the first training set; if yes, go to step 36; if no, extract the next first data record of the first training set as a new current training record and return to step 33;

[0021] Step 36, perform a round of traversal on all first data records of the first evaluation set; and in this round of traversal, the first data record currently traversed is taken as a corresponding current evaluation record; and the first true value electron microscope image of the current evaluation record is input into the first segmentation model to perform corresponding electron microscope image semantic segmentation processing to obtain a corresponding second predicted semantic segmentation map; and the second predicted semantic segmentation map and the first semantic segmentation map of the current evaluation record form a corresponding first prediction-label pair; and at the end of this round of traversal, the first accuracy, the first precision, the first recall and the first F1 score are obtained by calculating the first accuracy, the first precision, the first recall and the first F1 score according to all the first prediction-label pairs obtained;

[0022] Step 37, the first accuracy, the first precision, the first recall and the first F1 score are identified; if the first accuracy does not meet the preset first accuracy range or the first precision does not meet the preset first precision range or the first recall does not meet the preset first recall range or the first F1 score does not meet the preset first F1 score range, return to step 32 for continuous fine tuning; if the first accuracy meets the first accuracy range, the first precision meets the first precision range, the first recall meets the first recall range and the first F1 score meets the first F1 score range, stop training and confirm that the model fine tuning of the first segmentation model is completed.

[0023] Preferably, the first data set and the first segmentation model are used to fine tune the simulation electron microscope image generation performance of the first generation model, specifically including:

[0024] Step 41, the first data set is randomly divided into two sub data sets based on a preset second segmentation ratio, denoted as a corresponding second training set and a second evaluation set;

[0025] Wherein, the second training set and the second evaluation set are both composed of a plurality of first data records; the ratio of the total number of records of the second training set to the total number of records of the second evaluation set meets the second segmentation ratio;

[0026] Step 42, the first first data record of the second training set is extracted as a corresponding current training record;

[0027] Step 43, a preset simulation electron microscope image generation prompt template is used as a corresponding current prompt text;

[0028] Wherein, the simulation electron microscope image generation prompt template is a formatted text module; the simulation electron microscope image generation prompt template is used to set the image type of the current model as a semantic segmentation map, set the generated image type of the current model as an electron microscope image or a simulation electron microscope image, and inform the first generation model to perform corresponding simulation electron microscope image generation processing according to the image of the current model and output the simulation electron microscope image generated at the time as the corresponding current model generated image through a narrative text;

[0029] Step 44, the current prompt text and the first semantic segmentation map of the current training record are input into the first generation model for corresponding simulation electron microscope image generation processing to obtain a corresponding first predicted simulation electron microscope image;

[0030] Step 45, the first predicted simulation electron microscope image is input into the first segmentation model for corresponding electron microscope image semantic segmentation processing to obtain a corresponding third predicted semantic segmentation map;

[0031] Step 46, the third predicted semantic segmentation map is brought into a preset second model loss function with the first semantic segmentation map of the current training record; and a round of fine-tuning of the model parameters of the first generative model is performed based on a preset second model optimizer towards a direction of minimizing the second model loss function;

[0032] The second model loss function includes an L1 loss function, an L2 loss function, and a cross-entropy loss function; and the second model optimizer includes at least an ADAM series optimizer, a LoRA series optimizer, and a zero-order series optimizer.

[0033] Step 47, whether the current training record is the last first data record of the second training set is identified; if yes, step 48 is entered; if not, the next first data record of the second training set is extracted as a new current training record and step 44 is returned to;

[0034] Step 48, a round of traversal is performed on all the first data records of the second evaluation set; and in the round of traversal, the first data record currently traversed is taken as a corresponding current evaluation record; the current prompt text and the first semantic segmentation map of the current evaluation record are input into the first generative model to obtain a corresponding second predicted simulation TEM image through corresponding simulation TEM image generation processing; the second predicted simulation TEM image is input into the first segmentation model to obtain a corresponding fourth predicted semantic segmentation map through corresponding TEM image semantic segmentation processing; and the fourth predicted semantic segmentation map and the first semantic segmentation map of the current evaluation record form a corresponding second predicted-label pair; and at the end of the round of traversal, the second accuracy, the second precision, the second recall, and the second F1 score are obtained through corresponding accuracy, precision, recall, and F1 score calculation according to all the second predicted-label pairs obtained.

[0035] Step 49, the second accuracy, the second precision, the second recall, and the second F1 score obtained are identified; if the second accuracy does not satisfy a preset second accuracy range, or the second precision does not satisfy a preset second precision range, or the second recall does not satisfy a preset second recall range, or the second F1 score does not satisfy a preset second F1 score range, step 42 is returned to continue fine-tuning; if the second accuracy satisfies the second accuracy range, the second precision satisfies the second precision range, the second recall satisfies the second recall range, and the second F1 score satisfies the second F1 score range, the training is stopped and it is confirmed that the model fine-tuning of the first generative model is ended.

[0036] Preferably, the simulation semantic segmentation map library is created according to the preset second quantity and the base particle information library, and specifically includes:

[0037] Step 51, initializing a first counter to 1; and initializing the first simulation semantic segmentation map library to be empty;

[0038] Step 52, initializing a corresponding first simulation semantic segmentation map based on a preset semantic segmentation map shape;

[0039] The semantic segmentation map shape includes an image width W, an image height H, and a pixel feature vector length D; the shape of the first simulation semantic segmentation map is WxHxD; the first simulation semantic segmentation map is composed of WxH first pixel feature vectors with a length of D; the first pixel feature vector corresponds to a pixel point on the first simulation semantic segmentation map; the vector data of the first pixel feature vector at least includes a first pixel coordinate, a first pixel value, and a first semantic type; all the first pixel values on the first simulation semantic segmentation map at the initialization time are preset background pixel values, and all the first semantic types are preset background types;

[0040] Step 53, randomly selecting a plurality of base particle information from the base particle information library to form a corresponding current base particle information set;

[0041] Step 54, performing random particle image adding processing on the first simulation semantic segmentation map based on the current base particle information set, specifically: based on the particle shape type, the particle color, and the particle size of each base particle information in the current base particle information set, performing corresponding particle image adding on a random position on the first simulation semantic segmentation map to obtain a corresponding first particle image;

[0042] Step 55, performing random particle growth processing on the first simulation semantic segmentation map, specifically: randomly selecting one or more from all the first particle images obtained as corresponding to-be-grown particle images; and based on the particle growth condition constraint corresponding to each to-be-grown particle image, performing particle growth simulation and performing image copying on the current to-be-grown particle image at the current growth position each time a new growth position is generated on the first simulation semantic segmentation map;

[0043] Step 56, the first simulation semantic segmentation map on the part of the stack particles is shifted, specifically: the coincidence degree of each two first particle images on the first simulation semantic segmentation map is calculated to obtain a corresponding first coincidence degree; and the image position of each first coincidence degree corresponding to two first particle images exceeding a preset first coincidence degree threshold is adjusted until the first coincidence degree of any two first particle images does not exceed the first coincidence degree threshold;

[0044] Step 57, the first semantic type of the first pixel feature vector in each first particle image range on the first simulation semantic segmentation map is set as the particle type of the basic particle information corresponding to the current first particle image;

[0045] Step 58, the first simulation semantic segmentation map obtained this time is added to the first simulation semantic segmentation map library; and the first counter is incremented by 1;

[0046] Step 59, whether the first counter is greater than the second quantity is identified; if not, return to step 52; if yes, the latest first simulation semantic segmentation map library is output.

[0047] Preferably, the first simulation SEM image corresponding to the first simulation semantic segmentation map is generated by the first generation model according to each first simulation semantic segmentation map of the first simulation semantic segmentation map library, specifically including:

[0048] Each first simulation semantic segmentation map of the first simulation semantic segmentation map library is taken as a corresponding current semantic segmentation map; a preset simulation SEM image generation prompt template is taken as a corresponding current prompt text; and the current prompt text and the current semantic segmentation map are input into the first generation model to obtain a corresponding first simulation SEM image through corresponding simulation SEM image generation processing.

[0049] The second aspect of the embodiment of the application provides a device for implementing the processing method for generating an electron microscope image big data set in the first aspect, and the device comprises a small-scale data set preparation module, a model fine-tuning module and a big data set generation module.

[0050] The small-scale data set preparation module is used for collecting a first quantity of first true value SEM images; performing pixel-level semantic labeling on each first true value SEM image through an artificial labeling method or other machine labeling method to obtain a corresponding first semantic segmentation map; and each first true value SEM image and the corresponding first semantic segmentation map form a corresponding first data record; all the first data records obtained form a corresponding first data set; and the first quantity is a positive integer not more than 10.

[0051] The model fine-tuning module is configured to use a visual semantic segmentation large model that has completed model pre-training as a corresponding first segmentation model, and use an SDXL model that has completed model pre-training as a corresponding first generation model, and fine-tune the electron microscope image semantic segmentation performance of the first segmentation model according to the first data set, and fine-tune the simulated electron microscope image generation performance of the first generation model according to the first data set and the first segmentation model.

[0052] The big data set generation module is configured to create a first simulated semantic segmentation image library according to a preset second number and a basic particle information library, and generate a corresponding first simulated electron microscope image according to each first simulated semantic segmentation image in the first simulated semantic segmentation image library by using the first generation model, and form a corresponding second data record by using each first simulated electron microscope image and the corresponding first simulated semantic segmentation image, and form a corresponding electron microscope image big data set by using all the obtained second data records, wherein the first simulated semantic segmentation image library is composed of the first simulated semantic segmentation images of the second number, and the second number is a positive integer not less than 100.

[0053] The third aspect of the embodiment of the present application provides an electronic device, which comprises a memory, a processor and a transceiver.

[0054] The processor is configured to be coupled with the memory, read and execute instructions in the memory, so as to realize the method steps of the first aspect.

[0055] The transceiver is coupled with the processor, and the transceiver is controlled by the processor to perform message transceiving.

[0056] The fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores computer instructions, and when the computer instructions are executed by a computer, the computer instructions make the computer execute the instructions of the method of the first aspect.

[0057] The embodiment of the present application provides a processing method and device for generating an electron microscope image big data set, an electronic equipment and a computer readable storage medium. From the above content, it can be known that the embodiment of the present application only collects a small amount of true value electron microscope images, and obtains corresponding semantic segmentation images through pixel-level semantic labeling of each true value electron microscope image in an artificial labeling manner or other machine labeling manner, and each true value electron microscope image and the corresponding semantic segmentation image form a first data record, and all the obtained first data records form a corresponding small-scale data set, that is, a first data set; and a visual semantic segmentation big model which has completed model pre-training is used as a first segmentation model, and an SDXL (Stable Diffusion XL) model which has completed model pre-training is used as a first generation model, and the first segmentation model is fine-tuned using the first data set, and the first generation model is fine-tuned based on the first data set and the first segmentation model; after the fine-tuning is completed, a random simulation semantic segmentation image is created by using a basic particle information library, and a corresponding simulation semantic segmentation image library is obtained, and then the first generation model generates corresponding simulation electron microscope images based on each simulation semantic segmentation image in the image library, and each simulation electron microscope image and the corresponding simulation semantic segmentation image form a second data record, and all the obtained second data records form an electron microscope image big data set. The first segmentation model provided in the embodiment of the present application is used for fine-tuning the first generation model, the basic particle information library is used for constructing a diverse simulation semantic segmentation image library, and the first generation model is used for generating a diverse electron microscope image big data set according to the simulation semantic segmentation image library; through the embodiment of the present application, a large amount of electron microscope images can be automatically generated without the help of any manpower or external resources, the semantic labeling efficiency of the electron microscope images is improved, the acquisition cost of the electron microscope images is reduced, and the data preparation difficulty of the electron microscope image big data set is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 A processing method for generating an electron microscope image big data set is provided for the embodiment one of the present application;

[0059] Figure 2 A module structure diagram of a processing device for generating an electron microscope image big data set is provided for the embodiment two of the present application;

[0060] Figure 3 A structure schematic diagram of an electronic equipment is provided for the embodiment three of the present application. DETAILED DESCRIPTION

[0061] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0062] The embodiment one of the present application provides a processing method for generating electron microscope image big data set, which is shown in the schematic diagram of the processing method for generating electron microscope image big data set provided by the embodiment one of the present application. Figure 1 The embodiment one of the present application provides a processing method for generating electron microscope image big data set, which is shown in the schematic diagram of the processing method for generating electron microscope image big data set provided by the embodiment one of the present application. The method mainly includes the following steps:

[0063] Step 1, collect a first number of first true value electron microscope images; and perform pixel-level semantic labeling on each first true value electron microscope image by manual labeling or other machine labeling to obtain a corresponding first semantic segmentation image; and a corresponding first data record is composed of each first true value electron microscope image and the corresponding first semantic segmentation image; and a corresponding first data set is composed of all the obtained first data records;

[0064] Wherein, the first number is a positive integer not more than 10.

[0065] Here, the embodiment of the present application can collect true value electron microscope images by electron microscope experiment, or can collect true value electron microscope images through other public true value image acquisition channels; because the first number is less than 10, the difficulty of collecting a few true value electron microscope images is still relatively small; the first data set obtained by data collection and manual / machine labeling is composed of a first number of first data records, and each first data record is composed of a first true value electron microscope image and a first semantic segmentation image.

[0066] Step 2, a visual semantic segmentation large model that has completed model pre-training is used as a corresponding first segmentation model; an SDXL model that has completed model pre-training is used as a corresponding first generation model; the electron microscope image semantic segmentation performance of the first segmentation model is first fine-tuned according to the first data set; and the simulation electron microscope image generation performance of the first generation model is fine-tuned according to the first data set and the first segmentation model;

[0067] Specifically, step 21, a visual semantic segmentation large model that has completed model pre-training is used as a corresponding first segmentation model;

[0068] Here, the visual semantic segmentation large model of the embodiment of the application, i.e., the first segmentation model, at least includes a DeepLab series model, a DETR model, a Segformer model, and a SAM series model; the first segmentation model is used for performing semantic segmentation processing on the TEM image input by the model and outputting a corresponding semantic segmentation image; the use of the pre-trained model in the embodiment of the application can greatly shorten the training period of the model and improve the training efficiency of the model;

[0069] Step 22, an SDXL model that has completed model pre-training is used as a corresponding first generation model;

[0070] Here, the first generation model of the embodiment of the application is implemented based on an SDXL model, and the detailed model structure, pre-training scheme, and other information of the SDXL model can be understood by referring to the published technical literature A SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis and the related technical literature B High-Resolution Image Synthesis with Latent Diffusion Models, and will not be further described here. It should be noted that, as can be known from the technical literature A and B, the first generation model has processing capability for multiple image generation tasks after completing model pre-training, and the multiple image generation tasks mentioned here at least include an image generation task of generating a corresponding simulation image according to a semantic segmentation image. In addition, as can be known from the technical literature A and B, the first generation model belongs to a type of text-to-image generation model, and a prompt text needs to be input at the same time as the semantic segmentation image;

[0071] That is, the first generation model of the embodiment of the application is used for performing input image type and generated image type identification on the prompt text input by the model to obtain a corresponding first input image type and a first generated image type; and when the first input image type is a semantic segmentation image and the first generated image type is a TEM image or a simulation TEM image, performing corresponding simulation TEM image generation processing according to the semantic segmentation image input by the model and outputting a corresponding generated image;

[0072] Step 23, first, the semantic segmentation performance of the first segmentation model on the TEM image is fine-tuned according to the first data set;

[0073] Specifically, step 231, the first data set is randomly divided into two sub-data sets based on a preset first segmentation ratio, which are denoted as a corresponding first training set and a first evaluation set;

[0074] The first split ratio is a preset ratio parameter, for example, 8:2; the first training set and the first evaluation set each consist of a plurality of first data records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set meets the first split ratio;

[0075] Step 232, the first data record of the first training set is extracted as the corresponding current training record;

[0076] Step 233, the first true value electron microscope image of the current training record is input into the first segmentation model for corresponding electron microscope image semantic segmentation processing to obtain a corresponding first predicted semantic segmentation map;

[0077] Step 234, the first predicted semantic segmentation map and the first semantic segmentation map of the current training record are input into a preset first model loss function; and based on a preset first model optimizer, the model parameters of the first segmentation model are fine-tuned in the direction of minimizing the first model loss function;

[0078] The first model loss function includes an L1 loss function, an L2 loss function, and a cross-entropy loss function; the first model optimizer includes at least an ADAM series optimizer, a LoRA series optimizer, and a zero-order series optimizer;

[0079] Step 235, whether the current training record is the last first data record of the first training set is identified; if yes, go to step 236; if no, the next first data record of the first training set is extracted as a new current training record and returns to step 233;

[0080] Step 236, all first data records of the first evaluation set are iterated once; and in this round of iteration, the currently iterated first data record is taken as the corresponding current evaluation record; the first true value electron microscope image of the current evaluation record is input into the first segmentation model for corresponding electron microscope image semantic segmentation processing to obtain a corresponding second predicted semantic segmentation map; and the second predicted semantic segmentation map and the first semantic segmentation map of the current evaluation record form a corresponding first predicted-label pair; and at the end of this round of iteration, the first accuracy, the first precision, the first recall, and the first F1 score are calculated according to all the obtained first predicted-label pairs.

[0081] Step 237, the first accuracy, the first precision, the first recall and the first F1 score are identified; if the first accuracy does not satisfy the preset first accuracy range or the first precision does not satisfy the preset first precision range or the first recall does not satisfy the preset first recall range or the first F1 score does not satisfy the preset first F1 score range, return to step 232 for continuous fine tuning; if the first accuracy satisfies the first accuracy range, the first precision satisfies the first precision range, the first recall satisfies the first recall range and the first F1 score satisfies the first F1 score range, stop training and confirm that the model fine tuning of the first segmentation model is completed;

[0082] Here, the first accuracy range, the first precision range, the first recall range and the first F1 score range are four preset numerical ranges;

[0083] Step 24, the simulation electron microscope image generation performance of the first generation model is fine tuned according to the first data set and the first segmentation model;

[0084] Specifically, step 241, the first data set is randomly divided into two sub data sets according to a preset second segmentation ratio, which are recorded as a corresponding second training set and a second evaluation set;

[0085] Wherein, the second segmentation ratio is a preset ratio parameter, for example, 8:2; the second training set and the second evaluation set are both composed of a plurality of first data records; the ratio of the total number of records of the second training set to the total number of records of the second evaluation set satisfies the second segmentation ratio;

[0086] Step 242, the first first data record of the second training set is extracted as a corresponding current training record;

[0087] Step 243, a preset simulation electron microscope image generation prompt template is used as a corresponding current prompt text;

[0088] Here, the simulation electron microscope image generation prompt template of the embodiment of the application is a formatted text module; the simulation electron microscope image generation prompt template is used to set the image type of the current model as a semantic segmentation image, set the generated image type of the current model as an electron microscope image or a simulation electron microscope image, and inform the first generation model through a narrative text that the first generation model performs corresponding simulation electron microscope image generation processing according to the image of the current model and outputs the simulation electron microscope image generated at the moment as the corresponding generated image of the current model;

[0089] Step 244, the current prompt text and the first semantic segmentation image of the current training record are input into the first generation model to perform corresponding simulation electron microscope image generation processing to obtain a corresponding first predicted simulation electron microscope image;

[0090] Step 245, input the first predicted simulation electron microscope image into the first segmentation model for corresponding electron microscope image semantic segmentation processing to obtain a corresponding third predicted semantic segmentation map;

[0091] Step 246, input the third predicted semantic segmentation map and the first semantic segmentation map of the current training record into a preset second model loss function; and based on a preset second model optimizer, fine-tune the model parameters of the first generation model in the direction of minimizing the second model loss function;

[0092] Wherein, the second model loss function includes L1 loss function, L2 loss function and cross-entropy loss function; the second model optimizer at least includes ADAM series optimizer, LoRA series optimizer, zero-order series optimizer;

[0093] Step 247, identify whether the current training record is the last first data record of the second training set; if yes, go to step 248; if no, extract the next first data record of the second training set as a new current training record and return to step 244;

[0094] Step 248, iterate all first data records of the second evaluation set; and in this round of iteration, the current iteration first data record is taken as a corresponding current evaluation record; and the current prompt text and the first semantic segmentation map of the current evaluation record are input into the first generation model for corresponding simulation electron microscope image generation processing to obtain a corresponding second predicted simulation electron microscope image; and the second predicted simulation electron microscope image is input into the first segmentation model for corresponding electron microscope image semantic segmentation processing to obtain a corresponding fourth predicted semantic segmentation map; and the fourth predicted semantic segmentation map and the first semantic segmentation map of the current evaluation record form a corresponding second prediction-label pair; and at the end of this round of iteration, the second accuracy, the second precision, the second recall and the second F1 score are calculated according to all the obtained second prediction-label pairs;

[0095] Step 249, identify the obtained second accuracy, second precision, second recall and second F1 score; if the second accuracy does not satisfy the preset second accuracy range or the second precision does not satisfy the preset second precision range or the second recall does not satisfy the preset second recall range or the second F1 score does not satisfy the preset second F1 score range, return to step 242 for further fine-tuning; if the second accuracy satisfies the second accuracy range and the second precision satisfies the second precision range and the second recall satisfies the second recall range and the second F1 score satisfies the second F1 score range, stop training and confirm that the model fine-tuning of the first generation model is completed;

[0096] Here, the second accuracy range, the second precision range, the second recall rate range, and the second F1 score range are four pre-set numerical ranges.

[0097] Step 3, according to the pre-set second quantity and the basic particle information library, a corresponding first simulation semantic segmentation map library is created; and according to each first simulation semantic segmentation map of the first simulation semantic segmentation map library, a corresponding first simulation electron microscope image is generated by the first generation model; each first simulation electron microscope image and the corresponding first simulation semantic segmentation map form a corresponding second data record; and all the obtained second data records form a corresponding electron microscope image big data set;

[0098] Specifically, step 31, according to the pre-set second quantity and the basic particle information library, a corresponding first simulation semantic segmentation map library is created;

[0099] The basic particle information library includes a plurality of basic particle information; the basic particle information includes particle type, particle shape type, particle color, particle size constraint, and particle growth condition constraint; the type range of the particle type can be customized according to specific application requirements, and the conventional particle type can include, for example, electrons, various ions, various atoms, various molecules, various groups, etc.; the particle shape type includes spherical shape and irregular polygon, and the irregular polygon can be further subdivided based on specific application requirements; the particle size constraint limits the maximum and minimum size / volume of the current particle and gives the corresponding boundary value; the particle growth condition constraint limits the various growable conditions of the current particle and the corresponding growth termination condition;

[0100] The first simulation semantic segmentation map library is composed of a second quantity of first simulation semantic segmentation maps; the second quantity is a positive integer not less than 100, and the second quantity has no maximum limit and can be customized according to specific application requirements;

[0101] Specifically, step 311, the first counter is initialized to 1; and the first simulation semantic segmentation map library is initialized to be empty;

[0102] Step 312, based on the pre-set semantic segmentation map shape, a corresponding first simulation semantic segmentation map is initialized;

[0103] Here, the semantic segmentation map shape of the embodiment of the application includes image width W, image height H, and pixel feature vector length D, and the three parameters W, H, and D are positive integers;

[0104] The shape of the first simulation semantic segmentation graph is WxHxD; the first simulation semantic segmentation graph is composed of WxH first pixel feature vectors with a length of D; wherein the first pixel feature vector corresponds to a pixel point on the first simulation semantic segmentation graph one by one; the vector data of the first pixel feature vector at least includes a first pixel coordinate, a first pixel value and a first semantic type; all the first pixel values on the first simulation semantic segmentation graph at the initialization time are preset background pixel values, and all the first semantic types are preset background types; the background pixel value is generally set to 0, that is, the default is black, and can also be customized according to specific application requirements;

[0105] Step 313, randomly selecting a plurality of basic particle information from the basic particle information library to form a corresponding current basic particle information set;

[0106] Step 314, performing random particle image adding processing on the first simulation semantic segmentation graph based on the current basic particle information set;

[0107] Specifically, based on the particle shape type, particle color and particle size constraint of each basic particle information of the current basic particle information set, corresponding particle image adding is performed at a random position on the first simulation semantic segmentation graph to obtain a corresponding first particle image;

[0108] Step 315, performing random particle growth processing on the first simulation semantic segmentation graph;

[0109] Specifically, one or more of all the first particle images obtained are randomly selected as corresponding to-be-grown particle images; and based on the particle growth condition constraint of each to-be-grown particle image, particle growth simulation is performed, and each time a new growth position is generated, the current to-be-grown particle image is copied on the current growth position;

[0110] Step 316, performing displacement processing on part of the stacked particles on the first simulation semantic segmentation graph;

[0111] Specifically, the coincidence degree of each two first particle images on the first simulation semantic segmentation graph is calculated to obtain a corresponding first coincidence degree; and the image positions of the two first particle images corresponding to each first coincidence degree exceeding a preset first coincidence degree threshold are adjusted until the first coincidence degrees of any two first particle images do not exceed the first coincidence degree threshold;

[0112] Here, the first coincidence degree threshold is a preset threshold parameter;

[0113] Step 317, set the first semantic type of the first pixel feature vector in each first particle image range on the first simulated semantic segmentation map as the particle type of the basic particle information corresponding to the current first particle image;

[0114] Step 318, add the first simulated semantic segmentation map obtained this time to the first simulated semantic segmentation map library, and increase the first counter by 1;

[0115] Step 319, identify whether the first counter is greater than the second quantity; if not, return to step 312; if yes, output the latest first simulated semantic segmentation map library;

[0116] Step 32, obtain the corresponding first simulated electron microscope image by the first generation model according to the corresponding first simulated semantic segmentation map of the first simulated semantic segmentation map library;

[0117] Specifically, each first simulated semantic segmentation map of the first simulated semantic segmentation map library is taken as a corresponding current semantic segmentation map, a preset simulated electron microscope image generation prompt template is taken as a corresponding current prompt text, and the current prompt text and the current semantic segmentation map are input into the first generation model to obtain the corresponding first simulated electron microscope image through corresponding simulated electron microscope image generation processing.

[0118] Step 33, each first simulated electron microscope image and the corresponding first simulated semantic segmentation map form a corresponding second data record, and all the obtained second data records form a corresponding electron microscope image big data set.

[0119] Figure 2 A module structure diagram of a processing device for generating an electron microscope image big data set is provided for the second embodiment of the application. The device is a terminal device or a server for implementing the foregoing method embodiments, or a device capable of enabling the foregoing terminal device or server to implement the foregoing method embodiments, such as a device or a chip system of the foregoing terminal device or server. As shown in the figure, the device includes a small-scale data set preparation module 201, a model fine-tuning module 202, and a big data set generation module 203. Figure 2

[0120] The small-scale data set preparation module 201 is used to collect a first quantity of first true value electron microscope images, perform pixel-level semantic labeling on each first true value electron microscope image to obtain a corresponding first semantic segmentation map through manual labeling or other machine labeling, form a corresponding first data record from each first true value electron microscope image and the corresponding first semantic segmentation map, form a corresponding first data set from all the obtained first data records, and the first quantity is a positive integer not greater than 10.

[0121] ​The model fine-tuning module 202 is configured to take a visual semantic segmentation large model that has completed model pre-training as a corresponding first segmentation model, and take an SDXL model that has completed model pre-training as a corresponding first generation model, and fine-tune the electron microscope image semantic segmentation performance of the first segmentation model according to a first data set, and fine-tune the simulated electron microscope image generation performance of the first generation model according to the first data set and the first segmentation model.

[0122] The large data set generation module 203 is configured to create a corresponding first simulated semantic segmentation image library according to a preset second quantity and a basic particle information library, and generate a corresponding first simulated electron microscope image according to each first simulated semantic segmentation image in the first simulated semantic segmentation image library by the first generation model, and form a corresponding second data record by each first simulated electron microscope image and the corresponding first simulated semantic segmentation image, and form a corresponding electron microscope image large data set by all the obtained second data records, and the first simulated semantic segmentation image library is composed of a second quantity of first simulated semantic segmentation images, and the second quantity is a positive integer not less than 100.

[0123] The processing device for generating an electron microscope image large data set provided in the embodiment of the present application can execute the method steps in the method embodiments described above, and has similar implementation principles and technical effects, which will not be described here in detail.

[0124] It should be noted that the division of each module of the above device is only a logical functional division, and all or part of the modules can be integrated into one physical entity, or can be physically separated. These modules can all be implemented in the form of software called by a processing element; all can be implemented in the form of hardware; or part of the modules can be implemented in the form of software called by a processing element, and part of the modules can be implemented in the form of hardware. For example, the small-scale data set preparation module can be a separately established processing element, or can be integrated in a chip of the above device, in addition, it can also be stored in the form of program code in the memory of the above device, and the function of the above determination module is called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or can be independently implemented. The processing element described herein can be an integrated circuit with signal processing capability. In the implementation process, each step of the above method or each module can be completed by the integrated logic circuit of hardware or the instruction of software in the processing element.

[0125] For example, the above modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), or one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs), etc. For another example, when a certain module above is implemented in the form of a processing element scheduling code, the processing element can be a general purpose processor, such as a Central Processing Unit (CPU) or other processor that can invoke code. For another example, the modules can be integrated together to implement in the form of a System-on-a-chip (SOC).

[0126] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed by a computer, the computer instructions generate all or part of the processes or functions described in the above method embodiments. The computer can be a general purpose computer, a special purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0127] Figure 3 A structural schematic diagram of an electronic device is provided for Embodiment Three of the present application. The electronic device can be a terminal device or a server implementing the method of the above embodiments, or a terminal device or a server connected to the terminal device or the server implementing the method of the above embodiments. As shown in FIG. 3, the electronic device includes a processor 301, a memory 302, a transceiver 303 and an antenna 304. The processor 301, the memory 302, the transceiver 303 and the antenna 304 can be connected to each other through a bus or other suitable connection means. The processor 301 can be configured to implement the method of the above embodiments. The memory 302 can be configured to store the computer instructions of the processor 301. The transceiver 303 can be configured to transmit and receive signals. The antenna 304 can be configured to transmit and receive signals. Figure 3As shown, the electronic device can include a processor 301 (such as a CPU), a memory 302, a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transceiving action of the transceiver 303. The memory 302 can store various instructions for completing various processing functions and implementing the processing steps described in the foregoing embodiment method description. Preferably, the electronic device related to the embodiments of the present application further includes a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize the communication connection between elements. The above-mentioned communication port 306 is used for connection communication between the electronic device and other peripherals.

[0128] In Figure 3 The system bus 305 mentioned in the above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to realize the communication between the database access device and other devices (such as the client, the read-write library and the read-only library). The memory can contain a Random Access Memory (RAM), and can also include a Non-Volatile Memory, such as at least one disk memory.

[0129] The above-mentioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), a Graphics Processing Unit (GPU), etc.; it can also be a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0130] It should be noted that the embodiments of the present application also provide a computer-readable storage medium, which stores instructions when running on a computer, so that the computer executes the method and processing procedure provided in the above embodiments.

[0131] Embodiments of the present invention provide a processing method, device, electronic device, and computer-readable storage medium for generating a large data set of electron microscope images. As can be seen from the above content, the embodiment of the present invention only collects a small number of true electron micrographs, and performs pixel-level semantic annotation on each true electron micrograph through manual annotation or other machine annotation methods to obtain a corresponding semantic segmentation map, and each true electron micrograph and the corresponding semantic segmentation map constitute a first data record, and all the obtained first data records constitute a corresponding small-scale data set, namely the first data set; and a class of visual semantic segmentation large models that have completed model pre-training is used as the first segmentation model, and an SDXL (StableDiffusion XL) model that has completed model pre-training is used as the first generative model, and the first segmentation model is first fine-tuned using the first data set, and then the first generative model is fine-tuned based on the first data set and the first segmentation model; after the fine-tuning is completed, the basic particle information library is used to randomly create simulated semantic segmentation maps and obtain a corresponding simulated semantic segmentation map library, and then the first generative model performs corresponding simulated electron micrograph generation processing based on each simulated semantic segmentation map in the map library, and then each simulated electron micrograph and the corresponding simulated semantic segmentation map constitute a second data record, and all the obtained second data records constitute the electron micrograph large data set. The first segmentation model provided by the embodiment of the present invention is used to fine-tune the first generation model, the basic particle information library is used to construct a diverse simulation semantic segmentation library, and the first generation model is used to generate a diverse electron microscopy image large data set based on the simulation semantic segmentation library; through the embodiment of the present invention, a large number of electron microscopy images can be automatically generated without the help of any manpower or external resources, thereby improving the semantic annotation efficiency of electron microscopy images, reducing the cost of obtaining electron microscopy images, and reducing the difficulty of data preparation for large electron microscopy image data sets.

[0132] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0133] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A processing method for generating electron microscopy graph big data sets, characterized in that, The method comprises: Collecting a first number of first true value electron microscope images; and performing pixel-level semantic labeling on each of the first true value electron microscope images by manual labeling or other machine labeling to obtain a corresponding first semantic segmentation map; and each of the first true value electron microscope images and the corresponding first semantic segmentation map forms a corresponding first data record; and all of the obtained first data records form a corresponding first data set; the first number is a positive integer not exceeding 10; A visual semantic segmentation large model that has completed model pre-training is used as a corresponding first segmentation model; and an SDXL model that has completed model pre-training is used as a corresponding first generation model; and the electron microscope image semantic segmentation performance of the first segmentation model is fine-tuned according to the first data set; and the simulation electron microscope image generation performance of the first generation model is fine-tuned according to the first data set and the first segmentation model; A corresponding first simulation semantic segmentation map library is created according to a preset second number and a basic particle information library; and a corresponding simulation electron microscope image is generated by the first generation model according to each first simulation semantic segmentation map of the first simulation semantic segmentation map library; and each of the first simulation electron microscope images and the corresponding first simulation semantic segmentation map forms a corresponding second data record; and all of the obtained second data records form a corresponding electron microscope image big data set; the first simulation semantic segmentation map library is composed of a first simulation semantic segmentation map of the second number; the second number is a positive integer not less than 100.

2. The processing method for generating an electron microscope image big data set according to claim 1, characterized in that The first segmentation model at least includes a DeepLab series model, a DETR model, a Segformer model and a SAM series model; the first segmentation model is used for performing semantic segmentation processing on an electron microscope image input by the model and outputting a corresponding semantic segmentation map; The first generation model has processing capability for multiple image generation tasks after completing model pre-training, and the multiple image generation tasks at least include an image generation task of generating a corresponding simulation image according to a semantic segmentation map; The first generation model is used for identifying a first input image type and a first generation image type corresponding to a prompt text input by the model; and when the first input image type is a semantic segmentation map and the first generation image type is an electron microscope image or a simulation electron microscope image, performing corresponding simulation electron microscope image generation processing on the semantic segmentation map input by the model and outputting a corresponding generated image; The basic particle information library includes a plurality of basic particle information; the basic particle information includes particle type, particle shape type, particle color, particle size constraint and particle growth condition constraint; the particle shape type includes spherical shape and irregular polygon.

3. The processing method for generating electron microscopy graph big data sets of claim 2, wherein, The fine-tuning of the electron microscope image semantic segmentation performance of the first segmentation model according to the first data set specifically comprises: Step 31, randomly split the first data set into two sub-data sets based on a preset first split ratio, denoted as a corresponding first training set and a first evaluation set; Wherein, the first training set and the first evaluation set are both composed of a plurality of first data records; the ratio of the total number of records of the first training set to the total number of records of the first evaluation set meets the first split ratio; Step 32, extract the first first data record of the first training set as a corresponding current training record; Step 33, input the first true value electron microscope image of the current training record into the first segmentation model for corresponding electron microscope image semantic segmentation processing to obtain a corresponding first predicted semantic segmentation map; Step 34, bring the first predicted semantic segmentation map and the first semantic segmentation map of the current training record into a preset first model loss function; and based on a preset first model optimizer, a round of fine-tuning is performed on the model parameters of the first segmentation model in the direction of minimizing the first model loss function; Wherein, the first model loss function includes L1 loss function, L2 loss function and cross-entropy loss function; the first model optimizer at least includes ADAM series optimizer, LoRA series optimizer, zero-order series optimizer; Step 35, identify whether the current training record is the last first data record of the first training set; if yes, go to step 36; if not, extract the next first data record of the first training set as a new current training record and return to step 33; Step 36, a round of traversal is performed on all first data records of the first evaluation set; and in this round of traversal, the first data record currently traversed is taken as a corresponding current evaluation record; and the first true value electron microscope image of the current evaluation record is input into the first segmentation model for corresponding electron microscope image semantic segmentation processing to obtain a corresponding second predicted semantic segmentation map; and the second predicted semantic segmentation map and the first semantic segmentation map of the current evaluation record form a corresponding first prediction-label pair; and at the end of this round of traversal, the first accuracy, the first precision, the first recall and the first F1 score are obtained according to the first prediction-label pairs obtained in all rounds of traversal. Step 37, the first accuracy, the first precision, the first recall and the first F1 score are identified; if the first accuracy does not satisfy the preset first accuracy range or the first precision does not satisfy the preset first precision range or the first recall does not satisfy the preset first recall range or the first F1 score does not satisfy the preset first F1 score range, return to step 32 for continuous fine-tuning; if the first accuracy satisfies the first accuracy range, the first precision satisfies the first precision range, the first recall satisfies the first recall range and the first F1 score satisfies the first F1 score range, stop training and confirm that the model fine-tuning of the first segmentation model is completed.

4. The processing method for generating electron microscopy graph big data sets of claim 2, wherein, The first data set and the first segmentation model are used to fine-tune the simulation electron microscope image generation performance of the first generation model, and specifically include: Step 41, the first data set is randomly divided into two sub-data sets based on a preset second segmentation ratio, which are denoted as a corresponding second training set and a second evaluation set; Wherein, the second training set and the second evaluation set are both composed of a plurality of first data records; the ratio of the total number of records of the second training set to the total number of records of the second evaluation set satisfies the second segmentation ratio; Step 42, the first data record of the second training set is extracted as a corresponding current training record; Step 43, a preset simulation electron microscope image generation prompt template is used as a corresponding current prompt text; Wherein, the simulation electron microscope image generation prompt template is a formatted text module; the simulation electron microscope image generation prompt template is used to set the image type of the current model input as a semantic segmentation map, set the image type of the current model generation as an electron microscope image or a simulation electron microscope image, and inform the first generation model to perform corresponding simulation electron microscope image generation processing according to the current model input image and output the simulation electron microscope image generated at the current time as the corresponding current model generated image through a narrative text; Step 44, the current prompt text and the first semantic segmentation map of the current training record are input into the first generation model for corresponding simulation electron microscope image generation processing to obtain a corresponding first predicted simulation electron microscope image; Step 45, the first predicted simulation electron microscope image is input into the first segmentation model for corresponding electron microscope image semantic segmentation processing to obtain a corresponding third predicted semantic segmentation map; Step 46, the third predicted semantic segmentation map and the first semantic segmentation map of the current training record are brought into a preset second model loss function; and based on a preset second model optimizer, the model parameters of the first generation model are fine-tuned in the direction of minimizing the second model loss function; Wherein, the second model loss function includes L1 loss function, L2 loss function and cross-entropy loss function; the second model optimizer includes at least ADAM series optimizer, LoRA series optimizer and zero-order series optimizer; Step 47, identify whether the current training record is the last first data record of the second training set; if yes, go to step 48; if no, extract the next first data record of the second training set as a new current training record and return to step 44; Step 48, perform a round of traversal on all first data records of the second evaluation set; and during the round of traversal, the first data record currently traversed is taken as a corresponding current evaluation record; and the current prompt text and the first semantic segmentation map of the current evaluation record are input into the first generation model to generate a corresponding second predicted simulation electron microscope image through corresponding simulation electron microscope image generation processing; and the second predicted simulation electron microscope image is input into the first segmentation model to generate a corresponding fourth predicted semantic segmentation map through corresponding electron microscope image semantic segmentation processing; and the fourth predicted semantic segmentation map and the first semantic segmentation map of the current evaluation record form a corresponding second prediction-label pair; and at the end of the round of traversal, the second accuracy, the second precision, the second recall and the second F1 score are calculated according to all the second prediction-label pairs obtained through corresponding accuracy, precision, recall and F1 score calculation. Step 49, identify the second accuracy, the second precision, the second recall and the second F1 score obtained; if the second accuracy does not satisfy the preset second accuracy range or the second precision does not satisfy the preset second precision range or the second recall does not satisfy the preset second recall range or the second F1 score does not satisfy the preset second F1 score range, return to step 42 for further fine-tuning; if the second accuracy satisfies the second accuracy range, the second precision satisfies the second precision range, the second recall satisfies the second recall range and the second F1 score satisfies the second F1 score range, stop training and confirm that the model fine-tuning of the first generation model is completed.

5. The processing method for generating electron microscopy graph big data sets of claim 2, wherein, The first simulation semantic segmentation map library is created according to the preset second quantity and the basic particle information library, and specifically includes: Step 51, initialize the first counter to 1; and initialize the first simulation semantic segmentation map library to be empty; Step 52, initialize a corresponding first simulation semantic segmentation map based on a preset semantic segmentation map shape; The semantic segmentation graph shape includes an image width W, an image height H, and a pixel feature vector length D; the shape of the first simulated semantic segmentation graph is WxHxD; the first simulated semantic segmentation graph is composed of WxH first pixel feature vectors with a length of D; the first pixel feature vector corresponds to a pixel point on the first simulated semantic segmentation graph; the vector data of the first pixel feature vector at least includes a first pixel coordinate, a first pixel value, and a first semantic type; all the first pixel values on the first simulated semantic segmentation graph at the initialization time are preset background pixel values, and all the first semantic types are preset background types; Step 53, randomly selecting a plurality of the basic particle information from the basic particle information library to form a corresponding current basic particle information set; Step 54, performing random particle image adding processing on the first simulated semantic segmentation graph based on the current basic particle information set, specifically: performing corresponding particle image adding on a random position on the first simulated semantic segmentation graph based on the particle shape type, the particle color, and the particle size of each basic particle information in the current basic particle information set to obtain a corresponding first particle image; Step 55, performing random particle growth processing on the first simulated semantic segmentation graph, specifically: randomly selecting one or more from all the first particle images obtained as corresponding to-be-grown particle images; and performing particle growth simulation based on the particle growth condition constraint corresponding to each to-be-grown particle image, and each time a new growth position is generated, the current to-be-grown particle image is copied on the current growth position; Step 56, performing displacement processing on part of the stacked particles on the first simulated semantic segmentation graph, specifically: calculating the coincidence degree of each two first particle images on the first simulated semantic segmentation graph to obtain a corresponding first coincidence degree; and adjusting the image position of the two first particle images corresponding to each first coincidence degree that exceeds a preset first coincidence degree threshold until the first coincidence degree of any two first particle images does not exceed the first coincidence degree threshold; Step 57, setting the first semantic type of the first pixel feature vector within the range of each first particle image on the first simulated semantic segmentation graph as the particle type of the basic particle information corresponding to the current first particle image; Step 58, adding the first simulated semantic segmentation graph obtained this time to the first simulated semantic segmentation graph library; and adding 1 to the first counter; Step 59, identifying whether the first counter is greater than the second number; if not, returning to step 52; if yes, outputting the latest first simulated semantic segmentation graph library.

6. The processing method for generating electron microscopy graph big data sets of claim 2, wherein, The first generated model generates a corresponding simulated electron microscope image based on each first simulated semantic segmentation graph of the first simulated semantic segmentation graph library, specifically including: The first simulation semantic segmentation map library is used as a corresponding current semantic segmentation map, a preset simulation electron microscope image generation prompt template is used as a corresponding current prompt text, and the current prompt text and the current semantic segmentation map are input into the first generation model for corresponding simulation electron microscope image generation processing to obtain the first simulation electron microscope image.

7. An apparatus for performing the processing method for generating a large dataset of electron microscopy maps according to any one of claims 1-6, characterized in that, The device comprises a small-scale data set preparation module, a model fine-tuning module and a large data set generation module. The small-scale data set preparation module is used to collect a first number of first true value electron microscope images, perform pixel-level semantic annotation on each first true value electron microscope image by manual annotation or other machine annotation to obtain a corresponding first semantic segmentation map, and form a corresponding first data record from each first true value electron microscope image and the corresponding first semantic segmentation map, and form a corresponding first data set from all obtained first data records; the first number is a positive integer not greater than 10. The model fine-tuning module is used to use a visual semantic segmentation large model that has completed model pre-training as a corresponding first segmentation model, use an SDXL model that has completed model pre-training as a corresponding first generation model, fine-tune the electron microscope image semantic segmentation performance of the first segmentation model according to the first data set, and fine-tune the simulation electron microscope image generation performance of the first generation model according to the first data set and the first segmentation model. The large data set generation module is used to create a corresponding first simulation semantic segmentation map library according to a preset second number and a basic particle information library, generate a corresponding first simulation electron microscope image from each first simulation semantic segmentation map of the first simulation semantic segmentation map library by the first generation model, form a corresponding second data record from each first simulation electron microscope image and the corresponding first simulation semantic segmentation map, and form a corresponding electron microscope image large data set from all obtained second data records; the first simulation semantic segmentation map library comprises a first simulation semantic segmentation map of the second number; the second number is a positive integer not less than 100.

8. An electronic device, comprising: It comprises a memory, a processor and a transceiver. The processor is coupled with the memory, reads and executes instructions in the memory to implement the method of any one of claims 1-6. The transceiver is coupled with the processor and controlled by the processor to perform message transmission and reception. The computer readable storage medium stores computer instructions, which, when executed by a computer, cause the computer to execute the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, which, when executed by a computer, cause the computer to execute the method of any one of claims 1-6.

Images