Processing method and device for generating electron microscope graph big data set

Abstract

The embodiment of the invention relates to a processing method and device for generating an electron microscope graph big data set. The method comprises the following steps: constructing a first data set; taking the first-class visual semantic segmentation large model as a first segmentation model, and taking the SDXL model as a first generation model; performing fine adjustment on the first segmentation model according to the first data set, and then performing fine adjustment on the first generation model according to the first data set and the first segmentation model; constructing a first simulation semantic segmentation image library; and performing simulation electron microscope graph generation processing by a first generation model according to the first simulation semantic segmentation graph library, forming a second data record by each first simulation electron microscope graph and the corresponding first simulation semantic segmentation graph, and forming an electron microscope graph big data set by all the obtained second data records. According to the invention, a large number of electron micrographs can be automatically generated without the help of any manpower or external resources, and the data preparation difficulty of the electron micrograph big data set is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a processing method and device for generating a large data set of electron microscope images. Background Art

[0002] Experimental characterization is the process of studying and quantifying the physical, chemical and structural properties of materials or substances through a variety of analytical techniques and instruments. Experimental characterization can provide a deep understanding of the color, morphology, crystal structure, chemical composition and other characteristics of the material, which is helpful for quality control, performance optimization and new material development. Electron Microscopy (EM), also known as electron microscope, plays an important role in experimental characterization. It observes the structure and morphology of materials at different scales through imaging technology. For example, in the industrial field, scanning electron microscope (SEM) can analyze the morphological characteristics of materials, which is of great significance for the standardization and quality control of materials and products; in the field of scientific research, electron microscope can reveal the microstructure of materials, which helps to understand and develop new materials.

[0003] Traditional electron microscope image characterization and analysis relies on manual processing by software, such as ImageJ or DigitalMicrograph. The problems of this traditional analysis method are obvious: low analysis efficiency, poor consistency of analysis results, and inability to handle a large number of analysis tasks. In order to improve analysis efficiency, reduce analysis errors, improve analysis consistency, and improve the processing capacity of large number of analysis tasks, some researchers have begun to use deep learning models to handle electron microscope image characterization and analysis tasks. From a technical principle, the introduction of deep learning models can indeed achieve the above-mentioned improvement effects, but the premise is that the model used must be fully trained, that is, a large number (at least more than one hundred) or even massive (at least more than one thousand) electron microscope images need to be prepared to truly give play to the technical advantages of the model. However, it is very difficult to prepare a large number of electron microscope images in a real environment. The main reasons are as follows: the acquisition channels of electron microscope images are limited, the acquisition cost is high, and the pixel-level electron microscope image semantic annotation cycle based on the manual annotation mechanism is long. Summary of the invention

[0004] The purpose of the present invention is to address the defects of the prior art and provide a processing method, device, electronic device and computer-readable storage medium for generating a large data set of electron microscope images. The present invention collects only a small amount of true electron micrographs, and performs pixel-level semantic annotation on each true electron micrograph by manual annotation or other machine annotation methods to obtain a corresponding semantic segmentation map, and each true electron micrograph and the corresponding semantic segmentation map form a first data record, and all the obtained first data records form a corresponding small-scale data set, i.e., a first data set; and a large visual semantic segmentation model that has completed model pre-training is used as the first segmentation model, and an SDXL (Stable Diffusion XL) model that has completed model pre-training is used as the first generation model, and the first segmentation model is first fine-tuned using the first data set, and then the first generation model is fine-tuned based on the first data set and the first segmentation model; after the fine-tuning is completed, a basic particle information library is used to randomly create a simulated semantic segmentation map and obtain a corresponding simulated semantic segmentation map library, and then the first generation model performs corresponding simulated electron micrograph generation processing based on each simulated semantic segmentation map of the map library, and then each simulated electron micrograph and the corresponding simulated semantic segmentation map form a second data record, and all the obtained second data records form a large electron micrograph data set. The first segmentation model provided by the present invention is used to fine-tune the first generation model, the basic particle information library is used to construct a diverse simulation semantic segmentation library, and the first generation model is used to generate a diverse electron microscopy image large data set based on the simulation semantic segmentation library; through the present invention, a large number of electron microscopy images can be automatically generated without the aid of any manpower or external resources, which can improve the efficiency of semantic annotation of electron microscopy images, reduce the cost of obtaining electron microscopy images, and reduce the difficulty of data preparation for large electron microscopy image data sets.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present invention provides a processing method for generating a large data set of electron microscope images, the method comprising:

[0006] Collect a first number of first true electron micrographs; perform pixel-level semantic annotation on each of the first true electron micrographs by manual annotation or other machine annotation methods to obtain a corresponding first semantic segmentation map; and form a corresponding first data record from each of the first true electron micrographs and the corresponding first semantic segmentation map; and form a corresponding first data set from all the first data records obtained; the first number is a positive integer not exceeding 10;

[0007] A class of visual semantic segmentation large models that have completed model pre-training is used as the corresponding first segmentation model; and an SDXL model that has completed model pre-training is used as the corresponding first generation model; and the electron microscope image semantic segmentation performance of the first segmentation model is first fine-tuned according to the first data set; and the simulated electron microscope image generation performance of the first generation model is then fine-tuned according to the first data set and the first segmentation model;

[0008] A simulation semantic segmentation map is created according to a preset second quantity and a basic particle information library to obtain a corresponding first simulation semantic segmentation map library; and the first generation model performs corresponding simulation electron microscope image generation processing according to each first simulation semantic segmentation map of the first simulation semantic segmentation map library to obtain a corresponding first simulation electron microscope image; and each of the first simulation electron microscope images and the corresponding first simulation semantic segmentation map form a corresponding second data record; and all the obtained second data records form a corresponding electron microscope image big data set; the first simulation semantic segmentation map library is composed of the second quantity of first simulation semantic segmentation maps; the second quantity is a positive integer not less than 100.

[0009] Preferably, the first segmentation model includes at least a DeepLab series model, a DETR model, a Segformer model, and a SAM series model; the first segmentation model is used to perform semantic segmentation processing on the electron microscope image input by the model and output a corresponding semantic segmentation map;

[0010] The first generation model has the processing capability of a variety of image generation tasks after completing model pre-training, wherein the plurality of image generation tasks at least includes an image generation task of generating a corresponding simulation image according to a semantic segmentation image;

[0011] The first generation model is used to identify the input image type and the generated image type of the prompt text input to the model to obtain the corresponding first input image type and first generated image type; and when the first input image type is a semantic segmentation image and the first generated image type is an electron microscope image or a simulated electron microscope image, the corresponding simulated electron microscope image generation process is performed according to the semantic segmentation image input by the model and the corresponding generated image is output;

[0012] The basic particle information library includes a plurality of basic particle information; the basic particle information includes particle type, particle shape type, particle color, particle size constraint and particle growth condition constraint; the particle shape type includes spherical and irregular polygon;

[0013] Preferably, the fine-tuning of the semantic segmentation performance of the electron microscope image of the first segmentation model according to the first data set specifically includes:

[0014] Step 31, based on a preset first segmentation ratio, randomly split the first data set into two sub-data sets recorded as a corresponding first training set and a first evaluation set;

[0015] Wherein, both the first training set and the first evaluation set are composed of a plurality of the first data records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first segmentation ratio;

[0016] Step 32, extracting the first first data record of the first training set as the corresponding current training record;

[0017] Step 33, inputting the first true electron microscope image of the current training record into the first segmentation model to perform corresponding electron microscope image semantic segmentation processing to obtain a corresponding first predicted semantic segmentation image;

[0018] Step 34, subjecting the first predicted semantic segmentation map and the first semantic segmentation map of the current training record to a preset first model loss function; and based on a preset first model optimizer, performing a round of fine-tuning on the model parameters of the first segmentation model in a direction in which the first model loss function reaches a minimum value;

[0019] Among them, the first model loss function includes L1 loss function, L2 loss function and cross entropy loss function; the first model optimizer includes at least ADAM series optimizer, LoRA series optimizer and zero-order series optimizer;

[0020] Step 35, identifying whether the current training record is the last first data record of the first training set; if so, going to step 36; if not, extracting the next first data record of the first training set as the new current training record and returning to step 33;

[0021] Step 36, perform a round of traversal on all the first data records of the first evaluation set; and in this round of traversal, use the first data record currently traversed as the corresponding current evaluation record; and input the first true value electron microscope image of the current evaluation record into the first segmentation model for corresponding electron microscope image semantic segmentation processing to obtain a corresponding second predicted semantic segmentation map; and form a corresponding first prediction-label pair by the second predicted semantic segmentation map and the first semantic segmentation map of the current evaluation record; and at the end of this round of traversal, perform corresponding accuracy, precision, recall and F1 score calculations based on all the first prediction-label pairs obtained to obtain corresponding first accuracy, first precision, first recall and first F1 score;

[0022] Step 37, identifying the obtained first accuracy, the first precision, the first recall rate and the first F1 score; if the first accuracy does not satisfy the preset first accuracy range or the first accuracy does not satisfy the preset first accuracy range or the first recall rate does not satisfy the preset first recall rate range or the first F1 score does not satisfy the preset first F1 score range, returning to step 32 to continue fine-tuning; if the first accuracy satisfies the first accuracy range and the first accuracy satisfies the first accuracy range and the first recall rate satisfies the first recall rate range and the first F1 score satisfies the first F1 score range, stopping training and confirming that the model fine-tuning of the first segmentation model is completed.

[0023] Preferably, the fine-tuning of the simulated electron microscope image generation performance of the first generation model according to the first data set and the first segmentation model specifically includes:

[0024] Step 41, based on a preset second segmentation ratio, randomly split the first data set into two sub-data sets recorded as corresponding second training sets and second evaluation sets;

[0025] Wherein, both the second training set and the second evaluation set are composed of a plurality of the first data records; the ratio of the total number of records in the second training set to the total number of records in the second evaluation set satisfies the second segmentation ratio;

[0026] Step 42, extracting the first of the first data records in the second training set as the corresponding current training record;

[0027] Step 43, using the preset simulated electron microscope image generation prompt template as the corresponding current prompt text;

[0028] The simulated electron microscope image generation prompt template is a formatted text module; the simulated electron microscope image generation prompt template is used to set the model input image type to a semantic segmentation image, and set the model generated image type to an electron microscope image or a simulated electron microscope image, and inform the first generation model through a narrative text to perform corresponding simulated electron microscope image generation processing according to the model input image and output the generated simulated electron microscope image as the corresponding model generated image;

[0029] Step 44, inputting the current prompt text and the first semantic segmentation map of the current training record into the first generation model to perform corresponding simulated electron microscope image generation processing to obtain a corresponding first predicted simulated electron microscope image;

[0030] Step 45, inputting the first predicted simulated electron microscope image into the first segmentation model to perform corresponding electron microscope image semantic segmentation processing to obtain a corresponding third predicted semantic segmentation image;

[0031] Step 46, subjecting the third predicted semantic segmentation map and the first semantic segmentation map of the current training record to a preset second model loss function; and based on a preset second model optimizer, performing a round of fine-tuning on the model parameters of the first generation model in a direction in which the second model loss function reaches a minimum value;

[0032] Wherein, the second model loss function includes L1 loss function, L2 loss function and cross entropy loss function; the second model optimizer includes at least ADAM series optimizer, LoRA series optimizer and zero-order series optimizer;

[0033] Step 47, identifying whether the current training record is the last of the first data record in the second training set; if so, going to step 48; if not, extracting the next first data record in the second training set as the new current training record and returning to step 44;

[0034] Step 48, perform a round of traversal on all the first data records of the second evaluation set; and in this round of traversal, use the first data record currently traversed as the corresponding current evaluation record; and input the current prompt text and the first semantic segmentation map of the current evaluation record into the first generation model for corresponding simulation electron microscope image generation processing to obtain the corresponding second predicted simulation electron microscope image; and input the second predicted simulation electron microscope image into the first segmentation model for corresponding electron microscope image semantic segmentation processing to obtain the corresponding fourth predicted semantic segmentation map; and form a corresponding second prediction-label pair with the fourth predicted semantic segmentation map and the first semantic segmentation map of the current evaluation record; and at the end of this round of traversal, perform corresponding accuracy, precision, recall and F1 score calculations based on all the obtained second prediction-label pairs to obtain the corresponding second accuracy, second precision, second recall and second F1 score;

[0035] Step 49, identifying the obtained second accuracy, second precision, second recall and second F1 score; if the second accuracy does not meet the preset second accuracy range or the second accuracy does not meet the preset second accuracy range or the second recall does not meet the preset second recall range or the second F1 score does not meet the preset second F1 score range, return to step 42 to continue fine-tuning; if the second accuracy meets the second accuracy range and the second accuracy meets the second accuracy range and the second recall meets the second recall range and the second F1 score meets the second F1 score range, stop training and confirm that the model fine-tuning of the first generated model is completed.

[0036] Preferably, the step of creating a simulated semantic segmentation map according to the preset second quantity and the basic particle information library to obtain a corresponding first simulated semantic segmentation map library specifically includes:

[0037] Step 51, initializing a first counter to 1; and initializing the first simulation semantic segmentation library to be empty;

[0038] Step 52, initializing a corresponding first simulation semantic segmentation map based on a preset semantic segmentation map shape;

[0039] Among them, the shape of the semantic segmentation map includes image width W, image height H and pixel feature vector length D; the shape of the first simulation semantic segmentation map is W×H×D; the first simulation semantic segmentation map is composed of W×H first pixel feature vectors with a length of D; the first pixel feature vectors correspond one-to-one to pixel points on the first simulation semantic segmentation map; the vector data of the first pixel feature vector includes at least a first pixel coordinate, a first pixel value and a first semantic type; all the first pixel values ​​on the first simulation semantic segmentation map during initialization are preset background pixel values, and all the first semantic types are preset background types;

[0040] Step 53, randomly selecting a plurality of the basic particle information from the basic particle information library to form a corresponding current basic particle information set;

[0041] Step 54, performing random particle image addition processing on the first simulation semantic segmentation map based on the current basic particle information set, specifically: adding a corresponding particle image at a random position on the first simulation semantic segmentation map based on the particle shape type, the particle color and the particle size constraints of each of the basic particle information in the current basic particle information set to obtain a corresponding first particle image;

[0042] Step 55, performing random particle growth processing on the first simulation semantic segmentation map, specifically: randomly selecting one or more from all the first particle images obtained as corresponding particle images to be grown; and performing particle growth simulation based on the particle growth condition constraints corresponding to each of the particle images to be grown, and performing an image replication on the current particle image to be grown at the current growth position each time a new growth position is generated on the first simulation semantic segmentation map;

[0043] Step 56, performing shift processing on some stacked particles on the first simulation semantic segmentation map, specifically: calculating the overlap degree of every two first particle images on the first simulation semantic segmentation map to obtain the corresponding first overlap degree; and adjusting the image positions of two first particle images corresponding to each first overlap degree exceeding a preset first overlap degree threshold, until the first overlap degree of any two first particle images does not exceed the first overlap degree threshold;

[0044] Step 57, setting the first semantic type of the first pixel feature vector within the range of each first particle image on the first simulation semantic segmentation map as the particle type of the basic particle information corresponding to the current first particle image;

[0045] Step 58, adding the first simulation semantic segmentation map obtained this time to the first simulation semantic segmentation map library; and adding 1 to the first counter;

[0046] Step 59, identifying whether the first counter is greater than the second number; if not, returning to step 52; if so, outputting the latest first simulation semantic segmentation library.

[0047] Preferably, the first generation model performs corresponding simulation electron microscope image generation processing according to each first simulation semantic segmentation image of the first simulation semantic segmentation image library to obtain the corresponding first simulation electron microscope image, specifically including:

[0048] Each first simulation semantic segmentation map in the first simulation semantic segmentation map library is used as the corresponding current semantic segmentation map; and the preset simulation electron microscope image generation prompt template is used as the corresponding current prompt text; and the current prompt text and the current semantic segmentation map are input into the first generation model to perform corresponding simulation electron microscope image generation processing to obtain the corresponding first simulation electron microscope image.

[0049] A second aspect of an embodiment of the present invention provides a device for implementing the processing method for generating a large data set of electron microscope images described in the first aspect, the device comprising: a small-scale data set preparation module, a model fine-tuning module and a large data set generation module;

[0050] The small-scale data set preparation module is used to collect a first number of first true value electron microscopic images; and perform pixel-level semantic annotation on each of the first true value electron microscopic images by manual annotation or other machine annotation methods to obtain a corresponding first semantic segmentation map; and each of the first true value electron microscopic images and the corresponding first semantic segmentation map constitute a corresponding first data record; and all the first data records obtained constitute a corresponding first data set; the first number is a positive integer not exceeding 10;

[0051] The model fine-tuning module is used to use a type of visual semantic segmentation large model that has completed model pre-training as the corresponding first segmentation model; and use a SDXL model that has completed model pre-training as the corresponding first generation model; and first fine-tune the electron microscope image semantic segmentation performance of the first segmentation model according to the first data set; and then fine-tune the simulated electron microscope image generation performance of the first generation model according to the first data set and the first segmentation model;

[0052] The large data set generation module is used to create a simulation semantic segmentation map according to a preset second number and a basic particle information library to obtain a corresponding first simulation semantic segmentation map library; and the first generation model performs corresponding simulation electron microscope image generation processing according to each first simulation semantic segmentation map of the first simulation semantic segmentation map library to obtain a corresponding first simulation electron microscope image; and each of the first simulation electron microscope images and the corresponding first simulation semantic segmentation map form a corresponding second data record; and all the obtained second data records form a corresponding electron microscope image large data set; the first simulation semantic segmentation map library is composed of the second number of first simulation semantic segmentation maps; the second number is a positive integer not less than 100.

[0053] A third aspect of an embodiment of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0054] The processor is used to be coupled to the memory, read and execute instructions in the memory, so as to implement the method steps described in the first aspect above;

[0055] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0056] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions. When the computer instructions are executed by a computer, the computer executes the instructions of the method described in the first aspect above.

[0057] Embodiments of the present invention provide a processing method, device, electronic device and computer-readable storage medium for generating a large data set of electron microscope images. As can be seen from the above content, the embodiment of the present invention only collects a small amount of true electron micrographs, and performs pixel-level semantic annotation on each true electron micrograph through manual annotation or other machine annotation methods to obtain a corresponding semantic segmentation map, and each true electron micrograph and the corresponding semantic segmentation map form a first data record, and all the obtained first data records form a corresponding small-scale data set, namely, a first data set; and a class of visual semantic segmentation large models that have completed model pre-training is used as the first segmentation model, and an SDXL (StableDiffusion XL) model that has completed model pre-training is used as the first generation model, and the first segmentation model is first fine-tuned using the first data set, and then the first generation model is fine-tuned based on the first data set and the first segmentation model; after the fine-tuning is completed, the basic particle information library is used to randomly create simulated semantic segmentation maps and obtain a corresponding simulated semantic segmentation map library, and then the first generation model performs corresponding simulated electron micrograph generation processing based on each simulated semantic segmentation map of the library, and then each simulated electron micrograph and the corresponding simulated semantic segmentation map form a second data record, and all the obtained second data records form an electron micrograph large data set. The first segmentation model provided in the embodiment of the present invention is used to fine-tune the first generation model, the basic particle information library is used to construct a diverse simulation semantic segmentation library, and the first generation model is used to generate a diverse electron microscopy image large data set based on the simulation semantic segmentation library; through the embodiment of the present invention, a large number of electron microscopy images can be automatically generated without the aid of any manpower or external resources, thereby improving the semantic annotation efficiency of electron microscopy images, reducing the cost of obtaining electron microscopy images, and reducing the difficulty of data preparation for large electron microscopy image data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic diagram of a processing method for generating a large electron microscope image data set provided in the first embodiment of the present invention;

[0059] Figure 2 A module structure diagram of a processing device for generating a large data set of electron microscope images provided in the second embodiment of the present invention;

[0060] Figure 3 A schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] Embodiment 1 of the present invention provides a processing method for generating a large data set of electron microscope images, such as Figure 1 As shown in the schematic diagram of a processing method for generating a large electron microscope image data set provided in the first embodiment of the present invention, the method mainly comprises the following steps:

[0063] Step 1, collecting a first number of first true value electron microscope images; and performing pixel-level semantic annotation on each first true value electron microscope image by manual annotation or other machine annotation methods to obtain a corresponding first semantic segmentation map; and each first true value electron microscope image and the corresponding first semantic segmentation map form a corresponding first data record; and all the obtained first data records form a corresponding first data set;

[0064] The first number is a positive integer not exceeding 10.

[0065] Here, the embodiment of the present invention can collect true electron microscope images through electron microscope experiments, or through other public true value image acquisition channels; because the first number <10, the difficulty of collecting only a few true electron microscope images is relatively small; the first data set obtained by data collection and manual / machine annotation is composed of a first number of first data records, and each first data record is composed of a first true electron microscope image and a first semantic segmentation image.

[0066] Step 2: a large visual semantic segmentation model that has completed model pre-training is used as the corresponding first segmentation model; and a SDXL model that has completed model pre-training is used as the corresponding first generation model; and the semantic segmentation performance of the electron microscope image of the first segmentation model is fine-tuned according to the first data set; and the simulated electron microscope image generation performance of the first generation model is fine-tuned according to the first data set and the first segmentation model;

[0067] Specifically, it includes: step 21, taking a class of visual semantic segmentation large models that have completed model pre-training as the corresponding first segmentation model;

[0068] Here, the visual semantic segmentation large model of the embodiment of the present invention, i.e., the first segmentation model, includes at least a DeepLab series model, a DETR model, a Segformer model, and a SAM series model; the first segmentation model is used to perform semantic segmentation processing on the electron microscope image input to the model and output the corresponding semantic segmentation map; the embodiment of the present invention uses a pre-trained model to greatly shorten the model training cycle and improve the model training efficiency;

[0069] Step 22, using an SDXL model that has completed model pre-training as the corresponding first generation model;

[0070] Here, the first generation model of the embodiment of the present invention is implemented based on the SDXL model. The detailed model structure, pre-training scheme and other information of the SDXL model can be understood by referring to the public technical document A "SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis" and the related technical document B "High-Resolution Image Synthesis with Latent Diffusion Models", which will not be further described here. It should be noted that, it can be seen from the technical documents A and B that the first generation model has the processing capability of multiple image generation tasks after completing the model pre-training. The multiple image generation tasks mentioned here at least include the image generation task of generating corresponding simulation images according to the semantic segmentation map. In addition, it can be seen from the technical documents A and B that the first generation model belongs to a type of generation model from text to image, and a prompt text needs to be input when the semantic segmentation map is input to it.

[0071] That is to say: the first generation model of the embodiment of the present invention is used to identify the input image type and the generated image type of the prompt text input by the model to obtain the corresponding first input image type and first generated image type; and when the first input image type is a semantic segmentation image and the first generated image type is an electron microscope image or a simulated electron microscope image, the corresponding simulated electron microscope image generation process is performed according to the semantic segmentation image input by the model and the corresponding generated image is output;

[0072] Step 23, first fine-tuning the semantic segmentation performance of the electron microscope image of the first segmentation model according to the first data set;

[0073] Specifically, it includes: step 231, randomly dividing the first data set into two sub-data sets based on a preset first division ratio and recording them as a corresponding first training set and a first evaluation set;

[0074] The first segmentation ratio is a preset ratio parameter, such as 8:2; the first training set and the first evaluation set are both composed of a plurality of first data records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first segmentation ratio;

[0075] Step 232, extracting the first first data record of the first training set as the corresponding current training record;

[0076] Step 233, inputting the first true electron microscope image of the current training record into the first segmentation model to perform corresponding electron microscope image semantic segmentation processing to obtain a corresponding first predicted semantic segmentation image;

[0077] Step 234, bringing the first predicted semantic segmentation map and the first semantic segmentation map of the current training record into a preset first model loss function; and based on the preset first model optimizer, performing a round of fine-tuning on the model parameters of the first segmentation model in a direction to minimize the first model loss function;

[0078] Among them, the first model loss function includes L1 loss function, L2 loss function and cross entropy loss function; the first model optimizer includes at least ADAM series optimizer, LoRA series optimizer and zero-order series optimizer;

[0079] Step 235, identifying whether the current training record is the last first data record of the first training set; if so, going to step 236; if not, extracting the next first data record of the first training set as the new current training record and returning to step 233;

[0080] Step 236, perform a round of traversal on all the first data records of the first evaluation set; and in this round of traversal, use the first data record currently traversed as the corresponding current evaluation record; and input the first true value electron microscope image of the current evaluation record into the first segmentation model to perform corresponding electron microscope image semantic segmentation processing to obtain a corresponding second predicted semantic segmentation image; and form a corresponding first prediction-label pair by the second predicted semantic segmentation image and the first semantic segmentation image of the current evaluation record; and at the end of this round of traversal, perform corresponding accuracy, precision, recall and F1 score calculations based on all the obtained first prediction-label pairs to obtain the corresponding first accuracy, first precision, first recall and first F1 score;

[0081] Step 237, identifying the obtained first accuracy, first precision, first recall and first F1 score; if the first accuracy does not satisfy the preset first accuracy range or the first precision does not satisfy the preset first precision range or the first recall does not satisfy the preset first recall range or the first F1 score does not satisfy the preset first F1 score range, returning to step 232 to continue fine-tuning; if the first accuracy satisfies the first accuracy range and the first precision satisfies the first precision range and the first recall satisfies the first recall range and the first F1 score satisfies the first F1 score range, stopping training and confirming that the model fine-tuning of the first segmentation model is completed;

[0082] Here, the first accuracy range, the first precision range, the first recall range, and the first F1 score range are four preset value ranges;

[0083] Step 24, fine-tuning the simulated electron microscope image generation performance of the first generation model according to the first data set and the first segmentation model;

[0084] Specifically, it includes: step 241, randomly dividing the first data set into two sub-data sets based on a preset second division ratio and recording them as corresponding second training sets and second evaluation sets;

[0085] The second segmentation ratio is a preset ratio parameter, such as 8:2; the second training set and the second evaluation set are both composed of a plurality of first data records; the ratio of the total number of records in the second training set to the total number of records in the second evaluation set satisfies the second segmentation ratio;

[0086] Step 242, extracting the first first data record of the second training set as the corresponding current training record;

[0087] Step 243, using the preset simulated electron microscope image generation prompt template as the corresponding current prompt text;

[0088] Here, the simulated electron microscope image generation prompt template of the embodiment of the present invention is a formatted text module; the simulated electron microscope image generation prompt template is used to set the model input image type to a semantic segmentation image, and set the model generated image type to an electron microscope image or a simulated electron microscope image, and inform the first generation model through a narrative text to perform corresponding simulated electron microscope image generation processing according to the model input image and output the generated simulated electron microscope image as the corresponding model generated image at that time;

[0089] Step 244, inputting the current prompt text and the first semantic segmentation map of the current training record into the first generation model to perform corresponding simulated electron microscope image generation processing to obtain a corresponding first predicted simulated electron microscope image;

[0090] Step 245, inputting the first predicted simulated electron microscope image into the first segmentation model to perform corresponding electron microscope image semantic segmentation processing to obtain a corresponding third predicted semantic segmentation image;

[0091] Step 246, bringing the third predicted semantic segmentation map and the first semantic segmentation map of the current training record into a preset second model loss function; and based on the preset second model optimizer, performing a round of fine-tuning on the model parameters of the first generation model in a direction to minimize the second model loss function;

[0092] Among them, the second model loss function includes L1 loss function, L2 loss function and cross entropy loss function; the second model optimizer includes at least ADAM series optimizer, LoRA series optimizer and zero-order series optimizer;

[0093] Step 247, identifying whether the current training record is the last first data record of the second training set; if so, going to step 248; if not, extracting the next first data record of the second training set as the new current training record and returning to step 244;

[0094] Step 248, perform a round of traversal on all the first data records of the second evaluation set; and in this round of traversal, use the currently traversed first data record as the corresponding current evaluation record; and input the current prompt text and the first semantic segmentation map of the current evaluation record into the first generation model for corresponding simulation electron microscope image generation processing to obtain the corresponding second predicted simulation electron microscope image; and input the second predicted simulation electron microscope image into the first segmentation model for corresponding electron microscope image semantic segmentation processing to obtain the corresponding fourth predicted semantic segmentation map; and form a corresponding second prediction-label pair from the fourth predicted semantic segmentation map and the first semantic segmentation map of the current evaluation record; and at the end of this round of traversal, perform corresponding accuracy, precision, recall and F1 score calculations based on all the obtained second prediction-label pairs to obtain the corresponding second accuracy, second precision, second recall and second F1 score;

[0095] Step 249, identifying the obtained second accuracy, second precision, second recall and second F1 score; if the second accuracy does not meet the preset second accuracy range or the second precision does not meet the preset second precision range or the second recall does not meet the preset second recall range or the second F1 score does not meet the preset second F1 score range, returning to step 242 to continue fine-tuning; if the second accuracy meets the second accuracy range and the second precision meets the second precision range and the second recall meets the second recall range and the second F1 score meets the second F1 score range, stopping training and confirming that the model fine-tuning of the first generated model is completed;

[0096] Here, the second accuracy range, the second precision range, the second recall range and the second F1 score range are four preset value ranges.

[0097] Step 3, creating a simulation semantic segmentation map according to the preset second quantity and the basic particle information library to obtain a corresponding first simulation semantic segmentation library; and using the first generation model to perform corresponding simulation electron microscope image generation processing according to each first simulation semantic segmentation map of the first simulation semantic segmentation library to obtain a corresponding first simulation electron microscope image; and each first simulation electron microscope image and the corresponding first simulation semantic segmentation map form a corresponding second data record; and all the obtained second data records form a corresponding electron microscope image large data set;

[0098] Specifically, it includes: step 31, creating a simulation semantic segmentation map according to the preset second quantity and the basic particle information library to obtain a corresponding first simulation semantic segmentation map library;

[0099] Among them, the basic particle information library includes multiple basic particle information; the basic particle information includes particle type, particle shape type, particle color, particle size constraint and particle growth condition constraint; the type range of particle types can be customized according to specific application requirements, and generally can include electrons, various ions, various atoms, various molecules, various groups, etc.; particle shape types include spherical and irregular polygons, and the irregular polygons here can be further subdivided based on specific application requirements; particle size constraints restrict the maximum and minimum size / volume of the current particle and give corresponding boundary values; particle growth condition constraints restrict various growth conditions of the current particle and the corresponding growth termination conditions;

[0100] The first simulation semantic segmentation map library is composed of a second number of first simulation semantic segmentation maps; the second number is a positive integer not less than 100, and the second number has no maximum value limit and can be customized according to specific application requirements;

[0101] Specifically, it includes: step 311, initializing a first counter to 1; and initializing a first simulation semantic segmentation library to be empty;

[0102] Step 312, initializing a corresponding first simulated semantic segmentation map based on a preset semantic segmentation map shape;

[0103] Here, the shape of the semantic segmentation map in the embodiment of the present invention includes an image width W, an image height H, and a pixel feature vector length D, and these three parameters W, H, and D are all positive integers;

[0104] The shape of the first simulated semantic segmentation map of the embodiment of the present invention is W×H×D; the first simulated semantic segmentation map is composed of W×H first pixel feature vectors with a length of D; wherein the first pixel feature vectors correspond one-to-one to the pixel points on the first simulated semantic segmentation map; the vector data of the first pixel feature vectors at least include the first pixel coordinates, the first pixel value and the first semantic type; all the first pixel values ​​on the first simulated semantic segmentation map at the time of initialization are the preset background pixel values, and all the first semantic types are the preset background types; the background pixel value here is generally set to 0, that is, black by default, and can also be customized according to specific application requirements;

[0105] Step 313, randomly selecting a plurality of basic particle information from the basic particle information library to form a corresponding current basic particle information set;

[0106] Step 314, performing random particle image adding processing on the first simulation semantic segmentation map based on the current basic particle information set;

[0107] Specifically, it includes: adding a corresponding particle image to a random position on the first simulation semantic segmentation map based on the particle shape type, particle color and particle size constraints of each basic particle information of the current basic particle information set to obtain a corresponding first particle image;

[0108] Step 315, performing random particle growth processing on the first simulation semantic segmentation map;

[0109] Specifically, it includes: randomly selecting one or more of all the first particle images obtained as corresponding particle images to be grown; and performing particle growth simulation based on the particle growth condition constraints corresponding to each particle image to be grown, and performing an image copy of the current particle image to be grown at the current growth position each time a new growth position is generated on the first simulation semantic segmentation map;

[0110] Step 316, performing a displacement process on some of the stacked particles on the first simulation semantic segmentation map;

[0111] Specifically, the method comprises: calculating the overlap degree of each two first particle images on the first simulation semantic segmentation map to obtain a corresponding first overlap degree; and adjusting the image positions of the two first particle images corresponding to each first overlap degree exceeding a preset first overlap degree threshold value, until the first overlap degree of any two first particle images does not exceed the first overlap degree threshold value;

[0112] Here, the first overlap threshold is a preset threshold parameter;

[0113] Step 317, setting the first semantic type of the first pixel feature vector within the range of each first particle image on the first simulation semantic segmentation map as the particle type of the basic particle information corresponding to the current first particle image;

[0114] Step 318, adding the first simulation semantic segmentation map obtained this time to the first simulation semantic segmentation map library; and adding 1 to the first counter;

[0115] Step 319, identifying whether the first counter is greater than the second number; if not, returning to step 312; if so, outputting the latest first simulation semantic segmentation library;

[0116] Step 32, the first generation model performs corresponding simulation electron microscope image generation processing according to each first simulation semantic segmentation image in the first simulation semantic segmentation image library to obtain a corresponding first simulation electron microscope image;

[0117] Specifically, it includes: taking each first simulation semantic segmentation map of the first simulation semantic segmentation map library as the corresponding current semantic segmentation map; taking a preset simulation electron microscope image generation prompt template as the corresponding current prompt text; and inputting the current prompt text and the current semantic segmentation map into the first generation model to perform corresponding simulation electron microscope image generation processing to obtain the corresponding first simulation electron microscope image;

[0118] Step 33, each first simulated electron microscope image and the corresponding first simulated semantic segmentation image form a corresponding second data record; and all the obtained second data records form a corresponding electron microscope image big data set.

[0119] Figure 2 The module structure diagram of a processing device for generating a large electron microscope image data set provided in the second embodiment of the present invention, the device is a terminal device or server that implements the aforementioned method embodiment, and can also be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiment, for example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 2 As shown, the device includes: a small-scale data set preparation module 201, a model fine-tuning module 202 and a large data set generation module 203.

[0120] The small-scale data set preparation module 201 is used to collect a first number of first true electron micrographs; and to perform pixel-level semantic annotation on each first true electron micrograph through manual annotation or other machine annotation methods to obtain a corresponding first semantic segmentation map; and to form a corresponding first data record from each first true electron micrograph and the corresponding first semantic segmentation map; and to form a corresponding first data set from all the obtained first data records; the first number is a positive integer not exceeding 10.

[0121] The model fine-tuning module 202 is used to use a type of visual semantic segmentation large model that has completed model pre-training as the corresponding first segmentation model; and use an SDXL model that has completed model pre-training as the corresponding first generation model; and first fine-tune the electron microscope image semantic segmentation performance of the first segmentation model according to the first data set; and then fine-tune the simulated electron microscope image generation performance of the first generation model according to the first data set and the first segmentation model.

[0122] The large data set generation module 203 is used to create a simulation semantic segmentation map according to a preset second number and a basic particle information library to obtain a corresponding first simulation semantic segmentation map library; and the first generation model performs corresponding simulation electron microscope image generation processing according to each first simulation semantic segmentation map of the first simulation semantic segmentation map library to obtain a corresponding first simulation electron microscope image; and each first simulation electron microscope image and the corresponding first simulation semantic segmentation map form a corresponding second data record; and all the obtained second data records form a corresponding electron microscope image large data set; the first simulation semantic segmentation map library is composed of a second number of first simulation semantic segmentation maps; the second number is a positive integer not less than 100.

[0123] An embodiment of the present invention provides a processing device for generating a large data set of electron microscope images, which can execute the method steps in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0124] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the small-scale data set preparation module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the above-mentioned module is determined. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0125] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0126] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the above method embodiments are generated. The above computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above-mentioned computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above-mentioned computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.) methods. The above-mentioned computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The above-mentioned available medium can be a magnetic medium (such as a floppy disk, a hard disk, a tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0127] Figure 3 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. The electronic device may be a terminal device or a server for implementing the method of the aforementioned embodiment, or may be a terminal device or a server for implementing the method of the aforementioned embodiment connected to the aforementioned terminal device or server. Figure 3As shown, the electronic device may include: a processor 301 (such as a CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transceiver 303. Various instructions may be stored in the memory 302 to complete various processing functions and implement the processing steps described in the aforementioned embodiment method. Preferably, the electronic device involved in the embodiment of the present invention also includes: a power supply 304, a system bus 305 and a communication port 306. The system bus 305 is used to realize the communication connection between components. The above-mentioned communication port 306 is used for connecting and communicating between the electronic device and other peripherals.

[0128] exist Figure 3 The system bus 305 mentioned in the figure can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to realize the communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM), and may also include non-volatile memory (Non-Volatile Memory), such as at least one disk storage.

[0129] The above-mentioned processor can be a general-purpose processor, including a central processing unit CPU, a network processor (Network Processor, NP), a graphics processing unit (Graphics Processing Unit, GPU), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0130] It should be noted that an embodiment of the present invention further provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the method and processing process provided in the above embodiments.

[0131] Embodiments of the present invention provide a processing method, device, electronic device and computer-readable storage medium for generating a large data set of electron microscope images. As can be seen from the above content, the embodiment of the present invention only collects a small amount of true electron micrographs, and performs pixel-level semantic annotation on each true electron micrograph through manual annotation or other machine annotation methods to obtain a corresponding semantic segmentation map, and each true electron micrograph and the corresponding semantic segmentation map form a first data record, and all the obtained first data records form a corresponding small-scale data set, namely, a first data set; and a class of visual semantic segmentation large models that have completed model pre-training is used as the first segmentation model, and an SDXL (StableDiffusion XL) model that has completed model pre-training is used as the first generation model, and the first segmentation model is first fine-tuned using the first data set, and then the first generation model is fine-tuned based on the first data set and the first segmentation model; after the fine-tuning is completed, the basic particle information library is used to randomly create simulated semantic segmentation maps and obtain a corresponding simulated semantic segmentation map library, and then the first generation model performs corresponding simulated electron micrograph generation processing based on each simulated semantic segmentation map of the library, and then each simulated electron micrograph and the corresponding simulated semantic segmentation map form a second data record, and all the obtained second data records form an electron micrograph large data set. The first segmentation model provided in the embodiment of the present invention is used to fine-tune the first generation model, the basic particle information library is used to construct a diverse simulation semantic segmentation library, and the first generation model is used to generate a diverse electron microscopy image large data set based on the simulation semantic segmentation library; through the embodiment of the present invention, a large number of electron microscopy images can be automatically generated without the aid of any manpower or external resources, thereby improving the semantic annotation efficiency of electron microscopy images, reducing the cost of obtaining electron microscopy images, and reducing the difficulty of data preparation for large electron microscopy image data sets.

[0132] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0133] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A processing method for generating a large data set of electron microscope images, characterized in that: The method comprises: Collect a first number of first true electron micrographs; perform pixel-level semantic annotation on each of the first true electron micrographs by manual annotation or other machine annotation methods to obtain a corresponding first semantic segmentation map; and form a corresponding first data record from each of the first true electron micrographs and the corresponding first semantic segmentation map; and form a corresponding first data set from all the first data records obtained; the first number is a positive integer not exceeding 10; A class of visual semantic segmentation large models that have completed model pre-training is used as the corresponding first segmentation model; and an SDXL model that has completed model pre-training is used as the corresponding first generation model; and the electron microscope image semantic segmentation performance of the first segmentation model is first fine-tuned according to the first data set; and the simulated electron microscope image generation performance of the first generation model is then fine-tuned according to the first data set and the first segmentation model; A simulation semantic segmentation map is created according to a preset second quantity and a basic particle information library to obtain a corresponding first simulation semantic segmentation map library; and the first generation model performs corresponding simulation electron microscope image generation processing according to each first simulation semantic segmentation map of the first simulation semantic segmentation map library to obtain a corresponding first simulation electron microscope image; and each of the first simulation electron microscope images and the corresponding first simulation semantic segmentation map form a corresponding second data record; and all the obtained second data records form a corresponding electron microscope image big data set; the first simulation semantic segmentation map library is composed of the second quantity of first simulation semantic segmentation maps; the second quantity is a positive integer not less than 100.

2. The processing method for generating a large data set of electron microscope images according to claim 1, characterized in that: The first segmentation model includes at least a DeepLab series model, a DETR model, a Segformer model, and a SAM series model; the first segmentation model is used to perform semantic segmentation processing on the electron microscope image input by the model and output a corresponding semantic segmentation map; The first generation model has the processing capability of a variety of image generation tasks after completing model pre-training, wherein the plurality of image generation tasks at least includes an image generation task of generating a corresponding simulation image according to a semantic segmentation image; The first generation model is used to identify the input image type and the generated image type of the prompt text input to the model to obtain the corresponding first input image type and first generated image type; and when the first input image type is a semantic segmentation image and the first generated image type is an electron microscope image or a simulated electron microscope image, the corresponding simulated electron microscope image generation process is performed according to the semantic segmentation image input by the model and the corresponding generated image is output; The basic particle information library includes multiple basic particle information; the basic particle information includes particle type, particle shape type, particle color, particle size constraint and particle growth condition constraint; the particle shape type includes spherical and irregular polygon.

3. The processing method for generating a large data set of electron microscope images according to claim 2, characterized in that: The fine-tuning of the semantic segmentation performance of the electron microscope image of the first segmentation model according to the first data set specifically includes: Step 31, based on a preset first segmentation ratio, randomly split the first data set into two sub-data sets recorded as a corresponding first training set and a first evaluation set; Wherein, both the first training set and the first evaluation set are composed of a plurality of the first data records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first segmentation ratio; Step 32, extracting the first first data record of the first training set as the corresponding current training record; Step 33, inputting the first true electron microscope image of the current training record into the first segmentation model to perform corresponding electron microscope image semantic segmentation processing to obtain a corresponding first predicted semantic segmentation image; Step 34, subjecting the first predicted semantic segmentation map and the first semantic segmentation map of the current training record to a preset first model loss function; and based on a preset first model optimizer, performing a round of fine-tuning on the model parameters of the first segmentation model in a direction in which the first model loss function reaches a minimum value; Among them, the first model loss function includes L1 loss function, L2 loss function and cross entropy loss function; the first model optimizer includes at least ADAM series optimizer, LoRA series optimizer and zero-order series optimizer; Step 35, identifying whether the current training record is the last first data record of the first training set; if so, going to step 36; if not, extracting the next first data record of the first training set as the new current training record and returning to step 33; Step 36, perform a round of traversal on all the first data records of the first evaluation set; and in this round of traversal, use the first data record currently traversed as the corresponding current evaluation record; and input the first true value electron microscope image of the current evaluation record into the first segmentation model for corresponding electron microscope image semantic segmentation processing to obtain a corresponding second predicted semantic segmentation map; and form a corresponding first prediction-label pair by the second predicted semantic segmentation map and the first semantic segmentation map of the current evaluation record; and at the end of this round of traversal, perform corresponding accuracy, precision, recall and F1 score calculations based on all the first prediction-label pairs obtained to obtain corresponding first accuracy, first precision, first recall and first F1 score; Step 37, identifying the obtained first accuracy, the first precision, the first recall rate and the first F1 score; if the first accuracy does not satisfy the preset first accuracy range or the first accuracy does not satisfy the preset first accuracy range or the first recall rate does not satisfy the preset first recall rate range or the first F1 score does not satisfy the preset first F1 score range, returning to step 32 to continue fine-tuning; if the first accuracy satisfies the first accuracy range and the first accuracy satisfies the first accuracy range and the first recall rate satisfies the first recall rate range and the first F1 score satisfies the first F1 score range, stopping training and confirming that the model fine-tuning of the first segmentation model is completed.

4. The processing method for generating a large data set of electron microscope images according to claim 2, characterized in that: The step of fine-tuning the simulated electron microscope image generation performance of the first generation model according to the first data set and the first segmentation model specifically includes: Step 41, based on a preset second segmentation ratio, randomly split the first data set into two sub-data sets recorded as corresponding second training sets and second evaluation sets; Wherein, both the second training set and the second evaluation set are composed of a plurality of the first data records; the ratio of the total number of records in the second training set to the total number of records in the second evaluation set satisfies the second segmentation ratio; Step 42, extracting the first of the first data records in the second training set as the corresponding current training record; Step 43, using the preset simulated electron microscope image generation prompt template as the corresponding current prompt text; The simulated electron microscope image generation prompt template is a formatted text module; the simulated electron microscope image generation prompt template is used to set the model input image type to a semantic segmentation image, and set the model generated image type to an electron microscope image or a simulated electron microscope image, and inform the first generation model through a narrative text to perform corresponding simulated electron microscope image generation processing according to the model input image and output the generated simulated electron microscope image as the corresponding model generated image; Step 44, inputting the current prompt text and the first semantic segmentation map of the current training record into the first generation model to perform corresponding simulated electron microscope image generation processing to obtain a corresponding first predicted simulated electron microscope image; Step 45, inputting the first predicted simulated electron microscope image into the first segmentation model to perform corresponding electron microscope image semantic segmentation processing to obtain a corresponding third predicted semantic segmentation image; Step 46, subjecting the third predicted semantic segmentation map and the first semantic segmentation map of the current training record to a preset second model loss function; and based on a preset second model optimizer, performing a round of fine-tuning on the model parameters of the first generation model in a direction in which the second model loss function reaches a minimum value; Wherein, the second model loss function includes L1 loss function, L2 loss function and cross entropy loss function; the second model optimizer includes at least ADAM series optimizer, LoRA series optimizer and zero-order series optimizer; Step 47, identifying whether the current training record is the last of the first data record in the second training set; if so, going to step 48; if not, extracting the next first data record in the second training set as the new current training record and returning to step 44; Step 48, perform a round of traversal on all the first data records of the second evaluation set; and in this round of traversal, use the first data record currently traversed as the corresponding current evaluation record; and input the current prompt text and the first semantic segmentation map of the current evaluation record into the first generation model for corresponding simulation electron microscope image generation processing to obtain the corresponding second predicted simulation electron microscope image; and input the second predicted simulation electron microscope image into the first segmentation model for corresponding electron microscope image semantic segmentation processing to obtain the corresponding fourth predicted semantic segmentation map; and form a corresponding second prediction-label pair with the fourth predicted semantic segmentation map and the first semantic segmentation map of the current evaluation record; and at the end of this round of traversal, perform corresponding accuracy, precision, recall and F1 score calculations based on all the obtained second prediction-label pairs to obtain the corresponding second accuracy, second precision, second recall and second F1 score; Step 49, identifying the obtained second accuracy, second precision, second recall and second F1 score; if the second accuracy does not meet the preset second accuracy range or the second accuracy does not meet the preset second accuracy range or the second recall does not meet the preset second recall range or the second F1 score does not meet the preset second F1 score range, returning to step 42 to continue fine-tuning; if the second accuracy meets the second accuracy range and the second accuracy meets the second accuracy range and the second recall meets the second recall range and the second F1 score meets the second F1 score range, stop training and confirm that the model fine-tuning of the first generated model is completed.

5. The processing method for generating a large data set of electron microscope images according to claim 2, characterized in that: The step of creating a simulated semantic segmentation graph according to the preset second quantity and the basic particle information library to obtain a corresponding first simulated semantic segmentation graph library specifically includes: Step 51, initializing a first counter to 1; and initializing the first simulation semantic segmentation library to be empty; Step 52, initializing a corresponding first simulation semantic segmentation map based on a preset semantic segmentation map shape; Among them, the shape of the semantic segmentation map includes image width W, image height H and pixel feature vector length D; the shape of the first simulation semantic segmentation map is W×H×D; the first simulation semantic segmentation map is composed of W×H first pixel feature vectors with a length of D; the first pixel feature vectors correspond one-to-one to pixel points on the first simulation semantic segmentation map; the vector data of the first pixel feature vector includes at least a first pixel coordinate, a first pixel value and a first semantic type; all the first pixel values ​​on the first simulation semantic segmentation map during initialization are preset background pixel values, and all the first semantic types are preset background types; Step 53, randomly selecting a plurality of the basic particle information from the basic particle information library to form a corresponding current basic particle information set; Step 54, performing random particle image addition processing on the first simulation semantic segmentation map based on the current basic particle information set, specifically: adding a corresponding particle image at a random position on the first simulation semantic segmentation map based on the particle shape type, the particle color and the particle size constraints of each of the basic particle information in the current basic particle information set to obtain a corresponding first particle image; Step 55, performing random particle growth processing on the first simulation semantic segmentation map, specifically: randomly selecting one or more from all the first particle images obtained as corresponding particle images to be grown; and performing particle growth simulation based on the particle growth condition constraints corresponding to each of the particle images to be grown, and performing an image replication on the current particle image to be grown at the current growth position each time a new growth position is generated on the first simulation semantic segmentation map; Step 56, performing shift processing on some stacked particles on the first simulation semantic segmentation map, specifically: calculating the overlap degree of every two first particle images on the first simulation semantic segmentation map to obtain the corresponding first overlap degree; and adjusting the image positions of two first particle images corresponding to each first overlap degree exceeding a preset first overlap degree threshold, until the first overlap degree of any two first particle images does not exceed the first overlap degree threshold; Step 57, setting the first semantic type of the first pixel feature vector within the range of each first particle image on the first simulation semantic segmentation map as the particle type of the basic particle information corresponding to the current first particle image; Step 58, adding the first simulation semantic segmentation map obtained this time to the first simulation semantic segmentation map library; and adding 1 to the first counter; Step 59, identifying whether the first counter is greater than the second number; if not, returning to step 52; if so, outputting the latest first simulation semantic segmentation library.

6. The processing method for generating a large data set of electron microscope images according to claim 2, characterized in that: The first generation model performs corresponding simulation electron microscope image generation processing according to each first simulation semantic segmentation image of the first simulation semantic segmentation image library to obtain the corresponding first simulation electron microscope image, specifically including: Each first simulation semantic segmentation map in the first simulation semantic segmentation map library is used as the corresponding current semantic segmentation map; and the preset simulation electron microscope image generation prompt template is used as the corresponding current prompt text; and the current prompt text and the current semantic segmentation map are input into the first generation model to perform corresponding simulation electron microscope image generation processing to obtain the corresponding first simulation electron microscope image.

7. A device for executing the processing method for generating a large electron microscope image data set according to any one of claims 1 to 6, characterized in that: The device comprises: a small-scale data set preparation module, a model fine-tuning module and a large data set generation module; The small-scale data set preparation module is used to collect a first number of first true value electron microscopic images; and perform pixel-level semantic annotation on each of the first true value electron microscopic images by manual annotation or other machine annotation methods to obtain a corresponding first semantic segmentation map; and each of the first true value electron microscopic images and the corresponding first semantic segmentation map constitute a corresponding first data record; and all the first data records obtained constitute a corresponding first data set; the first number is a positive integer not exceeding 10; The model fine-tuning module is used to use a type of visual semantic segmentation large model that has completed model pre-training as the corresponding first segmentation model; and use a SDXL model that has completed model pre-training as the corresponding first generation model; and first fine-tune the electron microscope image semantic segmentation performance of the first segmentation model according to the first data set; and then fine-tune the simulated electron microscope image generation performance of the first generation model according to the first data set and the first segmentation model; The large data set generation module is used to create a simulation semantic segmentation map according to a preset second number and a basic particle information library to obtain a corresponding first simulation semantic segmentation map library; and the first generation model performs corresponding simulation electron microscope image generation processing according to each first simulation semantic segmentation map of the first simulation semantic segmentation map library to obtain a corresponding first simulation electron microscope image; and each of the first simulation electron microscope images and the corresponding first simulation semantic segmentation map form a corresponding second data record; and all the obtained second data records form a corresponding electron microscope image large data set; the first simulation semantic segmentation map library is composed of the second number of first simulation semantic segmentation maps; the second number is a positive integer not less than 100.

8. An electronic device, characterized in that: include: memory, processors, and transceivers; The processor is used to couple with the memory, read and execute instructions in the memory, so as to implement the method according to any one of claims 1 to 6; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a computer, the computer executes the method according to any one of claims 1 to 6.

Images