Method and device for identifying scale of electron microscope graph and scale information combination features

Abstract

The embodiment of the invention relates to a scale of an electron microscope graph and a scale information combination feature identification method and device. The method comprises the following steps: constructing a target detection model based on YOLOv8; constructing a first data set by taking the summarized four types of scale-information combinations and the two types of relative relationships as conditions, and training a target detection model based on the data set; after training is finished, performing target detection on any electron microscope graph based on a target detection model, extracting target detection frames of scale marks and text information in a target detection frame sequence to form a first detection frame sequence and a second detection frame sequence, and identifying scale features based on the first detection frame sequence; the scale information text is identified based on the information bar position of the current electron microscope graph and the first and second detection frame sequences, and the scale feature and the scale information text form a combined feature. According to the invention, the identification efficiency and the identification quality can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and device for identifying the scale of an electron microscope image and the combined features of scale information. Background Art

[0002] Electron Microscopy (EM), also known as electron microscopy, is a high-resolution imaging technology that is widely used in materials science, biology and medical research. The output image of an electron microscope is called an electron microscope image. Although there are certain parameter / information differences between the electron microscope images provided by various electron microscope manufacturers, they all have a scale parameter, which is composed of a pixel scale and scale information (i.e., distance unit information). At present, most of the scale parameter recognition tasks of electron microscope images are still based on manual measurement. However, due to the influence of human factors and the level of manual measurement, this conventional processing method is bound to have the problem that it is difficult to improve both the recognition efficiency and the recognition quality. Summary of the invention

[0003] The purpose of the present invention is to address the defects of the prior art and provide a method, device, electronic device and computer-readable storage medium for identifying the scale and scale information combination features of an electron microscope image. The present invention preliminarily builds an electron microscope image target detection model based on the YOLOv8 model, which is recorded as a first target detection model, and summarizes the combined features of the scale-information combination in the electron microscope image to obtain four types of scale-information combinations, and summarizes the relative position relationship between the scale-information combination and the information column in the electron microscope image to obtain two types of relative relationships, and constructs a first data set with the four types of scale-information combinations and the two types of relative relationships as training image construction conditions, and trains a first target detection model based on the first data set; then after the model training is completed, firstly, target detection is performed on any electron microscope image based on the first target detection model, and then target detection frames of the scale mark type and the text information type in the target detection frame sequence are respectively extracted to form the first and second detection frame sequences, and then the scale feature (including the total number of scale marks and the scale pixel length) is identified based on the first detection frame sequence, and then the information column position is identified for the current electron microscope image, and the scale information text (including numbers and distance units) is identified based on the information column position and the first and second detection frame sequences, and the scale feature and the scale information text form a corresponding combined feature. On the one hand, the present invention can automatically identify the scale parameters of the electron microscope image, thereby improving the recognition efficiency and recognition quality; on the other hand, by constructing a diverse and enhanced training data set to train the model, the generalization of the model can be improved and the stability of the recognition quality can be ensured.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present invention provides a method for identifying the scale and scale information combination features of an electron microscope image, the method comprising:

[0005] Based on the YOLOv8 model, an electron microscope image target detection model is constructed and recorded as the corresponding first target detection model; and the combined features of the ruler-information combination composed of the ruler and the ruler information in the electron microscope image are summarized to obtain the corresponding four types of ruler-information combinations; and the relative position relationship between the ruler-information combination and the information column in the electron microscope image is summarized to obtain the corresponding two types of relative relationships; and the corresponding model training data set is constructed using the four types of ruler-information combinations and the two types of relative relationships as training image construction conditions to obtain the corresponding first data set; and the first target detection model is trained based on the first data set;

[0006] After the model training is completed, any electron microscope image is received as the corresponding first electron microscope image;

[0007] Performing target detection processing on the first electron microscope image based on the first target detection model to obtain a corresponding first target detection frame sequence; the first target detection frame sequence includes multiple first target detection frames; the first target detection frame includes the coordinates of the center point of the first detection frame, the width of the first detection frame, the height of the first detection frame, the type of the first detection frame and the confidence of the first detection type; the first detection frame type includes a ruler mark type, a text information type and other types;

[0008] Extracting the first target detection frames whose first detection frame type is a scale mark type from the first target detection frame sequence to form a corresponding first detection frame sequence, and extracting the first target detection frames whose first detection frame type is a text information type to form a corresponding second detection frame sequence;

[0009] Based on the first detection frame sequence, scale feature recognition is performed to obtain a corresponding first scale feature; and the information column position of the first electron microscope image is recognized to obtain a corresponding first information column position; and based on the first information column position, the first detection frame sequence and the second detection frame sequence, scale information text recognition is performed to obtain a corresponding first scale information text; and the first scale feature and the first scale information text form a corresponding first combined feature.

[0010] Preferably, the ruler-information combination in the electron microscope image consists of the ruler and the ruler information; wherein the ruler contains one or more ruler marks in the horizontal direction of the image, and all ruler marks of the ruler have the same shape type, consistent size, and the same color; and the ruler information is a text information containing numbers and distance units;

[0011] The four types of ruler-information combinations summarized include the first type of combination, the second type of combination, the third type of combination and the fourth type of combination; the combination feature of the first type of combination is: the ruler includes two ruler marks, and the width of the ruler mark is less than the height, and the two ruler marks are connected by a horizontal line, and the ruler information is at the center of the horizontal line; the combination feature of the second type of combination is: the ruler includes two ruler marks, and the width of the ruler mark is less than the height, and the two ruler marks are connected by a horizontal line, and the ruler information is above or below the horizontal line; the combination feature of the third type of combination is: the ruler includes more than two ruler marks, and the width of the ruler mark is less than the height, and there is no horizontal connection line between every two adjacent ruler marks, and the ruler information is above or below the ruler; the combination feature of the fourth type of combination is: the ruler includes only one ruler mark, and the width of the ruler mark is greater than the height, and the ruler information is above or below the ruler;

[0012] The two types of relative relationships summarized include a first type of relative relationship and a second type of relative relationship; the first type of relative relationship means that the ruler-information combination is within the range of the information column; the second type of relative relationship means that the ruler-information combination is outside the range of the information column;

[0013] The first target detection model is used to perform corresponding ruler mark and text information target detection processing on the model input electron microscope image and output a corresponding target detection frame sequence; the target detection frame sequence includes multiple target detection frames; each of the target detection frames includes the coordinates of the center point of the detection frame, the width of the detection frame, the height of the detection frame, the type of the detection frame and the confidence of the detection type; the detection frame type includes a ruler mark type, a text information type and other types;

[0014] The first data set includes multiple first data records; the first data record includes a first training electron microscope image and a first label detection frame sequence; the first label detection frame sequence includes multiple first label detection frames; the first label detection frame includes the coordinates of the center point of the first label frame, the width of the first label frame, the height of the first label frame, the type of the first label frame and the confidence of the first label type; the first label frame type includes a ruler mark type, a text information type and other types; the confidence of the first label type is 1;

[0015] The first scale feature includes a total number of first scale marks and a first scale pixel length.

[0016] Preferably, the constructing of a corresponding model training data set using the four types of scale-information combinations and the two types of relative relationships as training image construction conditions to obtain a corresponding first data set specifically includes:

[0017] Step 31, pre-collecting a plurality of original electron microscope images without scales, without any text information, and without information columns to form a corresponding first original electron microscope image set;

[0018] Wherein, the first original electron microscope image set includes a plurality of first original electron microscope images;

[0019] Step 32, taking the first of the first original electron microscopic images in the first original electron microscopic image set as the corresponding current bottom electron microscopic image;

[0020] Step 33, based on the random electron microscope image generation rule, random electron microscope image generation processing is performed according to the preset first specified number N and the current bottom electron microscope image to obtain corresponding N generated electron microscope images;

[0021] The random electron microscope image generation rule requirements are as follows:

[0022] For each generated electron microscope image, firstly, one of the two types of relative relationships is randomly selected as the corresponding relative relationship type; and then the relative relationship type is identified;

[0023] If the relative relationship type is the first type of relative relationship, randomly select an information bar sub-image from the preset information bar sub-image set and add it to the current underlying electron microscope image; and the area covered by the current information bar sub-image on the current underlying electron microscope image is recorded as the corresponding current information bar area, and the area not covered by the current information bar sub-image is recorded as the corresponding current other area; and randomly add one or more random text boxes in the current other area, and randomly set the text box background color, text box transparency, text box font, text box font color, text box text and text box position in each of the random text boxes; and randomly select an integer from the value range of 0 to the second specified number M as the corresponding current random number R, and when the current random number R is not 0, select an integer from the preset manufacturer LOGO A manufacturer's LOGO image is randomly selected from the image set as the corresponding current manufacturer's LOGO image, and R random image frames of random size that can adaptively adjust the image in the frame are randomly selected at R positions on the current underlying electron microscope image, and the image in the frame of each random image frame is set as the corresponding current manufacturer's LOGO image; and one of the four types of scale-information combinations is selected as the corresponding current combination; if the current combination is the first, second, third or fourth type of combination, a corresponding first, second, third or fourth type of scale-information image is selected from the preset first, second, third or fourth type of scale-information image set and added to an optional position in the current information column area; and the electron microscope image obtained after completing all the additions this time is taken as a corresponding generated electron microscope image;

[0024] If the relative relationship type is the second type of relative relationship, one or more random text boxes are randomly added to the current underlying electron microscope image, and the text box background color, text box transparency, text box font, text box font color, text box text and text box position in each of the random text boxes are randomly set; and an integer is randomly selected from the value range of 0 to the second specified number M as the corresponding current random number R, and when the current random number R is not 0, a manufacturer LOGO image is randomly selected from the manufacturer LOGO image set as the corresponding current manufacturer LOGO image, and a random integer is randomly selected on the current underlying electron microscope image. Add R random image frames of random size and capable of adaptively adjusting the images in the frames to R positions, and set the images in the frames of each random image frame to the corresponding current manufacturer LOGO image; and select one of the four types of scale-information combinations as the corresponding current combination; if the current combination is the first, second, third or fourth type of combination, select a corresponding first, second, third or fourth type of scale-information image from the preset first, second, third or fourth type of scale-information image set and add it to an optional position of the current underlying electron microscope image; and use the electron microscope image obtained after completing all the additions this time as a corresponding generated electron microscope image;

[0025] The information bar sub-image set includes a plurality of the information bar sub-images; the image backgrounds of all the information bar sub-images are monochrome; the foreground image information of all the information bar sub-images includes a series of text information and the foreground text information does not contain any image information related to the ruler and ruler information; the foreground image information of some of the information bar sub-images also includes one or more horizontal and / or vertical dividing lines; the ratio of foreground pixels to background pixels of all the information bar sub-images is less than 1; when adding the information bar sub-image to the current underlying electron microscope image, the size of the information bar sub-image is scaled proportionally according to the width ratio of the information bar sub-image to the current underlying electron microscope image, and the scaled information bar sub-image is added to the bottom or top of the current underlying electron microscope image;

[0026] The manufacturer LOGO image set includes a plurality of the manufacturer LOGO images;

[0027] The first type of ruler-information graph set includes a plurality of the first type of ruler-information graphs, and the rulers and ruler information features corresponding to the first type of ruler-information graphs meet the first type of combination; the second type of ruler-information graph set includes a plurality of the second type of ruler-information graphs, and the rulers and ruler information features corresponding to the second type of ruler-information graphs meet the second type of combination; the third type of ruler-information graph set includes a plurality of the third type of ruler-information graphs, and the rulers and ruler information features corresponding to the third type of ruler-information graphs meet the third type of combination; the fourth type of ruler-information graph set includes a plurality of the fourth type of ruler-information graphs, and the rulers and ruler information features corresponding to the fourth type of ruler-information graphs meet the fourth type of combination;

[0028] When adding the first, second, third or fourth type of ruler-information image to the current information bar area, appropriately scaling and moving the ruler-information image to ensure that the ruler-information image does not overlap with the foreground text information of the current information bar sub-image and does not exceed the current information bar area;

[0029] When adding the first, second, third or fourth type of ruler-information map to the current bottom electron microscope image, appropriately resizing and moving the ruler-information map to ensure that the ruler-information map added does not overlap with any of the random text boxes and any of the random image boxes on the current bottom electron microscope image, and does not exceed the image range of the current bottom electron microscope image;

[0030] Step 34, perform a round of traversal on the N generated electron microscope images obtained this time; and in this round of traversal, use the generated electron microscope image currently traversed as the corresponding current electron microscope image; and based on each of the random text boxes on the current electron microscope image, mark a corresponding first label detection box, and set the first label box center point coordinates, the first label box width and the first label box height of the corresponding first label detection box based on the text box center point coordinates, text box width and text box height of each of the random text boxes, and set the first label box type corresponding to each of the random text boxes to the text information type, and set each of the random text boxes to the first label detection box center point coordinates, the text box width and the text box height. The confidence of the first label type corresponding to the text box is set to 1; and when the number of the random image boxes on the current electron microscope image is not zero, a corresponding first label detection box is marked based on each of the random image boxes, and the first label box center point coordinates, the first label box width and the first label box height of the corresponding first label detection box are set based on the image box center point coordinates, the image box width and the image box height of each of the random image boxes, and the first label box type corresponding to each of the random image boxes is set to other types, and the confidence of the first label type corresponding to each of the random image boxes is set to 1; and when the relative relationship type corresponding to the current electron microscope image is the first type of relative relationship, each continuous text information on the information column sub-image in the current electron microscope image is marked with a corresponding label detection box by manual marking or other machine marking methods to obtain a plurality of corresponding first label detection boxes whose first label box type is set to the text information type and whose first label type confidence is set to 1; and the ruler information text in the unique ruler-information image on the current electron microscope image is marked as a first label box type set to the text information type and whose first label type confidence is set to 1 by manual marking or other machine marking methods. label detection frame; and mark each ruler mark sub-image area in the unique ruler-information image on the current electron microscope image into a first label detection frame in which the first label frame type is set to a ruler mark type and the first label type confidence is set to 1 by manual marking or other machine marking methods; and use the current electron microscope image as a corresponding first training electron microscope image; and form a corresponding first label detection frame sequence by all the first label detection frames corresponding to the current electron microscope image; and form a corresponding first data record by the first training electron microscope image and the first label detection frame sequence corresponding to the current electron microscope image;

[0031] Step 35, after a round of traversal of the N generated electron microscope images obtained this time is completed, identify whether the current bottom electron microscope image is the last of the first original electron microscope images in the first original electron microscope image set; if so, go to step 36; if not, extract the next first original electron microscope image in the first original electron microscope image set as the new current bottom electron microscope image and return to step 33;

[0032] Step 36, based on the preset data enhancement rules, data enhancement processing is performed on each of the first training electron micrographs currently obtained to obtain a corresponding plurality of first enhanced electron micrographs; and each of the first enhanced electron micrographs is used as the corresponding current enhanced electron micrograph, and the first training electron micrograph corresponding to the current enhanced electron micrograph is used as the corresponding current parent image, and the first label detection frame sequence corresponding to the current parent image is used as the corresponding parent image label detection frame sequence, and based on the spatial transformation relationship between the current enhanced electron micrograph and the current parent image, the first label frame center point coordinates, the first label frame width and the first label frame height of each of the first label detection frames in the parent image label detection frame sequence are adaptively adjusted, and the label detection frame sequence obtained after the adjustment is used as a new first label detection frame sequence, and the current enhanced electron micrograph is used as a new first training electron micrograph, and a new first data record is formed by a pair of new first training electron micrographs and the first label detection frame sequence corresponding to the current enhanced electron micrograph;

[0033] The data enhancement rules include at least a horizontal flip enhancement rule, an HSV color transformation enhancement rule and a low-angle rotation enhancement rule;

[0034] Step 36: All the first data records finally obtained form the corresponding first data set.

[0035] Preferably, the training of the first target detection model based on the first data set specifically includes:

[0036] Step 41, based on a preset first segmentation ratio, randomly split the first data set into two data subsets recorded as a corresponding first training set and a first evaluation set;

[0037] Wherein, both the first training set and the first evaluation set are composed of a plurality of the first data records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first segmentation ratio;

[0038] Step 42, extracting the first first data record of the first training set as the corresponding current training record;

[0039] Step 43, bringing the first training electron microscope image of the current training record into the first target detection model to perform corresponding scale marking and text information target detection processing to obtain a corresponding first predicted target detection frame sequence;

[0040] Step 44, bringing the first predicted target detection frame sequence and the first label detection frame sequence of the current training record into a preset first model loss function; and performing a round of optimization processing on the model parameters of the first target detection model in a direction in which the first model loss function reaches a minimum value based on a preset first model optimizer;

[0041] Wherein, the first model loss function is implemented based on the model loss function of the YOLOv8 model; the first model optimizer includes at least an SGD optimizer and an ADAM optimizer;

[0042] Step 45, identifying whether the current training record is the last first data record of the first training set; if so, going to step 46; if not, extracting the next first data record of the first training set as the new current training record and returning to step 43;

[0043] Step 46, perform a round of traversal on all the first data records of the first evaluation set; and in this round of traversal, use the first data record currently traversed as the corresponding current evaluation record; and bring the first training electron microscope image of the current evaluation record into the first target detection model for corresponding scale marking and text information target detection processing to obtain a corresponding second predicted target detection frame sequence; and form a corresponding first prediction-label pair by the second predicted target detection frame sequence and the first label detection frame sequence of the current training record; and at the end of this round of traversal, the model performance evaluation method based on the YOLOv8 model performs corresponding accuracy, precision, recall and F1 score calculations on all the first prediction-label pairs obtained to obtain the corresponding first accuracy, first precision, first recall and first F1 score;

[0044] Step 47, based on the preset first accuracy range, first precision range, first recall rate range and first F1 score range, compare the obtained first accuracy, first precision, first recall rate and first F1 score; if the first accuracy does not satisfy the corresponding first accuracy range or the first accuracy does not satisfy the corresponding first precision range or the first recall rate does not satisfy the corresponding first recall rate range or the first F1 score does not satisfy the corresponding first F1 score range, return to step 42 to continue training; if the first accuracy satisfies the corresponding first accuracy range and the first precision satisfies the corresponding first precision range and the first recall rate satisfies the corresponding first recall rate range and the first F1 score satisfies the corresponding first F1 score range, stop training and confirm that the model training is completed.

[0045] Preferably, the performing scale feature recognition based on the first detection frame sequence to obtain the corresponding first scale feature specifically includes:

[0046] Step 51, extracting the sub-images covered by each of the first target detection frames of the first detection frame sequence on the first electron microscope image as corresponding first detection frame sub-images; and performing binary image conversion on each of the first detection frame sub-images to obtain corresponding first binary images;

[0047] Wherein, the first binary image includes a plurality of first pixel points; the pixel values ​​of the first pixel points include preset foreground pixel values ​​and background pixel values; the first pixel points whose pixel values ​​are the foreground pixel values ​​are recorded as corresponding first ruler mark pixel points, and the first pixel points whose pixel values ​​are the background pixel values ​​are recorded as corresponding first background pixel points;

[0048] Step 52, copying the first electron microscope image to obtain a corresponding first copied electron microscope image; firstly resetting the pixel values ​​of all pixels of the first copied electron microscope image to the background pixel values; then resetting the pixel values ​​of the pixels on the first copied electron microscope image corresponding to the first scale mark pixels of each of the first binary images to the foreground pixel values; and taking the first copied electron microscope image with the background and foreground pixel settings completed as the corresponding second binary image;

[0049] The second binary image includes a plurality of second pixel points; the pixel values ​​of the second pixel points include the foreground pixel value and the background pixel value; the second pixel point whose pixel value is the foreground pixel value is recorded as the corresponding second ruler mark pixel point, and the second pixel point whose pixel value is the background pixel value is recorded as the corresponding second background pixel point;

[0050] Step 53, based on a preset gradient operator, the horizontal gradient of the second binary image is calculated to obtain a corresponding first gradient image; and the sum of the pixel values ​​of each column of the first gradient image is calculated to obtain the corresponding sum of the pixel values ​​of the first column; and the absolute value of the sum of the pixel values ​​of each first column is taken to obtain the corresponding absolute value of the first column; and a two-dimensional coordinate plane is constructed with the absolute value of the pixel value sum as the vertical axis and the column index as the horizontal axis as the corresponding first coordinate plane; and on the first coordinate plane, corresponding coordinate point marking is performed based on the obtained absolute values ​​of each first column to obtain the corresponding first marking point; and all the obtained first marking points are sequentially connected to obtain the corresponding first marking curve; and the curve peak point identification of the first marking curve is performed to obtain the corresponding multiple first peak points; and the first peak points on the leftmost and rightmost sides are used as the corresponding starting peak point and ending peak point; and the integer column index closest to the starting and ending peak points on the first coordinate plane is used as the corresponding starting column index and ending column index; and the difference between the ending column index and the starting column index is used as the corresponding first scale pixel length;

[0051] The first gradient map includes a plurality of third pixel points; the third pixel points correspond to the second pixel points one by one; the pixel value of the third pixel point corresponds to the horizontal gradient value of the corresponding second pixel point;

[0052] Step 54, counting the total number of the first peak points to obtain the corresponding total number of first peaks; and calculating the corresponding total number of the first scale marks based on the total number of the first peaks, the total number of first scale marks=the total number of first peaks / 2;

[0053] Step 55: obtain the first scale feature corresponding to the total number of the first scale marks and the thickest pixel length of the first scale.

[0054] Preferably, the step of performing information column position recognition on the first electron microscope image to obtain the corresponding first information column position specifically includes:

[0055] Step 61, record the width of the first electron microscope image as width W; and extract the pixel values ​​of all pixel points in each row of the first electron microscope image and sort them in descending order to form a corresponding first pixel value sequence;

[0056] The first pixel value sequence is composed of a plurality of first pixel values ​​p i Composition, 1≤index i≤W;

[0057] Step 62, identifying the parity of the width W, if the width W is an odd number, setting the corresponding 50th percentile index a=(W+1) / 2, if the width W is an even number, setting the corresponding 50th percentile index a=W / 2; and setting the corresponding 5th percentile index b=floor(W×5 / 100), 95th percentile index c=floor(W×95 / 100), floor() is a floor rounding function;

[0058] Step 63: in each of the first pixel value sequences, the first pixel value p whose index i matches the 50th percentile index a, the 5th percentile index b, and the 95th percentile index c respectively. i＝a , the first pixel value p i＝b and the first pixel value p i＝c Extracted as the corresponding 50th percentile pixel value d 50 , 5th percentile pixel value d5 and 95th percentile pixel value d 95 ; and based on the 50th percentile pixel value d 50 , the 5th percentile pixel value d5 and the 95th percentile pixel value d 95 Set the corresponding first line parameter e = min(|d5-d 50 |,|d 50 -d 95 |);

[0059] Step 64: Check whether each of the first row parameters e is less than a preset first parameter threshold e. hold Identify; if the current first row parameter e is less than the first parameter threshold e hold , then the corresponding row of the first row parameter e in the first electron microscope image is recorded as the information column row;

[0060] Step 65, identifying the total number of the information column rows obtained; if the total number of the information column rows obtained is zero, setting the corresponding first information column position to be empty; if the total number of the information column rows obtained is greater than zero, identifying the maximum and minimum number of rows among all the information column rows obtained to obtain the corresponding maximum number of information column rows and minimum number of information column rows, and merging all rows from the minimum number of information column rows to the maximum number of information column rows in the first electron microscope image to obtain the obtained merged area as the corresponding first information column area, and extracting the four vertex coordinates of the first information column area in the first electron microscope image to form the corresponding first information column position.

[0061] Preferably, the performing scale information text recognition based on the first information column position, the first detection frame sequence, and the second detection frame sequence to obtain the corresponding first scale information text specifically includes:

[0062] Step 71, identifying whether the first information column position is empty; if the first information column position is not empty, recording the image area covered by the first information column position in the first electron microscope image as the corresponding current information column area, and recording the first target detection frame in the second detection frame sequence whose center point coordinates are within the current information column area as the corresponding to-be-selected detection frame; if the first information column position is empty, recording all the first target detection frames in the second detection frame sequence as the corresponding to-be-selected detection frames;

[0063] Step 72, calculating the mean of the first detection frame center point coordinates of all the first target detection frames of the first detection frame sequence and using the calculation result as the corresponding first scale center point coordinates; calculating the straight-line distance between the first detection frame center point coordinates of each of the to-be-selected detection frames and the first scale center point coordinates to obtain a corresponding first distance; and sorting the to-be-selected detection frames in ascending order of the first distances to form a corresponding to-be-selected detection frame sequence;

[0064] Step 73, extracting the first of the to-be-selected detection frames in the to-be-selected detection frame sequence as the corresponding current to-be-selected detection frame;

[0065] Step 74, extracting the sub-image area covered by the current to-be-selected detection frame in the first electron microscope image as the corresponding current sub-image; and performing text information recognition on the current sub-image based on a preset optical character recognition tool to obtain a corresponding first recognized text;

[0066] Step 75, identifying whether the first recognition text contains numbers and distance units; if the first recognition text contains both numbers and distance units, setting the corresponding first scale information text as the first recognition text, and going to step 76; if the first recognition text does not contain numbers or distance units, identifying whether the current to-be-selected detection frame is the last to-be-selected detection frame in the to-be-selected detection frame sequence, if so, setting the corresponding first scale information text to be empty and going to step 76, if not, extracting the next to-be-selected detection frame in the to-be-selected detection frame sequence as the new current to-be-selected detection frame and returning to step 74;

[0067] Step 76: output the first scale information text finally obtained as the recognition result.

[0068] A second aspect of an embodiment of the present invention provides a device for implementing the method for identifying the scale and scale information combination features of an electron microscope image described in the first aspect, the device comprising: a model building and training module, an electron microscope image receiving module, a target detection module, a detection frame clustering module and a combination feature recognition module;

[0069] The model construction and training module is used to construct an electron microscope image target detection model based on the YOLOv8 model, recorded as the corresponding first target detection model; and summarize the combined features of the scale-information combination composed of the scale and the scale information in the electron microscope image to obtain the corresponding four types of scale-information combinations; and summarize the relative position relationship between the scale-information combination and the information column in the electron microscope image to obtain the corresponding two types of relative relationships; and use the four types of scale-information combinations and the two types of relative relationships as training image construction conditions to construct a corresponding model training data set to obtain a corresponding first data set; and train the first target detection model based on the first data set;

[0070] The electron microscope image receiving module is used to receive any electron microscope image as the corresponding first electron microscope image after the model training is completed;

[0071] The target detection module is used to perform target detection processing on the first electron microscope image based on the first target detection model to obtain a corresponding first target detection frame sequence; the first target detection frame sequence includes multiple first target detection frames; the first target detection frame includes the coordinates of the center point of the first detection frame, the width of the first detection frame, the height of the first detection frame, the type of the first detection frame and the confidence of the first detection type; the first detection frame type includes a ruler mark type, a text information type and other types;

[0072] The detection frame clustering module is used to extract the first target detection frames whose first detection frame type is a scale mark type in the first target detection frame sequence to form a corresponding first detection frame sequence, and extract the first target detection frames whose first detection frame type is a text information type to form a corresponding second detection frame sequence;

[0073] The combined feature recognition module is used to perform scale feature recognition based on the first detection frame sequence to obtain a corresponding first scale feature; and to perform information column position recognition on the first electron microscope image to obtain a corresponding first information column position; and to perform scale information text recognition based on the first information column position, the first detection frame sequence and the second detection frame sequence to obtain a corresponding first scale information text; and the first scale feature and the first scale information text form a corresponding first combined feature.

[0074] A third aspect of an embodiment of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0075] The processor is used to be coupled to the memory, read and execute instructions in the memory, so as to implement the method steps described in the first aspect above;

[0076] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0077] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions. When the computer instructions are executed by a computer, the computer executes the instructions of the method described in the first aspect above.

[0078] The embodiments of the present invention provide a method, device, electronic device and computer-readable storage medium for identifying the scale and scale information combination features of an electron microscope image. As can be seen from the above content, the embodiment of the present invention preliminarily constructs an electron microscope image target detection model based on the YOLOv8 model, recorded as the first target detection model, and summarizes the combined features of the scale-information combination in the electron microscope image to obtain four types of scale-information combinations, and summarizes the relative position relationship between the scale-information combination and the information bar in the electron microscope image to obtain two types of relative relationships, and constructs a first data set with the four types of scale-information combinations and the two types of relative relationships as training image construction conditions, and trains the first target detection model based on the first data set; then after the model training is completed, firstly perform target detection on any electron microscope image based on the first target detection model, and then extract the target detection frames of the scale mark type and the text information type in the target detection frame sequence to form the first and second detection frame sequences, and then identify the scale feature (including the total number of scale marks and the scale pixel length) based on the first detection frame sequence, and then identify the information bar position for the current electron microscope image, and identify the scale information text (including numbers and distance units) based on the information bar position and the first and second detection frame sequences, and the scale feature and the scale information text form a corresponding combination feature. The embodiment of the present invention can automatically identify the scale parameters of the electron microscope image, thereby improving the recognition efficiency and recognition quality; the embodiment of the present invention trains the model by constructing a diverse and enhanced training data set, thereby improving the generalization of the model and ensuring the stability of the recognition quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 A schematic diagram of a method for identifying the scale and scale information combination features of an electron microscope image provided in the first embodiment of the present invention;

[0080] Figure 2 A schematic diagram of a ruler, ruler information, and an information column provided in the first embodiment of the present invention;

[0081] Figure 3 A schematic diagram of four types of scale-information combination examples and two types of relative relationship examples provided in the first embodiment of the present invention;

[0082] Figure 4 Schematic diagrams of two examples of generating electron microscope images provided in the first embodiment of the present invention;

[0083] Figure 5 A module structure diagram of a device for identifying the scale and scale information combination features of an electron microscope image provided in the second embodiment of the present invention;

[0084] Figure 6 A schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0085] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0086] Embodiment 1 of the present invention provides a method for identifying the scale and scale information combination features of an electron microscope image, such as Figure 1 As shown in the schematic diagram of a method for identifying the scale and scale information combination features of an electron microscope image provided by the first embodiment of the present invention, the method mainly includes the following steps:

[0087] Step 1, based on the YOLOv8 model, build an electron microscope image target detection model recorded as the corresponding first target detection model; and summarize the combined features of the scale-information combination composed of the scale and the scale information in the electron microscope image to obtain the corresponding four types of scale-information combinations; and summarize the relative position relationship between the scale-information combination and the information column in the electron microscope image to obtain the corresponding two types of relative relationships; and use the four types of scale-information combinations and the two types of relative relationships as training image construction conditions to construct a corresponding model training data set to obtain the corresponding first data set; and train the first target detection model based on the first data set;

[0088] Specifically comprising: step 11, building an electron microscope image target detection model based on the YOLOv8 model and recording it as the corresponding first target detection model;

[0089] Here, the ruler-information combination in the electron microscope image of the embodiment of the present invention is composed of a ruler and ruler information; wherein the ruler contains one or more ruler marks in the horizontal direction of the image, and all ruler marks of the ruler have the same shape type, consistent size, and the same color; the ruler information is a text information containing numbers and distance units; Figure 2 This is a schematic diagram of the ruler, ruler information and information bar provided in the first embodiment of the present invention. The ruler mark, ruler, ruler information, etc. can be referred to as Figure 2 To understand;

[0090] The first target detection model of the embodiment of the present invention is used to perform corresponding ruler mark and text information target detection processing on the model input electron microscope image and output a corresponding target detection frame sequence; wherein the target detection frame sequence includes multiple target detection frames; each target detection frame includes the coordinates of the center point of the detection frame, the width of the detection frame, the height of the detection frame, the type of the detection frame and the confidence of the detection type; the detection frame type includes a ruler mark type, a text information type and other types;

[0091] Step 12, summarizing the combined features of the scale-information combination consisting of the scale and the scale information in the electron microscope image to obtain the corresponding four types of scale-information combinations;

[0092] Here, the four types of scale-information combinations summarized in the embodiment of the present invention include a first type combination, a second type combination, a third type combination and a fourth type combination; wherein,

[0093] The combination characteristics of the first type of combination are: the ruler includes two ruler marks, the width of the ruler mark is smaller than the height, the two ruler marks are connected by a horizontal line, and the ruler information is at the center of the horizontal line;

[0094] The combination characteristics of the second type of combination are: the ruler includes two ruler marks, the width of the ruler mark is smaller than the height, the two ruler marks are connected by a horizontal line, and the ruler information is above or below the horizontal line;

[0095] The combination characteristics of the third type of combination are: the ruler includes more than two ruler marks, the width of the ruler mark is smaller than the height, there is no horizontal connecting line between every two adjacent ruler marks, and the ruler information is above or below the ruler;

[0096] The combination characteristics of the fourth type of combination are: the ruler includes only one ruler mark, the width of the ruler mark is greater than the height, and the ruler information is above or below the ruler;

[0097] Figure 3 Schematic diagram of four types of ruler-information combination examples and two types of relative relationship examples provided in the first embodiment of the present invention. The above four types of ruler-information combination can refer to Figure 3 To understand;

[0098] Step 13, summarizing the relative position relationship between the scale-information combination and the information column in the electron microscope image to obtain two corresponding relative relationships;

[0099] Here, the two types of relative relationships summarized in the embodiment of the present invention include a first type of relative relationship and a second type of relative relationship; the first type of relative relationship indicates that the ruler-information combination is within the range of the information column; the second type of relative relationship indicates that the ruler-information combination is outside the range of the information column; here, the information column can refer to Figure 2 To understand, the two types of relative relationships can be referred to Figure 3 To understand;

[0100] Step 14, constructing a corresponding model training data set using four types of scale-information combinations and two types of relative relationships as training image construction conditions to obtain a corresponding first data set;

[0101] The first data set includes a plurality of first data records; the first data record includes a first training electron microscope image and a first label detection frame sequence; the first label detection frame sequence includes a plurality of first label detection frames; the first label detection frame includes the coordinates of the center point of the first label frame, the width of the first label frame, the height of the first label frame, the type of the first label frame and the confidence of the first label type; the type of the first label frame includes a ruler mark type, a text information type and other types; the confidence of the first label type is 1;

[0102] Specifically, it includes: step 141, pre-collecting a plurality of original electron microscope images without scales, any text information, and no information bar to form a corresponding first original electron microscope image set;

[0103] Wherein, the first original electron microscope image set includes a plurality of first original electron microscope images;

[0104] Step 142, taking the first first original electron microscopic image of the first original electron microscopic image set as the corresponding current bottom electron microscopic image;

[0105] Step 143, based on the random electron microscope image generation rule, random electron microscope image generation processing is performed according to the preset first specified number N and the current bottom electron microscope image to obtain corresponding N generated electron microscope images;

[0106] Here, the first specified number N is a preset positive integer;

[0107] The random electron microscope image generation rule of the embodiment of the present invention requires that when generating each generated electron microscope image:

[0108] Step A1, first randomly select one of the two types of relative relationships as the corresponding relative relationship type;

[0109] Step A2, identifying the relative relationship type;

[0110] Step A3, if the relative relationship type is the first type of relative relationship, randomly select an information bar sub-image from the preset information bar sub-image set and add it to the current bottom electron microscope image; and the area covered by the current information bar sub-image on the current bottom electron microscope image is recorded as the corresponding current information bar area, and the area not covered by the current information bar sub-image is recorded as the corresponding current other area; and randomly add one or more random text boxes in the current other area, and randomly set the text box background color, text box transparency, text box font, text box font color, text box text and text box position in each random text box; and randomly select an integer from the value range of 0 to the second specified number M as the corresponding current random number R, and when the current random number R is not 0, select an integer from the preset manufacturer L A manufacturer's LOGO image is randomly selected from the OGO image set as the corresponding current manufacturer's LOGO image, and R random image frames of random size that can adaptively adjust the image in the frame are randomly selected at R positions on the current underlying electron microscope image, and the image in each random image frame is set as the corresponding current manufacturer's LOGO image; and one of the four types of ruler-information combinations is selected as the corresponding current combination; if the current combination is the first, second, third or fourth type of combination, a corresponding first, second, third or fourth type of ruler-information image is selected from the preset first, second, third or fourth type of ruler-information image set and added to an optional position in the current information bar area; and the electron microscope image obtained after completing all the additions this time is used as a corresponding generated electron microscope image;

[0111] Here, Figure 4 Taking two schematic diagrams of generating electron microscope images provided in the first embodiment of the present invention as an example, Figure 4 The generated electron microscope image A in is a generated electron microscope image obtained when the relative relationship type is the first type of relative relationship;

[0112] Step A4, if the relative relationship type is the second type of relative relationship, one or more random text boxes are randomly added to the current underlying electron microscope image, and the text box background color, text box transparency, text box font, text box font color, text box text and text box position in each random text box are randomly set; and an integer is randomly selected from the value range of 0 to the second specified number M as the corresponding current random number R, and when the current random number R is not 0, a manufacturer LOGO image is randomly selected from the manufacturer LOGO image set as the corresponding current manufacturer LOGO image, and R is randomly selected on the current underlying electron microscope image. Add R random image frames of random size and capable of adaptively adjusting the images in the frames to the positions, and set the images in each random image frame to the corresponding current manufacturer's LOGO image; and select one of the four types of scale-information combinations as the corresponding current combination; if the current combination is the first, second, third or fourth type combination, select a corresponding first, second, third or fourth type of scale-information image from the preset first, second, third or fourth type of scale-information image set and add it to an optional position of the current underlying electron microscope image; and use the electron microscope image obtained after completing all the additions this time as a corresponding generated electron microscope image;

[0113] Here, Figure 4 For example, Figure 4 The generated electron microscope image B in is a generated electron microscope image obtained when the relative relationship type is the second type of relative relationship;

[0114] Among them, the information bar sub-image set includes multiple information bar sub-images; the image backgrounds of all information bar sub-images are monochrome; the foreground image information of all information bar sub-images includes a series of text information and the foreground text information does not contain any image information related to the ruler and the ruler information; the foreground image information of some information bar sub-images also includes one or more horizontal and / or vertical dividing lines; the ratio of foreground pixels / background pixels of all information bar sub-images is less than 1; when adding the information bar sub-image to the current underlying electron microscope image, the size of the information bar sub-image is proportionally scaled according to the width ratio of the information bar sub-image to the current underlying electron microscope image, and the scaled information bar sub-image is added to the bottom or top of the current underlying electron microscope image;

[0115] The manufacturer LOGO image set includes multiple manufacturer LOGO images;

[0116] The first type of ruler-information graph set includes multiple first type of ruler-information graphs, and the rulers and ruler information features corresponding to the first type of ruler-information graphs meet the first type of combination; the second type of ruler-information graph set includes multiple second type of ruler-information graphs, and the rulers and ruler information features corresponding to the second type of ruler-information graphs meet the second type of combination; the third type of ruler-information graph set includes multiple third type of ruler-information graphs, and the rulers and ruler information features corresponding to the third type of ruler-information graphs meet the third type of combination; the fourth type of ruler-information graph set includes multiple fourth type of ruler-information graphs, and the rulers and ruler information features corresponding to the fourth type of ruler-information graphs meet the fourth type of combination; after combining the first, second, and third types of rulers, the rulers and ruler information features corresponding to the fourth type of ruler-information graphs meet the fourth type of combination; When the third or fourth type of ruler-information map is added to the current information bar area, the ruler-information map currently added is appropriately scaled and moved to ensure that the ruler-information map currently added does not overlap with the foreground text information of the current information bar sub-image, and does not exceed the current information bar area; when the first, second, third or fourth type of ruler-information map is added to the current underlying electron microscope image, the ruler-information map currently added is appropriately scaled and moved to ensure that the ruler-information map currently added does not overlap with any random text box and any random image box on the current underlying electron microscope image, and does not exceed the image range of the current underlying electron microscope image;

[0117] Step 144, perform a round of traversal on the N generated electron microscope images obtained this time; and in this round of traversal, use the generated electron microscope image currently traversed as the corresponding current electron microscope image; and based on each random text box on the current electron microscope image, mark a corresponding first label detection frame, and set the first label box center point coordinates, first label box width and first label box height of the corresponding first label detection frame based on the text box center point coordinates, text box width and text box height of each random text box, and set the first label box type corresponding to each random text box to the text information type, and set the first label type confidence corresponding to each random text box to 1; and when the number of random image frames on the current electron microscope image is not zero, mark a corresponding first label detection frame based on each random image frame, and set the first label box center point coordinates, first label box width and first label box height of the corresponding first label detection frame based on the image box center point coordinates, image box width and image box height of each random image frame, and set the first label box type corresponding to each random image frame to other types, and set the first label type confidence corresponding to each random image frame to 1; and in the current electron microscope image, When the relative relationship type corresponding to the figure is the first type of relative relationship, each continuous text information on the information column sub-image in the current electron microscope image is marked with a corresponding label detection frame by manual marking or other machine marking methods to obtain a plurality of corresponding first label detection frames whose first label frame type is set to the text information type and whose first label type confidence is set to 1; and the ruler information text in the unique ruler-information image on the current electron microscope image is marked as a first label detection frame whose first label frame type is set to the text information type and whose first label type confidence is set to 1 by manual marking or other machine marking methods; and each ruler mark sub-image area in the unique ruler-information image on the current electron microscope image is marked as a first label detection frame whose first label frame type is set to the ruler mark type and whose first label type confidence is set to 1 by manual marking or other machine marking methods; and the current electron microscope image is used as a corresponding first training electron microscope image; and all the first label detection frames corresponding to the current electron microscope image form a corresponding first label detection frame sequence; and the first training electron microscope image and the first label detection frame sequence corresponding to the current electron microscope image form a corresponding first data record;

[0118] Step 145, after a round of traversal of the N generated electron microscope images obtained this time is completed, it is identified whether the current bottom electron microscope image is the last first original electron microscope image of the first original electron microscope image set; if so, go to step 146; if not, extract the next first original electron microscope image of the first original electron microscope image set as the new current bottom electron microscope image and return to step 143;

[0119] Step 146, based on the preset data enhancement rules, data enhancement processing is performed on each of the currently obtained first training electron micrographs to obtain a corresponding plurality of first enhanced electron micrographs; and each first enhanced electron micrograph is used as the corresponding current enhanced electron micrograph, and the first training electron micrograph corresponding to the current enhanced electron micrograph is used as the corresponding current parent image, and the first label detection frame sequence corresponding to the current parent image is used as the corresponding parent image label detection frame sequence, and based on the spatial transformation relationship between the current enhanced electron micrograph and the current parent image, the first label frame center point coordinates, the first label frame width and the first label frame height of each first label detection frame in the parent image label detection frame sequence are adaptively adjusted, and the label detection frame sequence obtained after the adjustment is used as a new first label detection frame sequence, and the current enhanced electron micrograph is used as a new first training electron micrograph, and a new first data record is formed by a pair of new first training electron micrographs and the first label detection frame sequence corresponding to the current enhanced electron micrograph;

[0120] The data enhancement rules at least include a horizontal flip enhancement rule, an HSV color transformation enhancement rule, and a low-angle rotation enhancement rule;

[0121] Step 146, forming a corresponding first data set from all first data records finally obtained;

[0122] Step 15, training a first target detection model based on the first data set;

[0123] Specifically, it includes: step 151, randomly dividing the first data set into two data subsets based on a preset first division ratio and recording them as a corresponding first training set and a first evaluation set;

[0124] The first segmentation ratio is a preset ratio parameter, such as 8:2; the first training set and the first evaluation set are both composed of a plurality of first data records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first segmentation ratio;

[0125] Step 152, extracting the first first data record of the first training set as the corresponding current training record;

[0126] Step 153, bringing the first training electron microscope image of the current training record into the first target detection model to perform corresponding ruler marking and text information target detection processing to obtain a corresponding first predicted target detection frame sequence;

[0127] Step 154, bringing the first predicted target detection frame sequence and the first label detection frame sequence of the current training record into a preset first model loss function; and performing a round of optimization processing on the model parameters of the first target detection model in a direction to minimize the first model loss function based on a preset first model optimizer;

[0128] The first model loss function is implemented based on the model loss function of the YOLOv8 model; the first model optimizer includes at least an SGD optimizer and an ADAM optimizer;

[0129] Step 155, identifying whether the current training record is the last first data record of the first training set; if so, going to step 156; if not, extracting the next first data record of the first training set as the new current training record and returning to step 153;

[0130] Step 156, perform a round of traversal on all the first data records of the first evaluation set; and in this round of traversal, use the first data record currently traversed as the corresponding current evaluation record; and bring the first training electron microscope image of the current evaluation record into the first target detection model for corresponding scale marking and text information target detection processing to obtain a corresponding second predicted target detection frame sequence; and form a corresponding first prediction-label pair from the second predicted target detection frame sequence and the first label detection frame sequence of the current training record; and at the end of this round of traversal, the model performance evaluation method based on the YOLOv8 model performs corresponding accuracy, precision, recall and F1 score calculations on all the first prediction-label pairs obtained to obtain the corresponding first accuracy, first precision, first recall and first F1 score;

[0131] Step 157, based on the preset first accuracy range, first precision range, first recall range and first F1 score range, compare the obtained first accuracy, first precision, first recall and first F1 score; if the first accuracy does not satisfy the corresponding first accuracy range or the first precision does not satisfy the corresponding first precision range or the first recall does not satisfy the corresponding first recall range or the first F1 score does not satisfy the corresponding first F1 score range, return to step 152 to continue training; if the first accuracy satisfies the corresponding first accuracy range and the first precision satisfies the corresponding first precision range and the first recall satisfies the corresponding first recall range and the first F1 score satisfies the corresponding first F1 score range, stop training and confirm that the model training is finished;

[0132] Here, the first accuracy range, the first precision range, the first recall range and the first F1 score range are four preset value ranges.

[0133] Step 2: After the model training is completed, any electron microscope image is received as the corresponding first electron microscope image.

[0134] Step 3, performing target detection processing on the first electron microscope image based on the first target detection model to obtain a corresponding first target detection frame sequence;

[0135] Among them, the first target detection frame sequence includes multiple first target detection frames; the first target detection frame includes the coordinates of the center point of the first detection frame, the width of the first detection frame, the height of the first detection frame, the first detection frame type and the confidence of the first detection type; the first detection frame type includes a ruler mark type, a text information type and other types.

[0136] Step 4: extract the first target detection frames whose first detection frame type is a scale mark type in the first target detection frame sequence to form a corresponding first detection frame sequence, and extract the first target detection frames whose first detection frame type is a text information type to form a corresponding second detection frame sequence.

[0137] Step 5: performing scale feature recognition based on the first detection frame sequence to obtain a corresponding first scale feature; and performing information column position recognition on the first electron microscope image to obtain a corresponding first information column position; and performing scale information text recognition based on the first information column position, the first detection frame sequence, and the second detection frame sequence to obtain a corresponding first scale information text; and forming a corresponding first combined feature from the first scale feature and the first scale information text;

[0138] Specifically, it includes: step 51, performing scale feature recognition based on the first detection frame sequence to obtain a corresponding first scale feature;

[0139] The first scale feature includes the total number of first scale marks and the first scale pixel length;

[0140] Specifically, it includes: step 511, extracting sub-images covered by each first target detection frame of the first detection frame sequence on the first electron microscope image as corresponding first detection frame sub-images; and performing binary image conversion on each first detection frame sub-image to obtain a corresponding first binary image;

[0141] The first binary image includes a plurality of first pixel points; the pixel values ​​of the first pixel points include preset foreground pixel values ​​and background pixel values; the first pixel point whose pixel value is the foreground pixel value is recorded as the corresponding first ruler mark pixel point, and the first pixel point whose pixel value is the background pixel value is recorded as the corresponding first background pixel point;

[0142] Step 512, copy the first electron microscope image to obtain a corresponding first copied electron microscope image; first reset the pixel values ​​of all pixels of the first copied electron microscope image to background pixel values; then reset the pixel values ​​of the pixels on the first copied electron microscope image corresponding to the first scale mark pixels of each first binary image to foreground pixel values; and use the first copied electron microscope image with background and foreground pixel settings completed as the corresponding second binary image;

[0143] The second binary image includes a plurality of second pixel points; the pixel values ​​of the second pixel points include foreground pixel values ​​and background pixel values; the second pixel points whose pixel values ​​are foreground pixel values ​​are recorded as corresponding second scale mark pixel points, and the second pixel points whose pixel values ​​are background pixel values ​​are recorded as corresponding second background pixel points;

[0144] Step 513, based on the preset gradient operator, the horizontal gradient of the second binary image is calculated to obtain the corresponding first gradient image; and the sum of the pixel values ​​of each column of the first gradient image is calculated to obtain the corresponding sum of the pixel values ​​of the first column; and the absolute value of the sum of the pixel values ​​of each first column is taken to obtain the corresponding absolute value of the first column; and a two-dimensional coordinate plane is constructed with the absolute value of the pixel value sum as the vertical axis and the column index as the horizontal axis as the corresponding first coordinate plane; and the corresponding coordinate points are marked on the first coordinate plane based on the obtained absolute values ​​of each first column to obtain the corresponding first marking point; and all the obtained first marking points are sequentially connected to obtain the corresponding first marking curve; and the curve peak point of the first marking curve is identified to obtain the corresponding multiple first peak points; and the leftmost and rightmost first peak points are used as the corresponding starting peak point and ending peak point; and the integer column index closest to the starting and ending peak points on the first coordinate plane is used as the corresponding starting column index and ending column index; and the difference between the ending column index and the starting column index is used as the corresponding first scale pixel length;

[0145] Wherein, the gradient operator includes at least a Sobel operator and a Prewitt operator; the first gradient map includes a plurality of third pixel points; the third pixel points correspond to the second pixel points one by one; the pixel value of the third pixel point is the horizontal direction gradient value of the corresponding second pixel point;

[0146] Step 514, counting the total number of first peak points to obtain the corresponding total number of first peaks; and calculating the corresponding total number of first scale marks based on the total number of first peaks, where the total number of first scale marks = the total number of first peaks / 2;

[0147] Step 515, obtaining the first scale feature corresponding to the total number of first scale marks and the thickest first scale pixel length;

[0148] Step 52, identifying the position of the information column on the first electron microscope image to obtain the corresponding first information column position;

[0149] Specifically, the method comprises: step 521, recording the width of the first electron microscope image as the width W; extracting the pixel values ​​of all the pixel points in each row of the first electron microscope image and sorting them in descending order to form a corresponding first pixel value sequence;

[0150] The first pixel value sequence is composed of a plurality of first pixel values ​​p i Composition, 1≤index i≤W;

[0151] Step 522, identify the parity of the width W, if the width W is an odd number, set the corresponding 50th percentile index a=(W+1) / 2, if the width W is an even number, set the corresponding 50th percentile index a=W / 2; and set the corresponding 5th percentile index b=floor(W×5 / 100), 95th percentile index c=floor(W×95 / 100), floor() is a floor rounding function;

[0152] Step 523: in each first pixel value sequence, the first pixel value p whose index i matches the 50th percentile index a, the 5th percentile index b, and the 95th percentile index c respectively. i＝a , the first pixel value p i＝b and the first pixel value p i＝c Extracted as the corresponding 50th percentile pixel value d 50 , 5th percentile pixel value d5 and 95th percentile pixel value d 95 ; and based on the 50th percentile pixel value d 50 , 5th percentile pixel value d5 and 95th percentile pixel value d 95 Set the corresponding first line parameter e = min(|d5-d 50 |,|d 50 -d 95 |);

[0153] Step 524: Check whether each first row parameter e is less than a preset first parameter threshold e. hold Identify; if the current first row parameter e is less than the first parameter threshold e hold , then the corresponding row of the current first row parameter e in the first electron microscope image is recorded as the information column row;

[0154] Here, the first parameter threshold e hold is a preset threshold parameter;

[0155] Step 525, identifying the total number of information column rows obtained; if the total number of information column rows obtained is zero, setting the corresponding first information column position to be empty; if the total number of information column rows obtained is greater than zero, identifying the maximum and minimum number of rows among all the information column rows obtained to obtain the corresponding maximum number of information column rows and minimum number of information column rows, and merging all rows from the minimum number of information column rows to the maximum number of information column rows in the first electron microscope image to obtain the obtained merged area as the corresponding first information column area, and extracting the four vertex coordinates of the first information column area in the first electron microscope image to form the corresponding first information column position;

[0156] Step 53, performing scale information text recognition based on the first information column position, the first detection frame sequence and the second detection frame sequence to obtain corresponding first scale information text;

[0157] Specifically, the method comprises: step 531, identifying whether the first information column position is empty; if the first information column position is not empty, recording the image area covered by the first information column position in the first electron microscope image as the corresponding current information column area, and recording the first target detection frame whose center point coordinates of the first detection frame in the second detection frame sequence are within the current information column area as the corresponding detection frame to be selected; if the first information column position is empty, recording all the first target detection frames in the second detection frame sequence as the corresponding detection frames to be selected;

[0158] Step 532, calculating the mean of the first detection frame center point coordinates of all first target detection frames in the first detection frame sequence and using the calculation result as the corresponding first scale center point coordinates; calculating the straight-line distance between the first detection frame center point coordinates of each candidate detection frame and the first scale center point coordinates to obtain the corresponding first distance; and sorting the candidate detection frames in ascending order of the first distance to form a corresponding candidate detection frame sequence;

[0159] Step 533, extracting the first to-be-selected detection frame in the sequence of to-be-selected detection frames as the corresponding current to-be-selected detection frame;

[0160] Step 534, extracting the sub-image area covered by the current to-be-selected detection frame in the first electron microscope image as the corresponding current sub-image; and performing text information recognition on the current sub-image based on a preset optical character recognition tool to obtain a corresponding first recognized text;

[0161] Here, the optical character recognition (OCR) tool is a pre-set OCR tool, service / interface, including at least PaddleOCR, Tianruo OCR, iFLYTEK OCR, etc.;

[0162] Step 535, whether the first recognition text contains numbers and distance units is identified; if the first recognition text contains both numbers and distance units, the corresponding first ruler information text is set as the first recognition text, and the process goes to step 536; if the first recognition text does not contain numbers or distance units, whether the current to-be-selected detection frame is the last to-be-selected detection frame in the sequence of to-be-selected detection frames is identified, if so, the corresponding first ruler information text is set to be empty and the process goes to step 536, if not, the next to-be-selected detection frame in the sequence of to-be-selected detection frames is extracted as the new current to-be-selected detection frame and the process goes back to step 534;

[0163] Step 536, outputting the first scale information text finally obtained as the recognition result;

[0164] Step 54: The first scale feature and the first scale information text form a corresponding first combined feature.

[0165] Here, the first combined feature obtained by the present invention is composed of the corresponding total number of first ruler marks (actually the total number of scales of the pixel ruler), the first ruler pixel length (actually the total length of the pixel ruler) and the first ruler information text. The pixel scale can be obtained from the total number of first ruler marks and the first ruler pixel length, and the conversion relationship between the corresponding pixel scale and the actual physical distance can be obtained from the data and units in the first ruler information text.

[0166] Figure 5 The module structure diagram of a device for identifying the scale and scale information combination features of an electron microscope image provided in the second embodiment of the present invention, the device is a terminal device or server that implements the aforementioned method embodiment, and can also be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiment, for example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 5 As shown, the device includes: a model building and training module 201, an electron microscope image receiving module 202, a target detection module 203, a detection frame clustering module 204 and a combined feature recognition module 205.

[0167] The model construction and training module 201 is used to construct an electron microscope image target detection model based on the YOLOv8 model, recorded as the corresponding first target detection model; and summarize the combined features of the scale-information combination composed of the scale and the scale information in the electron microscope image to obtain the corresponding four types of scale-information combinations; and summarize the relative position relationship between the scale-information combination and the information column in the electron microscope image to obtain the corresponding two types of relative relationships; and use the four types of scale-information combinations and the two types of relative relationships as training image construction conditions to construct a corresponding model training data set to obtain the corresponding first data set; and train the first target detection model based on the first data set.

[0168] The electron microscope image receiving module 202 is used to receive any electron microscope image as the corresponding first electron microscope image after the model training is completed.

[0169] The target detection module 203 is used to perform target detection processing on the first electron microscope image based on the first target detection model to obtain a corresponding first target detection frame sequence; the first target detection frame sequence includes multiple first target detection frames; the first target detection frame includes the coordinates of the center point of the first detection frame, the width of the first detection frame, the height of the first detection frame, the first detection frame type and the confidence of the first detection type; the first detection frame type includes a ruler mark type, a text information type and other types.

[0170] The detection frame clustering module 204 is used to extract the first target detection frames whose first detection frame type is a scale mark type in the first target detection frame sequence to form a corresponding first detection frame sequence, and extract the first target detection frames whose first detection frame type is a text information type to form a corresponding second detection frame sequence.

[0171] The combined feature recognition module 205 is used to perform scale feature recognition based on the first detection frame sequence to obtain the corresponding first scale feature; and to perform information column position recognition on the first electron microscope image to obtain the corresponding first information column position; and to perform scale information text recognition based on the first information column position, the first detection frame sequence and the second detection frame sequence to obtain the corresponding first scale information text; and the first scale feature and the first scale information text form the corresponding first combined feature.

[0172] An embodiment of the present invention provides a device for identifying the scale and scale information combination features of an electron microscope image, which can execute the method steps in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0173] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the model building and training module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the above-mentioned module is determined. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in software form.

[0174] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0175] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the above method embodiments are generated. The above computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above-mentioned computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above-mentioned computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.) methods. The above-mentioned computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The above-mentioned available medium can be a magnetic medium (such as a floppy disk, a hard disk, a tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0176] Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. The electronic device may be a terminal device or a server for implementing the method of the aforementioned embodiment, or may be a terminal device or a server for implementing the method of the aforementioned embodiment connected to the aforementioned terminal device or server. Figure 6As shown, the electronic device may include: a processor 301 (such as a CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transceiver 303. Various instructions may be stored in the memory 302 to complete various processing functions and implement the processing steps described in the aforementioned embodiment method. Preferably, the electronic device involved in the embodiment of the present invention also includes: a power supply 304, a system bus 305 and a communication port 306. The system bus 305 is used to realize the communication connection between components. The above-mentioned communication port 306 is used for connecting and communicating between the electronic device and other peripherals.

[0177] exist Figure 6 The system bus 305 mentioned in the figure can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to realize the communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM), and may also include non-volatile memory (Non-Volatile Memory), such as at least one disk storage.

[0178] The above-mentioned processor can be a general-purpose processor, including a central processing unit CPU, a network processor (Network Processor, NP), a graphics processing unit (Graphics Processing Unit, GPU), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0179] It should be noted that an embodiment of the present invention further provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the method and processing process provided in the above embodiments.

[0180] The embodiments of the present invention provide a method, device, electronic device and computer-readable storage medium for identifying the scale and scale information combination features of an electron microscope image. As can be seen from the above content, the embodiment of the present invention preliminarily constructs an electron microscope image target detection model based on the YOLOv8 model, recorded as the first target detection model, and summarizes the combined features of the scale-information combination in the electron microscope image to obtain four types of scale-information combinations, and summarizes the relative position relationship between the scale-information combination and the information bar in the electron microscope image to obtain two types of relative relationships, and constructs a first data set with the four types of scale-information combinations and the two types of relative relationships as training image construction conditions, and trains the first target detection model based on the first data set; then after the model training is completed, firstly perform target detection on any electron microscope image based on the first target detection model, and then extract the target detection frames of the scale mark type and the text information type in the target detection frame sequence to form the first and second detection frame sequences, and then identify the scale feature (including the total number of scale marks and the scale pixel length) based on the first detection frame sequence, and then identify the information bar position for the current electron microscope image, and identify the scale information text (including numbers and distance units) based on the information bar position and the first and second detection frame sequences, and the scale feature and the scale information text form a corresponding combination feature. The embodiment of the present invention can automatically identify the scale parameters of the electron microscope image, thereby improving the recognition efficiency and recognition quality; the embodiment of the present invention trains the model by constructing a diverse and enhanced training data set, thereby improving the generalization of the model and ensuring the stability of the recognition quality.

[0181] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0182] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for identifying the scale and scale information combination features of an electron microscope image, characterized in that: The method comprises: Based on the YOLOv8 model, an electron microscope image target detection model is constructed and recorded as the corresponding first target detection model; and the combined features of the ruler-information combination composed of the ruler and the ruler information in the electron microscope image are summarized to obtain the corresponding four types of ruler-information combinations; and the relative position relationship between the ruler-information combination and the information column in the electron microscope image is summarized to obtain the corresponding two types of relative relationships; and the corresponding model training data set is constructed using the four types of ruler-information combinations and the two types of relative relationships as training image construction conditions to obtain the corresponding first data set; and the first target detection model is trained based on the first data set; After the model training is completed, any electron microscope image is received as the corresponding first electron microscope image; Performing target detection processing on the first electron microscope image based on the first target detection model to obtain a corresponding first target detection frame sequence; the first target detection frame sequence includes multiple first target detection frames; the first target detection frame includes the coordinates of the center point of the first detection frame, the width of the first detection frame, the height of the first detection frame, the type of the first detection frame and the confidence of the first detection type; the first detection frame type includes a ruler mark type, a text information type and other types; Extracting the first target detection frames whose first detection frame type is a scale mark type from the first target detection frame sequence to form a corresponding first detection frame sequence, and extracting the first target detection frames whose first detection frame type is a text information type to form a corresponding second detection frame sequence; Based on the first detection frame sequence, scale feature recognition is performed to obtain a corresponding first scale feature; and the information column position of the first electron microscope image is recognized to obtain a corresponding first information column position; and based on the first information column position, the first detection frame sequence and the second detection frame sequence, scale information text recognition is performed to obtain a corresponding first scale information text; and the first scale feature and the first scale information text form a corresponding first combined feature.

2. The method for identifying the scale and scale information combination features of an electron microscope image according to claim 1, characterized in that: The ruler-information combination in the electron microscope image consists of the ruler and the ruler information; wherein the ruler contains one or more ruler marks in the horizontal direction of the image, and all ruler marks of the ruler have the same shape type, consistent size, and the same color; the ruler information is a text information containing numbers and distance units; The four types of ruler-information combinations summarized include the first type of combination, the second type of combination, the third type of combination and the fourth type of combination; the combination feature of the first type of combination is: the ruler includes two ruler marks, and the width of the ruler mark is less than the height, and the two ruler marks are connected by a horizontal line, and the ruler information is at the center of the horizontal line; the combination feature of the second type of combination is: the ruler includes two ruler marks, and the width of the ruler mark is less than the height, and the two ruler marks are connected by a horizontal line, and the ruler information is above or below the horizontal line; the combination feature of the third type of combination is: the ruler includes more than two ruler marks, and the width of the ruler mark is less than the height, and there is no horizontal connection line between every two adjacent ruler marks, and the ruler information is above or below the ruler; the combination feature of the fourth type of combination is: the ruler includes only one ruler mark, and the width of the ruler mark is greater than the height, and the ruler information is above or below the ruler; The two types of relative relationships summarized include a first type of relative relationship and a second type of relative relationship; the first type of relative relationship means that the ruler-information combination is within the range of the information column; the second type of relative relationship means that the ruler-information combination is outside the range of the information column; The first target detection model is used to perform corresponding ruler mark and text information target detection processing on the model input electron microscope image and output a corresponding target detection frame sequence; the target detection frame sequence includes multiple target detection frames; each of the target detection frames includes the coordinates of the center point of the detection frame, the width of the detection frame, the height of the detection frame, the type of the detection frame and the confidence of the detection type; the detection frame type includes a ruler mark type, a text information type and other types; The first data set includes multiple first data records; the first data record includes a first training electron microscope image and a first label detection frame sequence; the first label detection frame sequence includes multiple first label detection frames; the first label detection frame includes the coordinates of the center point of the first label frame, the width of the first label frame, the height of the first label frame, the type of the first label frame and the confidence of the first label type; the first label frame type includes a ruler mark type, a text information type and other types; the confidence of the first label type is 1; The first scale feature includes a total number of first scale marks and a first scale pixel length.

3. The method for identifying the scale and scale information combination features of an electron microscope image according to claim 2, characterized in that: The step of constructing a corresponding model training data set using the four types of scale-information combinations and the two types of relative relationships as training image construction conditions to obtain a corresponding first data set specifically includes: Step 31, pre-collecting a plurality of original electron microscope images without scales, without any text information, and without information columns to form a corresponding first original electron microscope image set; Wherein, the first original electron microscope image set includes a plurality of first original electron microscope images; Step 32, taking the first of the first original electron microscopic images in the first original electron microscopic image set as the corresponding current bottom electron microscopic image; Step 33, based on the random electron microscope image generation rule, random electron microscope image generation processing is performed according to the preset first specified number N and the current bottom electron microscope image to obtain corresponding N generated electron microscope images; The random electron microscope image generation rule requirements are as follows: For each generated electron microscope image, firstly, one of the two types of relative relationships is randomly selected as the corresponding relative relationship type; and then the relative relationship type is identified; If the relative relationship type is the first type of relative relationship, randomly select an information bar sub-image from the preset information bar sub-image set and add it to the current underlying electron microscope image; and the area covered by the current information bar sub-image on the current underlying electron microscope image is recorded as the corresponding current information bar area, and the area not covered by the current information bar sub-image is recorded as the corresponding current other area; and randomly add one or more random text boxes in the current other area, and randomly set the text box background color, text box transparency, text box font, text box font color, text box text and text box position in each of the random text boxes; and randomly select an integer from the value range of 0 to the second specified number M as the corresponding current random number R, and when the current random number R is not 0, select an integer from the preset manufacturer LOGO A manufacturer's LOGO image is randomly selected from the image set as the corresponding current manufacturer's LOGO image, and R random image frames of random size that can adaptively adjust the image in the frame are randomly selected at R positions on the current underlying electron microscope image, and the image in the frame of each random image frame is set as the corresponding current manufacturer's LOGO image; and one of the four types of scale-information combinations is selected as the corresponding current combination; if the current combination is the first, second, third or fourth type of combination, a corresponding first, second, third or fourth type of scale-information image is selected from the preset first, second, third or fourth type of scale-information image set and added to an optional position in the current information column area; and the electron microscope image obtained after completing all the additions this time is taken as a corresponding generated electron microscope image; If the relative relationship type is the second type of relative relationship, one or more random text boxes are randomly added to the current underlying electron microscope image, and the text box background color, text box transparency, text box font, text box font color, text box text and text box position in each of the random text boxes are randomly set; and an integer is randomly selected from the value range of 0 to the second specified number M as the corresponding current random number R, and when the current random number R is not 0, a manufacturer LOGO image is randomly selected from the manufacturer LOGO image set as the corresponding current manufacturer LOGO image, and a random integer is randomly selected on the current underlying electron microscope image. Add R random image frames of random size and capable of adaptively adjusting the images in the frames to R positions, and set the images in the frames of each random image frame to the corresponding current manufacturer LOGO image; and select one of the four types of scale-information combinations as the corresponding current combination; if the current combination is the first, second, third or fourth type of combination, select a corresponding first, second, third or fourth type of scale-information image from the preset first, second, third or fourth type of scale-information image set and add it to an optional position of the current underlying electron microscope image; and use the electron microscope image obtained after completing all the additions this time as a corresponding generated electron microscope image; The information bar sub-image set includes a plurality of the information bar sub-images; the image backgrounds of all the information bar sub-images are monochrome; the foreground image information of all the information bar sub-images includes a series of text information and the foreground text information does not contain any image information related to the ruler and ruler information; the foreground image information of some of the information bar sub-images also includes one or more horizontal and / or vertical dividing lines; the ratio of foreground pixels to background pixels of all the information bar sub-images is less than 1; when adding the information bar sub-image to the current underlying electron microscope image, the size of the information bar sub-image is scaled proportionally according to the width ratio of the information bar sub-image to the current underlying electron microscope image, and the scaled information bar sub-image is added to the bottom or top of the current underlying electron microscope image; The manufacturer LOGO image set includes a plurality of the manufacturer LOGO images; The first type of ruler-information graph set includes a plurality of the first type of ruler-information graphs, and the rulers and ruler information features corresponding to the first type of ruler-information graphs meet the first type of combination; the second type of ruler-information graph set includes a plurality of the second type of ruler-information graphs, and the rulers and ruler information features corresponding to the second type of ruler-information graphs meet the second type of combination; the third type of ruler-information graph set includes a plurality of the third type of ruler-information graphs, and the rulers and ruler information features corresponding to the third type of ruler-information graphs meet the third type of combination; the fourth type of ruler-information graph set includes a plurality of the fourth type of ruler-information graphs, and the rulers and ruler information features corresponding to the fourth type of ruler-information graphs meet the fourth type of combination; When adding the first, second, third or fourth type of ruler-information image to the current information bar area, appropriately scaling and moving the ruler-information image to ensure that the ruler-information image does not overlap with the foreground text information of the current information bar sub-image and does not exceed the current information bar area; When adding the first, second, third or fourth type of ruler-information map to the current bottom electron microscope image, appropriately resizing and moving the ruler-information map to ensure that the ruler-information map added does not overlap with any of the random text boxes and any of the random image boxes on the current bottom electron microscope image, and does not exceed the image range of the current bottom electron microscope image; Step 34, perform a round of traversal on the N generated electron microscope images obtained this time; and in this round of traversal, use the generated electron microscope image currently traversed as the corresponding current electron microscope image; and based on each of the random text boxes on the current electron microscope image, mark a corresponding first label detection box, and set the first label box center point coordinates, the first label box width and the first label box height of the corresponding first label detection box based on the text box center point coordinates, text box width and text box height of each of the random text boxes, and set the first label box type corresponding to each of the random text boxes to the text information type, and set each of the random text boxes to the first label detection box center point coordinates, the text box width and the text box height. The confidence of the first label type corresponding to the text box is set to 1; and when the number of the random image boxes on the current electron microscope image is not zero, a corresponding first label detection box is marked based on each of the random image boxes, and the first label box center point coordinates, the first label box width and the first label box height of the corresponding first label detection box are set based on the image box center point coordinates, the image box width and the image box height of each of the random image boxes, and the first label box type corresponding to each of the random image boxes is set to other types, and the confidence of the first label type corresponding to each of the random image boxes is set to 1; and when the relative relationship type corresponding to the current electron microscope image is the first type of relative relationship, each continuous text information on the information column sub-image in the current electron microscope image is marked with a corresponding label detection box by manual marking or other machine marking methods to obtain a plurality of corresponding first label detection boxes whose first label box type is set to the text information type and whose first label type confidence is set to 1; and the ruler information text in the unique ruler-information image on the current electron microscope image is marked as a first label box type set to the text information type and whose first label type confidence is set to 1 by manual marking or other machine marking methods. label detection frame; and mark each ruler mark sub-image area in the unique ruler-information image on the current electron microscope image into a first label detection frame in which the first label frame type is set to a ruler mark type and the first label type confidence is set to 1 by manual marking or other machine marking methods; and use the current electron microscope image as a corresponding first training electron microscope image; and form a corresponding first label detection frame sequence by all the first label detection frames corresponding to the current electron microscope image; and form a corresponding first data record by the first training electron microscope image and the first label detection frame sequence corresponding to the current electron microscope image; Step 35, after a round of traversal of the N generated electron microscope images obtained this time is completed, identify whether the current bottom electron microscope image is the last of the first original electron microscope images in the first original electron microscope image set; if so, go to step 36; if not, extract the next first original electron microscope image in the first original electron microscope image set as the new current bottom electron microscope image and return to step 33; Step 36, based on the preset data enhancement rules, data enhancement processing is performed on each of the first training electron micrographs currently obtained to obtain a corresponding plurality of first enhanced electron micrographs; and each of the first enhanced electron micrographs is used as the corresponding current enhanced electron micrograph, and the first training electron micrograph corresponding to the current enhanced electron micrograph is used as the corresponding current parent image, and the first label detection frame sequence corresponding to the current parent image is used as the corresponding parent image label detection frame sequence, and based on the spatial transformation relationship between the current enhanced electron micrograph and the current parent image, the first label frame center point coordinates, the first label frame width and the first label frame height of each of the first label detection frames in the parent image label detection frame sequence are adaptively adjusted, and the label detection frame sequence obtained after the adjustment is used as a new first label detection frame sequence, and the current enhanced electron micrograph is used as a new first training electron micrograph, and a new first data record is formed by a pair of new first training electron micrographs and the first label detection frame sequence corresponding to the current enhanced electron micrograph; The data enhancement rules include at least a horizontal flip enhancement rule, an HSV color transformation enhancement rule and a low-angle rotation enhancement rule; Step 36: All the first data records finally obtained form the corresponding first data set.

4. The method for identifying the scale and scale information combination features of an electron microscope image according to claim 2, characterized in that: The training of the first target detection model based on the first data set specifically includes: Step 41, based on a preset first segmentation ratio, randomly split the first data set into two data subsets recorded as a corresponding first training set and a first evaluation set; Wherein, both the first training set and the first evaluation set are composed of a plurality of the first data records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first segmentation ratio; Step 42, extracting the first first data record of the first training set as the corresponding current training record; Step 43, bringing the first training electron microscope image of the current training record into the first target detection model to perform corresponding scale marking and text information target detection processing to obtain a corresponding first predicted target detection frame sequence; Step 44, bringing the first predicted target detection frame sequence and the first label detection frame sequence of the current training record into a preset first model loss function; and performing a round of optimization processing on the model parameters of the first target detection model in a direction in which the first model loss function reaches a minimum value based on a preset first model optimizer; Wherein, the first model loss function is implemented based on the model loss function of the YOLOv8 model; the first model optimizer includes at least an SGD optimizer and an ADAM optimizer; Step 45, identifying whether the current training record is the last first data record of the first training set; if so, going to step 46; if not, extracting the next first data record of the first training set as the new current training record and returning to step 43; Step 46, perform a round of traversal on all the first data records of the first evaluation set; and in this round of traversal, use the first data record currently traversed as the corresponding current evaluation record; and bring the first training electron microscope image of the current evaluation record into the first target detection model for corresponding scale marking and text information target detection processing to obtain a corresponding second predicted target detection frame sequence; and form a corresponding first prediction-label pair by the second predicted target detection frame sequence and the first label detection frame sequence of the current training record; and at the end of this round of traversal, the model performance evaluation method based on the YOLOv8 model performs corresponding accuracy, precision, recall and F1 score calculations on all the first prediction-label pairs obtained to obtain the corresponding first accuracy, first precision, first recall and first F1 score; Step 47, based on the preset first accuracy range, first precision range, first recall rate range and first F1 score range, compare the obtained first accuracy, first precision, first recall rate and first F1 score; if the first accuracy does not satisfy the corresponding first accuracy range or the first accuracy does not satisfy the corresponding first precision range or the first recall rate does not satisfy the corresponding first recall rate range or the first F1 score does not satisfy the corresponding first F1 score range, return to step 42 to continue training; if the first accuracy satisfies the corresponding first accuracy range and the first precision satisfies the corresponding first precision range and the first recall rate satisfies the corresponding first recall rate range and the first F1 score satisfies the corresponding first F1 score range, stop training and confirm that the model training is completed.

5. The method for identifying the scale and scale information combination features of an electron microscope image according to claim 2, characterized in that: The performing scale feature recognition based on the first detection frame sequence to obtain the corresponding first scale feature specifically includes: Step 51, extracting the sub-images covered by each of the first target detection frames of the first detection frame sequence on the first electron microscope image as corresponding first detection frame sub-images; and performing binary image conversion on each of the first detection frame sub-images to obtain corresponding first binary images; Wherein, the first binary image includes a plurality of first pixel points; the pixel values ​​of the first pixel points include preset foreground pixel values ​​and background pixel values; the first pixel points whose pixel values ​​are the foreground pixel values ​​are recorded as corresponding first ruler mark pixel points, and the first pixel points whose pixel values ​​are the background pixel values ​​are recorded as corresponding first background pixel points; Step 52, copying the first electron microscope image to obtain a corresponding first copied electron microscope image; firstly resetting the pixel values ​​of all pixels of the first copied electron microscope image to the background pixel values; then resetting the pixel values ​​of the pixels on the first copied electron microscope image corresponding to the first scale mark pixels of each of the first binary images to the foreground pixel values; and taking the first copied electron microscope image with the background and foreground pixel settings completed as the corresponding second binary image; The second binary image includes a plurality of second pixel points; the pixel values ​​of the second pixel points include the foreground pixel value and the background pixel value; the second pixel point whose pixel value is the foreground pixel value is recorded as the corresponding second ruler mark pixel point, and the second pixel point whose pixel value is the background pixel value is recorded as the corresponding second background pixel point; Step 53, based on a preset gradient operator, the horizontal gradient of the second binary image is calculated to obtain a corresponding first gradient image; and the sum of the pixel values ​​of each column of the first gradient image is calculated to obtain the corresponding sum of the pixel values ​​of the first column; and the absolute value of the sum of the pixel values ​​of each first column is taken to obtain the corresponding absolute value of the first column; and a two-dimensional coordinate plane is constructed with the absolute value of the pixel value sum as the vertical axis and the column index as the horizontal axis as the corresponding first coordinate plane; and on the first coordinate plane, corresponding coordinate point marking is performed based on the obtained absolute values ​​of each first column to obtain the corresponding first marking point; and all the obtained first marking points are sequentially connected to obtain the corresponding first marking curve; and the curve peak point identification of the first marking curve is performed to obtain the corresponding multiple first peak points; and the first peak points on the leftmost and rightmost sides are used as the corresponding starting peak point and ending peak point; and the integer column index closest to the starting and ending peak points on the first coordinate plane is used as the corresponding starting column index and ending column index; and the difference between the ending column index and the starting column index is used as the corresponding first scale pixel length; The first gradient map includes a plurality of third pixel points; the third pixel points correspond to the second pixel points one by one; the pixel value of the third pixel point corresponds to the horizontal gradient value of the corresponding second pixel point; Step 54, counting the total number of the first peak points to obtain the corresponding total number of first peaks; and calculating the corresponding total number of the first scale marks based on the total number of the first peaks, the total number of first scale marks=the total number of first peaks / 2; Step 55: obtain the first scale feature corresponding to the total number of the first scale marks and the thickest pixel length of the first scale.

6. The method for identifying the scale and scale information combination features of an electron microscope image according to claim 2, characterized in that: The step of identifying the position of the information column on the first electron microscope image to obtain the corresponding first information column position specifically includes: Step 61, record the width of the first electron microscope image as width W; and extract the pixel values ​​of all pixel points in each row of the first electron microscope image and sort them in descending order to form a corresponding first pixel value sequence; The first pixel value sequence is composed of a plurality of first pixel values ​​p i Composition, 1≤index i≤W; Step 62, identifying the parity of the width W, if the width W is an odd number, setting the corresponding 50th percentile index a=(W+1) / 2, if the width W is an even number, setting the corresponding 50th percentile index a=W / 2; and setting the corresponding 5th percentile index b=floor(W×5 / 100), 95th percentile index c=floor(W×95 / 100), floor() is a floor rounding function; Step 63: in each of the first pixel value sequences, the first pixel value p whose index i matches the 50th percentile index a, the 5th percentile index b, and the 95th percentile index c respectively. i＝a , the first pixel value p i＝b and the first pixel value p i＝c Extracted as the corresponding 50th percentile pixel value d 50 , 5th percentile pixel value d5 and 95th percentile pixel value d 95 ; and based on the 50th percentile pixel value d 50 , the 5th percentile pixel value d5 and the 95th percentile pixel value d 95 Set the corresponding first line parameter e = min(|d5-d 50 |,|d 50 -d 95 |); Step 64: Check whether each of the first row parameters e is less than a preset first parameter threshold e. hold Identify; if the current first row parameter e is less than the first parameter threshold e hold , then the corresponding row of the first row parameter e in the first electron microscope image is recorded as the information column row; Step 65, identifying the total number of the information column rows obtained; if the total number of the information column rows obtained is zero, setting the corresponding first information column position to be empty; if the total number of the information column rows obtained is greater than zero, identifying the maximum and minimum number of rows among all the information column rows obtained to obtain the corresponding maximum number of information column rows and minimum number of information column rows, and merging all rows from the minimum number of information column rows to the maximum number of information column rows in the first electron microscope image to obtain the obtained merged area as the corresponding first information column area, and extracting the four vertex coordinates of the first information column area in the first electron microscope image to form the corresponding first information column position.

7. The method for identifying the scale and scale information combination features of an electron microscope image according to claim 2, characterized in that: The performing scale information text recognition based on the first information column position, the first detection frame sequence, and the second detection frame sequence to obtain the corresponding first scale information text specifically includes: Step 71, identifying whether the first information column position is empty; if the first information column position is not empty, recording the image area covered by the first information column position in the first electron microscope image as the corresponding current information column area, and recording the first target detection frame in the second detection frame sequence whose center point coordinates are within the current information column area as the corresponding to-be-selected detection frame; if the first information column position is empty, recording all the first target detection frames in the second detection frame sequence as the corresponding to-be-selected detection frames; Step 72, calculating the mean of the first detection frame center point coordinates of all the first target detection frames of the first detection frame sequence and using the calculation result as the corresponding first scale center point coordinates; calculating the straight-line distance between the first detection frame center point coordinates of each of the to-be-selected detection frames and the first scale center point coordinates to obtain a corresponding first distance; and sorting the to-be-selected detection frames in ascending order of the first distances to form a corresponding to-be-selected detection frame sequence; Step 73, extracting the first of the to-be-selected detection frames in the to-be-selected detection frame sequence as the corresponding current to-be-selected detection frame; Step 74, extracting the sub-image area covered by the current to-be-selected detection frame in the first electron microscope image as the corresponding current sub-image; and performing text information recognition on the current sub-image based on a preset optical character recognition tool to obtain a corresponding first recognized text; Step 75, identifying whether the first recognition text contains numbers and distance units; if the first recognition text contains both numbers and distance units, setting the corresponding first scale information text as the first recognition text, and going to step 76; if the first recognition text does not contain numbers or distance units, identifying whether the current to-be-selected detection frame is the last to-be-selected detection frame in the to-be-selected detection frame sequence, if so, setting the corresponding first scale information text to be empty and going to step 76, if not, extracting the next to-be-selected detection frame in the to-be-selected detection frame sequence as the new current to-be-selected detection frame and returning to step 74; Step 76: output the first scale information text finally obtained as the recognition result.

8. A device for executing the method for identifying the scale and scale information combination features of an electron microscope image as described in any one of claims 1 to 7, characterized in that: The device comprises: a model building and training module, an electron microscope image receiving module, a target detection module, a detection frame clustering module and a combined feature recognition module; The model construction and training module is used to construct an electron microscope image target detection model based on the YOLOv8 model, recorded as the corresponding first target detection model; and summarize the combined features of the scale-information combination composed of the scale and the scale information in the electron microscope image to obtain the corresponding four types of scale-information combinations; and summarize the relative position relationship between the scale-information combination and the information column in the electron microscope image to obtain the corresponding two types of relative relationships; and use the four types of scale-information combinations and the two types of relative relationships as training image construction conditions to construct a corresponding model training data set to obtain a corresponding first data set; and train the first target detection model based on the first data set; The electron microscope image receiving module is used to receive any electron microscope image as the corresponding first electron microscope image after the model training is completed; The target detection module is used to perform target detection processing on the first electron microscope image based on the first target detection model to obtain a corresponding first target detection frame sequence; the first target detection frame sequence includes multiple first target detection frames; the first target detection frame includes the coordinates of the center point of the first detection frame, the width of the first detection frame, the height of the first detection frame, the type of the first detection frame and the confidence of the first detection type; the first detection frame type includes a ruler mark type, a text information type and other types; The detection frame clustering module is used to extract the first target detection frames whose first detection frame type is a scale mark type in the first target detection frame sequence to form a corresponding first detection frame sequence, and extract the first target detection frames whose first detection frame type is a text information type to form a corresponding second detection frame sequence; The combined feature recognition module is used to perform scale feature recognition based on the first detection frame sequence to obtain a corresponding first scale feature; and to perform information column position recognition on the first electron microscope image to obtain a corresponding first information column position; and to perform scale information text recognition based on the first information column position, the first detection frame sequence and the second detection frame sequence to obtain a corresponding first scale information text; and the first scale feature and the first scale information text form a corresponding first combined feature.

9. An electronic device, characterized in that: include: memory, processors, and transceivers; The processor is used to couple with the memory, read and execute instructions in the memory, so as to implement the method according to any one of claims 1 to 7; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a computer, the computer is enabled to execute the method according to any one of claims 1 to 7.

Images