A gold core pellet image generation method and system based on multi-scale feature comparison

Abstract

The application provides a golden core particle image generation method and system based on multi-scale feature comparison, acquires multiple images of the same type of core particles that may have defect information through random image acquisition; through pose alignment and brightness adjustment, the multiple core particles are kept consistent in the physical space; through a pre-trained convolutional neural network model, multi-scale features are extracted, difference information of the region position is judged, and no difference regions are selected for content filling on the basis thereof; pixel-level information of a plurality of no difference regions is fused to generate pixel content of the final golden core particle; specifically, on the basis of a plurality of images that may contain defective core particles, the standard morphology of each position without defects is determined through the mutual relationship between the images, so that pixel-level synthesis of the golden core particle is realized, and effective reference basis is provided for a wafer defect detection process with a pattern.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of core particle detection, and particularly relates to a gold core particle image generation method and system based on multi-scale feature comparison. BACKGROUND

[0002] In the semiconductor wafer defect detection link, the detection on the core particle with a pattern wafer is usually carried out in a contrast manner. In the recipe construction stage, the user needs to provide a gold core particle image without any defects as a reference for a specific type of core particle. In the mass production test link, the potential defect information is output by comparing the differences between the input core particle and the gold core particle image.

[0003] In the prior art, the core particle design bottom map is usually used as a reference, but design bottom map information cannot be obtained in all process quality inspection links. Some methods synthesize a gold core particle image by position splicing, but this requires high preconditions for image alignment. If there is no accurate alignment, image damage information may be caused in the synthesis process. SUMMARY

[0004] Therefore, the embodiment of the present application provides a gold core particle image generation method and system based on multi-scale feature comparison, aiming to improve the generation efficiency and generation quality of the gold core particle image.

[0005] The first aspect of the embodiment of the present application provides a gold core particle image generation method based on multi-scale feature comparison, which comprises:

[0006] Obtaining a plurality of core particle images of the same type of core particle;

[0007] Determining a plurality of feature points of the core particle image through frequency domain analysis, and determining standard position information of the feature points according to the physical space distribution of the plurality of feature points;

[0008] Establishing a mapping matrix of each feature point and the standard position information for each core particle, and transforming the core particle image by using affine transformation to complete core particle pose alignment;

[0009] According to a preset aggregation region, the pixel brightness mean value of the core particle image in the feature dimension is counted, the mapping information of gamma correction of each core particle compared with the pixel brightness mean value is solved, and the pixel increment repair of the core particle image is carried out according to the mapping information to complete the core particle brightness alignment;

[0010] According to a pre-trained convolutional neural network model, the multi-scale features of the core particle image after core particle pose and core particle brightness alignment are extracted, and the non-difference region is determined according to the multi-scale features, and the non-difference region is selected for filling;

[0011] The filled non-difference areas are subjected to Poisson fusion processing to generate a golden core particle image.

[0012] Further, the step of calculating the average pixel brightness of the core particle image in the feature dimension according to the preset aggregation area, solving the mapping information of gamma correction of each core particle compared with the average pixel brightness, and performing pixel increment repair on the core particle image according to the mapping information to complete the core particle brightness alignment step includes:

[0013] Step one, according to the preset aggregation area, the average pixel brightness of the core particle image in the feature dimension is calculated, and the average pixel brightness is calculated according to the average pixel brightness, and the average pixel brightness is taken as the standard brightness reference of all core particles;

[0014] Step two, for each core particle, compare the average pixel brightness of the preset aggregation area of the core particle with the standard brightness reference to determine the brightness deviation degree, and establish a gamma correction mapping table for the corresponding core particle according to the brightness deviation degree;

[0015] Step three, traverse each pixel point of the core particle image, determine the corrected brightness value according to the original pixel brightness value of the pixel point and the gamma correction mapping table, and replace the original pixel brightness value;

[0016] Step four, re-calculate the average pixel brightness of the core particle in the preset aggregation area, and judge whether the deviation of the recalculated average pixel brightness of the core particle in the preset aggregation area from the standard brightness reference is within the preset range;

[0017] Step five, if not, re-adjust the parameters of the gamma correction mapping table, repeat the operation in step four until the deviation of the average pixel brightness of the core particle in the preset aggregation area from the standard brightness reference is within the preset range.

[0018] Further, the step of extracting the multi-scale features of the core particle pose and the core particle brightness aligned core particle image according to the pre-trained convolutional neural network model, and determining the non-difference area according to the multi-scale features includes:

[0019] The core particle pose and the core particle brightness aligned core particle image are input into the pre-trained convolutional neural network model, the output features of each layer of the convolutional neural network model are obtained, and each of the output features is up-sampled to the original size of the input core particle image, so that all features are guaranteed to be the same size;

[0020] The up-sampling results of each layer of the convolutional neural network model are obtained, and the cosine similarity of each pixel point between the core particle pose and the core particle brightness aligned core particle image is calculated;

[0021] Determine the minimum value of the cosine similarity of each pixel point as the difference information of the corresponding pixel point, integrate the difference information of all pixel points to form a difference distribution map with the same size as the original map;

[0022] According to the preset threshold, the difference distribution map is binarized, and according to the binarization operation result, the non-difference region is determined.

[0023] Further, the step of selecting the non-difference region for filling comprises:

[0024] According to the non-difference region, the pixel values of the corresponding positions are extracted from a preset number of core particle images, and for each position of the non-difference region, the preset number of pixel samples are conditionally weighted and fused, and the conditionally weighted and fused result is written to the corresponding position to complete the filling.

[0025] Further, in the step of obtaining the up-sampling result, calculating the cosine similarity of each pixel point between the core particle images after aligning the core particle pose and the core particle brightness, the calculation formula of the cosine similarity is:

[0026] ;

[0027] Wherein, is the cosine similarity of the core particle image a and the core particle image b at each pixel point, is the up-sampling result of the core particle image a at the i-th layer, is the up-sampling result of the core particle image b at the i-th layer, U is the up-sampling operation, cos_sim is the cosine similarity of the calculation of two feature vectors, and N is the total number of layers of the convolutional neural network model.

[0028] Further, in the step of performing Poisson fusion processing on the filled non-difference region to generate a golden core particle image, the gradient features of the second non-difference region to be fused are retained and the boundary values of the first non-difference region are fitted, to realize seamless fusion.

[0029] Further, the convolutional neural network model is a ResNet-50 model, which includes 4 convolutional layers.

[0030] The second aspect of the embodiment of the application provides a golden core particle image generation system based on multi-scale feature comparison, which is used for realizing the golden core particle image generation method based on multi-scale feature comparison of the first aspect, and the system comprises:

[0031] The acquisition module is used for acquiring a plurality of core particle images of the same type of core particle.

[0032] The determining module is configured to determine a plurality of feature points of the core image by frequency domain analysis, and determine standard position information of the feature points according to a physical space distribution of the plurality of feature points.

[0033] The pose alignment module is configured to establish a mapping matrix of each feature point and the standard position information for each core, and transform the core image by using affine transformation to complete core pose alignment.

[0034] The brightness alignment module is configured to calculate a pixel brightness average of the core image in a feature dimension according to a preset aggregation area, solve mapping information of gamma correction of each core compared with the pixel brightness average, and perform pixel increment repair on the core image according to the mapping information to complete core brightness alignment.

[0035] The filling module is configured to extract multi-scale features of the core image after core pose alignment and core brightness alignment according to a pre-trained convolutional neural network model, determine a non-difference area according to the multi-scale features, and select the non-difference area for filling.

[0036] The generating module is configured to perform Poisson fusion processing on the filled non-difference areas to generate a golden core image.

[0037] A third aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the golden core image generation method based on multi-scale feature comparison provided in the first aspect.

[0038] A fourth aspect of the embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the golden core image generation method based on multi-scale feature comparison provided in the first aspect when executing the program.

[0039] The golden core particle image generation method and system based on multi-scale feature comparison provided in the embodiment of the present application, by acquiring a plurality of core particle images of the same type of core particle; by frequency domain analysis, a plurality of feature points of the core particle image are determined, and the standard position information of the feature points is determined according to the physical space distribution of the plurality of feature points; for each core particle, a mapping matrix of each feature point and the standard position information is established, and the core particle image is transformed by using affine transformation to complete the core particle pose alignment; according to the preset aggregation region, the pixel brightness average of the core particle image in the feature dimension is counted, the mapping information of the gamma correction of each core particle compared with the pixel brightness average is solved, and the pixel increment repair of the core particle image is performed according to the mapping information to complete the core particle brightness alignment; according to the pre-trained convolutional neural network model, the multi-scale features of the core particle image after the core particle pose and core particle brightness alignment are extracted, and the non-difference region is determined according to the multi-scale features, and the non-difference region is selected for filling; the filled non-difference regions are processed by Poisson fusion to generate a golden core particle image. Specifically, on the basis of a plurality of possible images that may contain defective core particles, the standard morphology of each position without defects is determined through the mutual relationship between each other, so that the pixel-level synthesis of the golden core particle is realized, and effective reference basis is provided for the pattern wafer defect detection process. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 The implementation flowchart of the golden core particle image generation method based on multi-scale feature comparison provided for the embodiment one of the present application is provided.

[0041] Figure 2 The structural block diagram of the golden core particle image generation system based on multi-scale feature comparison provided for the embodiment two of the present application is provided.

[0042] Figure 3 The structural block diagram of the electronic device provided for the embodiment three of the present application is provided. DETAILED DESCRIPTION

[0043] In order to facilitate the understanding of the present application, the present application will be described more fully below with reference to the related drawings. The drawings show several embodiments of the present application. However, the present application can be realized in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0044] It should be noted that when an element is referred to as being "fixedly attached" to another element, it can be directly on the other element or there can be an intervening element. When an element is referred to as being "connected" to another element, it can be directly connected to the other element or intervening elements can be present. The terms "vertical", "horizontal", "left", "right", and the like as used herein are for purposes of illustration only.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0046] Embodiment one

[0047] According to the embodiment of the present application, a golden core grain image generation method based on multi-scale feature comparison is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in a different order.

[0048] In this embodiment one, a golden core grain image generation method based on multi-scale feature comparison is provided, which can be used in electronic devices such as computers. Please refer to Figure 1 , Figure 1 The implementation flowchart of the golden core grain image generation method based on multi-scale feature comparison provided by the embodiment one of the present application is shown, which specifically includes steps S01 to S06.

[0049] Step S01, obtaining a plurality of core grain images of the same type of core grain.

[0050] Specifically, the wafer is placed on the stage, the stage movement and camera imaging are controlled by the master control platform, the image of the partial region or the whole region of the wafer surface is shot by the linear array camera or the area array camera, and the scanning original drawing is obtained; according to the pre-extracted layout information, the core grain is cropped at the corresponding position of the scanning original drawing, so that the cropped region contains the complete core grain content; according to the preset core grain positioning core information, the foreground of the core grain is accurately positioned by using the template matching method, and then the core grain image is converted to the standard size according to the point information of the core grain obtained by positioning.

[0051] Step S02, determining a plurality of feature points of the core grain image by frequency domain analysis, and determining the standard position information of the feature points according to the physical space distribution of the plurality of feature points.

[0052] Specifically, by frequency domain analysis (such as Fourier transform, wavelet transform, etc.), the feature points with spatial uniqueness and structural stability are screened out from the core particle image, and then spatial clustering analysis is performed on all feature points, that is, the average coordinates and variance of the feature points in the same class are calculated, the abnormal points deviating too much are removed, and the average coordinates are taken as the standard position of the feature points, so that the standard position can represent the distribution rule of the feature points of most core particles, and finally a standard feature point set is obtained, each feature point has a unique standard (x, y) coordinate, and a unified spatial coordinate system of the core particle is formed.

[0053] In step S03, a mapping matrix of each feature point and the standard position information is established for each core particle, and an affine transformation is used to transform the core particle image to complete the core particle pose alignment.

[0054] It should be noted that the feature point coordinates actually extracted from the core particle are matched with the preset standard position coordinates to establish a spatial mapping relationship therebetween, and a mapping matrix, that is, an affine transformation matrix, is generated. The matrix includes geometric transformation parameters such as translation (elimination of position deviation), rotation (elimination of angle deviation), and scaling (elimination of size deviation) that need to be adjusted for the core particle, and can ensure that each feature point in the source coordinates can be accurately corresponded to the standard position coordinates after matrix operation.

[0055] The constructed affine transformation matrix is applied to the entire core particle image (not only the feature points), and all pixel points in the image are subjected to synchronous geometric transformation. It can be understood that after the transformation, all structures (including feature points and non-feature point regions) of the core particle will be “corrected” to the standard coordinate system, and complete alignment in the spatial pose with other candidate core particles is achieved.

[0056] In step S04, the pixel brightness average in the feature dimension of the core particle image is counted according to the preset aggregation region, the mapping information of gamma correction of each core particle compared with the pixel brightness average is solved, and the pixel increment repair of the core particle image is performed according to the mapping information to complete the core particle brightness alignment.

[0057] In the embodiment of the present application, the step of core particle brightness alignment specifically includes:

[0058] In step one, the pixel brightness average in the feature dimension of the core particle image is counted according to the preset aggregation region, and the pixel brightness average is calculated according to the pixel brightness average, and the pixel brightness average is taken as the standard brightness reference of all core particles. The preset aggregation region refers to a plurality of regular or irregular aggregation regions (such as functional regions, pin array regions, edge blank regions, etc.) of the core particle which are preliminarily defined according to the structural characteristics of the core particle, and the purpose is to avoid distortion of the average value due to local abnormal regions (such as edge dark spots of individual core particles) when directly counting the brightness of the full image;

[0059] Step two, for each core particle, compare the average pixel brightness of its own preset aggregation area with the standard brightness reference, determine the brightness deviation degree, and establish the gamma correction mapping table for the corresponding core particle according to the brightness deviation degree. It can be understood that the gamma correction is to adjust the non-linear transformation curve (gamma value) of the pixel brightness, and the "dark" processing is performed on the bright core particle, and the "bright" processing is performed on the dark core particle, so that the brightness average value of the corrected aggregation area can accurately match the standard brightness reference, and the mapping information will clearly record the corrected brightness value corresponding to each original brightness value (such as 0-255 gray value), forming a one-to-one mapping table of "original brightness→corrected brightness";

[0060] Step three, traverse each pixel point of the core particle image, determine the corrected brightness value according to the original pixel brightness value of the pixel point and the gamma correction mapping table, and replace the original pixel brightness value;

[0061] Step four, re-count the average pixel brightness of the core particle in the preset aggregation area, and judge whether the deviation of the re-counted average pixel brightness of the core particle in the preset aggregation area from the standard brightness reference is within the preset range;

[0062] Step five, if not, re-adjust the parameters of the gamma correction mapping table, repeat the operation in step four until the deviation of the average pixel brightness of the core particle in the preset aggregation area from the standard brightness reference is within the preset range.

[0063] Step S05, according to the pre-trained convolutional neural network model, extract the multi-scale features of the core particle image after aligning the core particle pose and the core particle brightness, and determine the non-difference area according to the multi-scale features, and select the non-difference area for filling.

[0064] Specifically, the core particle image after aligning the core particle pose and the core particle brightness is input into the pre-trained convolutional neural network model, the output features of each layer of the convolutional neural network model are obtained, and each output feature is up-sampled to the original size of the input core particle image, so that all features have the same size, wherein the convolutional neural network model is a ResNet-50 model, including 4 convolutional layers.

[0065] The up-sampling results of each layer of the convolutional neural network model are obtained, the cosine similarity of each pixel point between the core particle images after aligning the core particle pose and the core particle brightness is calculated, and the calculation formula of the cosine similarity is:

[0066] ;

[0067] Wherein, is the cosine similarity of the core particle image a and the core particle image b at each pixel point, is the up-sampling result of the core particle image a at the i-th layer, is the up-sampling result of the core particle image b at the i-th layer, U is an up-sampling operation, cos_sim is a cosine similarity between two feature vectors, N is the total number of layers of the convolutional neural network model, and in the embodiment of the present application, N is 4;

[0068] The minimum value of the cosine similarity of each pixel point is determined as the difference information of the corresponding pixel point, and the difference information of all pixel points is integrated to form a difference distribution map with the same size as the original map.

[0069] According to the preset threshold, the difference distribution map is binarized, and according to the binarization result, the non-difference region is determined, wherein the preset threshold is set according to the consistency requirement of the core particle, for example, the preset threshold is smaller if the requirement is strict consistency, and the preset threshold is larger if slight difference is allowed, and generally needs to be adjusted in combination with the process characteristics and defect detection requirements of the actual core particle. Further, the difference information at each position in the difference distribution map is compared with the preset threshold, if the difference information is less than or equal to the preset threshold, it is determined as a non-difference region (assigned value 1), if the difference information is greater than the preset threshold, it is determined as a difference region (assigned value 0), and the region with value 1 in the mask obtained after binarization is the region where the characteristics of all K core particles are consistent at this position, which is considered as the "inherent normal structure" of the core particle, and can be used as the candidate filling region of the golden core particle.

[0070] The step of selecting the non-difference region for filling includes:

[0071] According to the non-difference region, pixel values at corresponding positions are extracted from a preset number of core particle images, and for each position of the non-difference region, conditional weighted fusion processing is performed on the preset number of pixel samples, and the conditional weighted fusion processing result is written to the corresponding position to complete the filling. Specifically, for each position of the non-difference region, statistical analysis is performed on the preset number of pixel samples to identify the principal component information (for each position of the non-difference region, the preset number of pixel samples), and the reliability of the pixel (such as removing abnormal pixels obviously deviating from the principal component and giving higher weight to pixels close to the principal component) is combined to calculate the weighted average value as the final pixel value of the position. The calculated pixel value is written to the corresponding position of the golden core particle image to complete the filling of the current non-difference region.

[0072] In step S06, the filled non-difference regions are subjected to Poisson fusion processing to generate a golden core particle image.

[0073] It should be noted that by retaining the gradient features of the second indifferent region to be fused and fitting the boundary values of the first indifferent region, seamless fusion is achieved. Specifically, first, the key regions are determined, including a target region T (a blank area in the first indifferent region to be filled with the second indifferent region), a source region S (the second indifferent region), and a boundary region between the two, and a target mask M T (the position marked T) and a source mask M S (the range marked S), the source region S and the target region T are mapped in a unified coordinate system to ensure position matching.

[0074] Further, the gradient field of the second indifferent region in the x direction and the y direction is calculated, the texture detail features are retained, and the boundary pixel values of T and the first indifferent region (partial T) are extracted as the hard constraint condition for fusion.

[0075] Further, for the non-boundary pixels in T, a linear equation group with "Laplacian operator = source region divergence" as the core is established, and the conjugate gradient method or the multi-grid method is used to solve the sparse linear equation group to obtain the pixel value of the T region.

[0076] Finally, the pixel value of the T region obtained by solving is replaced in the corresponding position of the original image to form a combined image containing the first indifferent region and the second indifferent region, and gray adjustment or edge smoothing is performed to eliminate potential differences and ensure overall consistency.

[0077] In summary, the golden core particle image generation method based on multi-scale feature comparison in the above embodiments of the present application acquires a plurality of core particle images of the same type of core particle; determines a plurality of feature points of the core particle image through frequency domain analysis, and determines standard position information of the feature points according to the physical space distribution of the plurality of feature points; establishes a mapping matrix of each feature point and the standard position information for each core particle, and transforms the core particle image by using affine transformation to complete core particle pose alignment; according to a preset aggregation region, the pixel brightness mean value of the core particle image in the feature dimension is counted, the mapping information of gamma correction of each core particle compared with the pixel brightness mean value is solved, and the core particle image is repaired by pixel increment according to the mapping information to complete core particle brightness alignment; the multi-scale features of the core particle image after core particle pose and core particle brightness alignment are extracted according to a pre-trained convolutional neural network model, and the multi-scale features are used to determine indifferent regions, and the indifferent regions are selected for filling; the filled indifferent regions are processed by Poisson fusion to generate a golden core particle image. Specifically, on the basis of a plurality of possible images of defective core particles, the standard morphology of each position without defects is determined through the mutual relationship between them, so that the pixel-level synthesis of the golden core particle is realized, and effective reference basis is provided for the pattern wafer defect detection process.

[0078] Embodiment Two

[0079] Referring now to the drawings Figure 2 , Figure 2 is a structural block diagram of a golden core grain image generation system based on multi-scale feature comparison provided by Embodiment Two of the present application, which is used to implement the above-mentioned embodiments and preferred embodiments, and has been described above. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware, or a combination of software and hardware, implementation is also possible and is contemplated.

[0080] Specifically, the golden core grain image generation system based on multi-scale feature comparison 200 includes an acquisition module 21, a determination module 22, a pose alignment module 23, a brightness alignment module 24, a filling module 25, and a generation module 26, wherein:

[0081] The acquisition module 21 is configured to acquire a plurality of core grain images of the same type of core grain.

[0082] The determination module 22 is configured to determine a plurality of feature points of the core grain images through frequency domain analysis, and determine standard position information of the feature points according to the physical space distribution of the plurality of feature points.

[0083] The pose alignment module 23 is configured to establish a mapping matrix of each of the feature points and the standard position information for each core grain, and transform the core grain image using affine transformation to complete core grain pose alignment.

[0084] The brightness alignment module 24 is configured to calculate the pixel brightness mean value of the core grain image in the feature dimension according to a preset aggregation region, solve the mapping information of the gamma correction of each core grain with respect to the pixel brightness mean value, and perform pixel increment repair on the core grain image according to the mapping information to complete core grain brightness alignment.

[0085] The filling module 25 is configured to extract multi-scale features of the core grain image after core grain pose and brightness alignment according to a pre-trained convolutional neural network model, determine a non-difference region according to the multi-scale features, and select the non-difference region for filling, wherein the convolutional neural network model is a ResNet-50 model including four convolutional layers.

[0086] The generation module 26 is configured to perform Poisson fusion processing on each of the filled non-difference regions to generate a golden core grain image, wherein the non-seam fusion is achieved by retaining the gradient features of the second non-difference region to be fused and adhering to the boundary values of the first non-difference region.

[0087] Further, in some optional embodiments of the present application, the brightness alignment module 24 comprises:

[0088] a first calculation unit configured to calculate the average pixel brightness of the core particle image in the feature dimension according to the preset aggregation region, and calculate the average pixel brightness according to the average pixel brightness, and take the average pixel brightness as the standard brightness reference of all core particles;

[0089] a comparison unit configured to compare the average pixel brightness of the preset aggregation region of each core particle with the standard brightness reference to determine the brightness deviation degree, and establish the gamma correction mapping table of the corresponding core particle according to the brightness deviation degree;

[0090] a traversal unit configured to traverse each pixel point of the core particle image, and determine the corrected brightness value according to the original pixel brightness value of the pixel point and the gamma correction mapping table, and replace the original pixel brightness value;

[0091] a judgment unit configured to re-calculate the average pixel brightness of the core particle in the preset aggregation region, and judge whether the deviation of the re-calculated average pixel brightness of the core particle in the preset aggregation region from the standard brightness reference is within the preset range;

[0092] a loop unit configured to re-adjust the parameters of the gamma correction mapping table when the deviation of the re-calculated average pixel brightness of the core particle in the preset aggregation region from the standard brightness reference is not within the preset range, repeat the operation in step four until the deviation of the average pixel brightness of the core particle in the preset aggregation region from the standard brightness reference is within the preset range.

[0093] Further, in some optional embodiments of the present application, the filling module 25 comprises:

[0094] an input unit configured to input the core particle image after the core particle pose and the core particle brightness are aligned into the pre-trained convolutional neural network model, obtain the output features of each layer of the convolutional neural network model, and up-sample each output feature to the original size of the input core particle image, so that all features are guaranteed to have the same size;

[0095] a second calculation unit configured to obtain the up-sampling results of each layer of the convolutional neural network model, and calculate the cosine similarity between each pixel point of the core particle images after the core particle pose and the core particle brightness are aligned, and the calculation formula of the cosine similarity is:

[0096] ;

[0097] wherein, is the cosine similarity between the core particle image a and the core particle image b at each pixel point, is an up-sampling result of the core particle image a at the i-th layer, is an up-sampling result of the core particle image b at the i-th layer, U is an up-sampling operation, cos_sim is a cosine similarity between two feature vectors, and N is a total number of layers of the convolutional neural network model.

[0098] The integration unit is configured to determine a minimum value of the cosine similarity of each pixel point as the difference information of the corresponding pixel point, integrate the difference information of all pixel points, and form a difference distribution map with the same size as the original image.

[0099] The binarization operation unit is configured to perform a binarization operation on the difference distribution map according to a preset threshold, and determine the non-difference region according to a binarization operation result.

[0100] Further, in some optional embodiments of the present application, the filling module 25 further includes:

[0101] The fusion processing unit is configured to extract pixel values of corresponding positions from a preset number of core particle images according to the non-difference region, perform a conditional weighted fusion processing on a preset number of pixel samples for each position of the non-difference region, and write a conditional weighted fusion processing result to the corresponding position to complete the filling.

[0102] Embodiment three

[0103] Another aspect of the present application also provides an electronic device, please refer to Figure 3 , which is an electronic device in the embodiment three of the present application, including a memory 20, a processor 10, and a computer program 30 stored in the memory and executable on the processor, wherein the processor 10 implements the above-mentioned golden core particle image generation method based on multi-scale feature comparison when executing the computer program 30.

[0104] In some embodiments, the processor 10 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips, configured to run program codes or process data stored in the memory 20, such as executing access restriction programs.

[0105] The memory 20 includes at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. The memory 20 can be an internal storage unit of the electronic device in some embodiments, such as a hard disk of the electronic device. The memory 20 can also be an external storage device of the electronic device in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the memory 20 can include both the internal storage unit and the external storage device of the electronic device. The memory 20 can be used not only to store application software and various data of the electronic device, but also to temporarily store data that has been output or will be output.

[0106] It should be noted that, Figure 3 The illustrated structure does not limit the electronic device, and in other embodiments, the electronic device can include fewer or more components than illustrated, or combine certain components, or arrange the components differently.

[0107] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program. The program is executed by a processor to implement the method for generating a golden core particle image based on multi-scale feature comparison as described above.

[0108] Those skilled in the art can understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a list of executable instructions for implementing the logic function, which can be specifically implemented in any computer readable medium for use by or in conjunction with an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor or other system that can fetch and execute instructions from the instruction execution system, device or apparatus). For the present specification, the "computer readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in conjunction with the instruction execution system, device or apparatus, or in conjunction with these instruction execution systems, devices or apparatus.

[0109] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via an optical scanner, then compiled, interpreted, or otherwise processed, and stored in a computer memory in a form that is then employable by a computer. Examples of computer-readable media that are further within the spirit of the present application are a computer program product, a computer readable storage medium, and a computer.

[0110] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the embodiments described above, various steps or methods can be implemented, for example, by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following techniques, which are well known in the art, can be used to implement the application: a hybrid of the above techniques, discrete logic circuit(s) having logic gates for implementing logic functions upon request pins of the discrete logic circuit(s), programmable logic array(s) (PLAs), field programmable gate array(s) (FPGAs), etc.

[0111] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or more embodiments or examples.

[0112] The above embodiments only express several implementation manners of the application, which are described in a more specific and detailed manner, but cannot be understood as a limitation on the patent scope of the application. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the application, which are all within the protection scope of the application. Therefore, the patent protection scope of the application should be subject to the appended claims.

Claims

1. A method for generating gold core images based on multi-scale feature comparison, characterized in that, The method includes: Obtain images of several core particles of the same type; By frequency domain analysis, several feature points of the core image are determined, and the standard position information of the feature points is determined based on the physical spatial distribution of the feature points. For each core particle, a mapping matrix between each feature point and the standard position information is established, and an affine transformation is used to transform the core particle image to complete the core particle pose alignment. Based on the preset aggregation region, the average pixel brightness of the core image in the feature dimension is statistically analyzed, the gamma correction mapping information of each core relative to the average pixel brightness is solved, and the core image is subjected to pixel increment repair based on the mapping information to complete the core brightness alignment. Based on the pre-trained convolutional neural network model, multi-scale features of the core image after core pose and core brightness alignment are extracted, and indifferent regions are determined based on the multi-scale features, and the indifferent regions are selected for filling. The filled, indistinguishable regions are then subjected to Poisson fusion processing to generate a gold core image. The steps of calculating the average pixel brightness of the core image in the feature dimension according to the preset aggregation region, solving the gamma correction mapping information of each core relative to the average pixel brightness, and performing pixel increment repair on the core image according to the mapping information to complete the core brightness alignment include: Step 1: Based on the preset aggregation area, calculate the average pixel brightness of the core image and use the average pixel brightness as the standard brightness benchmark for all cores. Step 2: For each core, compare the average pixel brightness of its preset aggregation area with the standard brightness benchmark to determine the degree of brightness deviation, and establish a gamma correction mapping table for the corresponding core based on the degree of brightness deviation. Step 3: Traverse each pixel of the core image, determine the corrected brightness value based on the original pixel brightness value and the gamma correction mapping table, and replace the original pixel brightness value. Step 4: Recalculate the average pixel brightness of the core particles within the preset aggregation area, and determine whether the deviation between the recalculated average pixel brightness of the core particles within the preset aggregation area and the standard brightness benchmark is within a preset range. Step 5: If not, readjust the parameters of the gamma correction mapping table and repeat the operation in Step 4 until the deviation between the average pixel brightness of the core in the preset aggregation area and the standard brightness reference is within the preset range.

2. The method for generating gold core images based on multi-scale feature comparison according to claim 1, characterized in that, The step of extracting multi-scale features of the core image after core pose and core brightness alignment based on a pre-trained convolutional neural network model, and determining indifferent regions based on the multi-scale features, includes: The core image, after aligning the core pose and core brightness, is input into a pre-trained convolutional neural network model to obtain the output features of each layer of the convolutional neural network model. The output features are then upsampled to the original size of the input core image to ensure that all features are of the same size. Obtain the upsampling results of each layer of the convolutional neural network model, and calculate the cosine similarity of each pixel between the core particle pose and the core particle brightness aligned images. The minimum cosine similarity of each pixel is determined as the difference information of the corresponding pixel. The difference information of all pixels is integrated to form a difference distribution map with the same size as the original image. The difference distribution map is binarized according to a preset threshold, and the indifferent region is determined based on the binarization result.

3. The method for generating gold core images based on multi-scale feature comparison according to claim 2, characterized in that, The step of selecting the indifferent region for filling includes: Based on the indistinct region, pixel values ​​at corresponding locations are extracted from a preset number of core particle images. For each location within the indistinct region, a preset number of pixel samples undergo conditional weighted fusion processing. The result of this conditional weighted fusion processing is then written to the corresponding location to complete the filling process. Specifically, this includes: For each location in the indifferent region, a preset number of pixel samples are statistically analyzed to identify principal component information. Combined with the reliability of the pixels, a weighted average value is calculated as the final pixel value. The calculated pixel value is written into the corresponding position of the gold core image to complete the filling of the current indifferent region.

4. The method for generating gold core images based on multi-scale feature comparison according to claim 3, characterized in that, In the step of obtaining the upsampling results and calculating the cosine similarity of each pixel point between the core particle images after aligning the core particle pose and core particle brightness, the formula for calculating the cosine similarity is: ； in, Let be the cosine similarity between core image a and core image b at each pixel. The upsampling result of core image a in the i-th layer. Let U be the upsampling result of the core image b at layer i, where U is the upsampling operation, cos_sim is the cosine similarity of two feature vectors, and N is the total number of layers in the convolutional neural network model.

5. The method for generating gold core images based on multi-scale feature comparison according to claim 4, characterized in that, The convolutional neural network model is the ResNet-50 model.

6. A gold core image generation system based on multi-scale feature comparison, characterized in that, For implementing the gold core image generation method based on multi-scale feature comparison as described in any one of claims 1-5, the system comprises: The acquisition module is used to acquire images of several core particles of the same type. The determination module is used to determine several feature points of the core image through frequency domain analysis, and to determine the standard position information of the feature points based on the physical spatial distribution of the feature points. The pose alignment module is used to establish a mapping matrix between each feature point and the standard position information for each core particle, and to transform the core particle image using affine transformation to complete the core particle pose alignment. The brightness alignment module is used to calculate the average pixel brightness of the core image in the feature dimension according to the preset aggregation area, solve the gamma correction mapping information of each core relative to the average pixel brightness, and perform pixel increment repair on the core image according to the mapping information to complete the core brightness alignment. The filling module is used to extract multi-scale features of the core image after core pose and core brightness alignment based on the pre-trained convolutional neural network model, and to determine the indifferent regions based on the multi-scale features, and to select the indifferent regions for filling. The generation module is used to perform Poisson fusion processing on the filled indifferent regions to generate a gold core image.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the gold core image generation method based on multi-scale feature comparison as described in any one of claims 1-5.

8. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the gold core image generation method based on multi-scale feature comparison as described in any one of claims 1-5.

Images