Diffraction image recognition method, system, device and medium based on class activation map

Abstract

The present invention provides a diffraction image recognition method, system, device and computer storage medium based on a class activation map, wherein the method includes: based on a sample data set, determining a model with the highest recognition performance among several pre-selected models as an image recognition model for the diffraction image, and determining a parameter configuration method for the diffraction image, preprocessing the diffraction image based on the parameter configuration method, using a global pooling layer in the class activation map to adjust the structure of the image recognition model to obtain an adjusted image recognition model, and using the adjusted image recognition model to perform image recognition on the preprocessed diffraction image to obtain a class activation map of the pattern spots. While improving the recognition accuracy of the pattern spots in the diffraction image, useless information in the diffraction image can be eliminated to compress the diffraction image in the memory used for transmission, processing and storage.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a diffraction image recognition method, system, device and computer storage medium based on a class activation map. Background Art

[0002] Coherent X-ray diffraction imaging (CDI), an X-ray imaging technique with nanoscale high resolution, has been widely applied in imaging materials science and biological samples. CDI reconstructs images based on coherent X-ray diffraction patterns (hereafter referred to as diffraction images). However, the X-ray facilities used to generate these diffraction images often operate at extremely high data rates, such as 100 GB / s, placing enormous computational, communication, and storage pressures on subsequent image processing.

[0003] To facilitate subsequent processing, communication, and storage, diffraction images are filtered to exclude images of target features, reducing the amount of image data required. Because diffraction images of objects that have hit the target contain Bragg spots, Bragg spot identification methods are commonly used to screen out these images. Currently, methods for identifying extremely small Bragg spots in diffraction images still face technical difficulties, such as insufficient precision, low accuracy, and slow recognition efficiency. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of the present invention is to provide a diffraction image recognition method, system, device and computer storage medium based on class activation map, which can solve the problems of low Bragg speckle recognition rate and low accuracy in existing diffraction images.

[0005] To achieve the above-mentioned purpose and other related purposes, the present invention provides, in a first aspect, a diffraction image recognition method of a class activation map, comprising: based on a sample data set, determining a model with the highest image recognition performance among several pre-selected models as the image recognition model of the diffraction image; based on the sample data set and the image recognition model, determining a parameter configuration method of the diffraction image; according to the parameter configuration method, preprocessing the diffraction image to obtain a preprocessed diffraction image; using the global pooling layer in the class activation map, adjusting the structure of the image recognition model to obtain the adjusted image recognition model; using the adjusted image recognition model, performing image recognition on the preprocessed diffraction image to obtain a class activation map of the pattern spot.

[0006] In one embodiment of the present invention, the diffraction image recognition method of the class activation map further includes: screening out the class activation map containing the thermal region from each of the class activation maps as a selected diffraction image; and intercepting the portion containing the thermal region from each of the selected diffraction images as a final diffraction image.

[0007] In one embodiment of the present invention, the parameter configuration method of the diffraction image is determined based on the sample data set and the image recognition model, including: setting a plurality of preset schemes for the diffraction image parameter configuration; each of the parameter configuration schemes includes a different setting for the same type of parameters in the diffraction image; based on the image recognition model, according to each of the preset schemes, the image recognition performance of the sample data set is tested respectively; and among the parameter configuration schemes, the parameter configuration scheme with the highest image recognition performance is obtained.

[0008] In one embodiment of the present invention, the parameter configuration scheme includes: setting the bit depth of a single channel of the image to 8 bits.

[0009] In one embodiment of the present invention, the structure of the image recognition model is adjusted by using the global pooling layer in the class activation map, including: replacing the fully connected layer after the last convolutional layer in the image recognition model structure with the global pooling layer in the class activation map.

[0010] In one embodiment of the present invention, the preselected model includes a Renet14 model, which is a new model obtained by adjusting the module structure and module parameters of the Renet18 model.

[0011] In one embodiment of the present invention, the module structure adjustment of the Renet18 model includes: deleting the last submodule of the Renet18 model in the module sequence

[0012] In a second aspect, the present invention provides a diffraction image recognition system based on a class activation map, comprising: a model selection module, for determining, based on a sample data set, a model with the highest image recognition performance from among several preselected models, as the image recognition model for the diffraction image; an image parameter configuration module, for determining, based on the sample data set and the image recognition model, a parameter configuration method for the diffraction image; an image preprocessing module, for preprocessing the diffraction image according to the parameter configuration method to obtain a preprocessed diffraction image; a model adjustment module, for adjusting the structure of the image recognition model using a global pooling layer in the class activation map to obtain an adjusted image recognition model; and a recognition processing module, for performing image recognition on the preprocessed diffraction image using the adjusted image recognition model to obtain a class activation map of the pattern.

[0013] In one embodiment of the present invention, the preselected model includes a Renet14 model, which is a new model obtained by adjusting the module structure and module parameters of the Renet18 model.

[0014] In one embodiment of the present invention, the diffraction image recognition system based on class activation maps further includes: a post-processing unit; used to screen out class activation maps containing thermal areas in each of the class activation maps as selected diffraction images; and, in each of the selected diffraction images, intercept the part containing the thermal area as the final diffraction image.

[0015] In a third aspect, the present invention provides an electronic device comprising: a processor, a communication interface, a memory, and a communication bus; the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory is used to store at least one instruction; the instruction enables the processor to execute any of the diffraction image recognition methods based on the class activation map as described above.

[0016] In a fourth aspect, the present invention provides a computer storage medium storing a computer program, wherein the computer program, when executed by a processor, implements any of the above-described diffraction image recognition methods based on a class activation map.

[0017] Compared with the existing technology, the diffraction image recognition method, system, device and computer storage medium of the class activation map provided by the present invention can significantly improve the model prediction accuracy, realize the screening of invalid diffraction images with a higher accuracy, and greatly reduce the storage size of the diffraction image through screening, which provides convenience for subsequent image processing, information extraction, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram showing a flow chart of a diffraction pattern recognition method according to a class activation map of the present invention in one embodiment

[0019] Figure 2 Shown is a schematic diagram of the structural framework of an embodiment of the adjusted image recognition model of the present invention;

[0020] Figure 3 Shown is a schematic diagram of the structure of the Resnet14 model described in the present invention;

[0021] Figure 4 A schematic diagram showing a thermodynamic region of the class activation map obtained by using the diffraction pattern recognition method of the class activation map in one embodiment;

[0022] Figure 5 A schematic diagram showing a thermodynamic region of the class activation map obtained by using the diffraction pattern recognition method of the class activation map in one embodiment (including a cutout of the thermodynamic region range);

[0023] Figure 6A schematic diagram showing the structure of a diffraction pattern recognition system according to an embodiment of the present invention, showing a class activation map;

[0024] Component number description

[0025] 2 Diffraction Image Recognition System Based on Class Activation Map

[0026] 21 Model selection module

[0027] 22 Image parameter configuration module

[0028] 23 Image Preprocessing Module

[0029] 24 Model Adjustment Module

[0030] 25 Identification and processing module DETAILED DESCRIPTION

[0031] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0032] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0033] The present invention provides a diffraction image recognition method, system, device and computer storage medium based on a class activation map, which are used to recognize Bragg spots in a diffraction image to obtain recognition results of the Bragg spots (hereinafter referred to as spots) in the diffraction image.

[0034] See also Figure 1 , which is a flow chart of the diffraction image recognition method based on class activation map.

[0035] like Figure 1 As shown, the method includes the following steps:

[0036] S101, based on a sample data set, testing the recognition performance of several pre-selected models, and determining an image recognition model for the diffraction image according to the test results;

[0037] Specifically, a sample data set of diffraction images to be identified is obtained; the sample data set includes a pre-classified target hit image subset, a non-target hit image subset, and a possible target hit image subset;

[0038] Several neural network models are pre-selected as the preset modules, and image recognition is performed based on the sample data set to obtain recognition performance parameters such as image recognition accuracy, rate and precision of each preset model; wherein, the preset module can be an existing neural network model or a newly constructed neural network model, which is not limited here.

[0039] Among the recognition performance parameters, one or more parameters are selected as evaluation indicators to evaluate the recognition performance of each preset model, so as to obtain the model with the best performance among the preset models as the image recognition model of the diffraction image.

[0040] S102, determining a parameter configuration mode of a diffraction image adapted to the image recognition model based on the sample data set and the image recognition model;

[0041] Specifically, based on the diffraction image to be identified, a plurality of preset schemes for image parameter configuration are pre-set; each preset scheme for parameter configuration includes a different setting for the same type of parameters in the diffraction image;

[0042] The image parameters include but are not limited to the number of image channels, the bit depth of each channel, or other existing image intrinsic parameter information.

[0043] Specifically, based on the image recognition model obtained in step S101, the image recognition performance of the sample data set is tested based on different image parameter configuration schemes, and the optimal parameter configuration scheme is obtained from each of the image parameter configuration schemes as the image parameter configuration method adapted to the image recognition model.

[0044] S103, preprocessing the diffraction image based on the parameter configuration to obtain a preprocessed diffraction image;

[0045] Specifically, based on the image parameter configuration mode obtained in step S102, each diffraction image to be identified is preprocessed to obtain each preprocessed diffraction image.

[0046] S104, using the global pooling layer in the class activation map, adjusting the structure of the image recognition model to obtain an adjusted image recognition model; the adjusted image recognition model includes the global pooling layer;

[0047] like Figure 2As shown, the GAP layer (Global Average Pooling) in the category activation map CAM replaces the fully connected layer in the image recognition model, that is, after the last convolution layer in the first image model, the obtained feature map is globally pooled, and the convolution feature vector of each pixel in the feature map is combined with the weight matrix (W1, W2, ..., Wn in the figure) to obtain a neural network model including a GAP layer as the adjusted image recognition model.

[0048] Among them, the weight matrix is ​​randomly initialized before model training.

[0049] Based on this model, each feature map can be calculated to obtain the contribution of each feature map to the discrimination result, that is, the contribution of each sub-region in the feature map to the discrimination result can be obtained.

[0050] Specifically, for the input image, for a given point (x, y), use f k (x, y) represents the convolution feature vector of the point, and after global pooling, it is used by F k Representative x，y , f k (x, y); for a single pixel k, use As each F in class c k The contribution of is:

[0051]

[0052]

[0053] Among them, S c The score for each category; M c It is the category activation map of each category c, which is used to reflect the contribution of each pixel to each category in the judgment process.

[0054] S105, performing image recognition on the preprocessed diffraction image using the adjusted image recognition model to obtain a class activation map of the pattern;

[0055] Specifically, the adjusted image recognition model is used to perform image recognition on the preprocessed diffraction image to obtain contribution information of each pixel in the image; the contribution information of each pixel in the image is displayed graphically, for example, the pixel with higher contribution has a darker display color, thereby forming a corresponding contribution feature map.

[0056] The contribution feature map is matched with the diffraction image before image recognition, that is, the contribution information of each pixel point is associated with the position of the pixel point in the diffraction image, so as to obtain the spot category activation map of the diffraction image.

[0057] S106, screening out the class activation map containing the thermal region from each of the class activation maps as a selected diffraction image; and intercepting the image containing the thermal region from each of the selected diffraction images to store as a final diffraction image.

[0058] Specifically, in each of the category activation maps, the minimum outward-expanding rectangular area of ​​the thermal area is extracted, the image data within the area is intercepted, and other image data outside the area is discarded to obtain the final diffraction image storage, thereby reducing the image storage space and the processing efficiency of the target image in the coherent X-ray diffraction imaging process.

[0059] The following will describe the diffraction pattern recognition method of the present invention in detail through a specific embodiment.

[0060] In step S101, an open source coherent X-ray image dataset is selected. [1] The L498 dataset in is used as a sample dataset of the diffraction images; in this dataset, each image has been marked as one of the three categories: HIT, MAYBE or MISS.

[0061] The neural network models shown in Table 1 below were pre-selected as preset models. Image recognition was performed on each preset model on an RTX2080 GPU device to obtain image recognition accuracy and recognition speed information, thereby obtaining the image recognition performance of each model. See Table 1 for details. Among them, the model with the best recognition performance was the Resnet14 model, which was used as the image recognition model.

[0062] Table 1 Information of each preset model and image recognition performance of each preset

[0063] Preselected Model Model size (M) Test time (seconds) Accuracy (%) Resnet14 22.3 14.79 85.93 Resnet18 89.5 16.17 74.87 Resnet34 170.0 18.05 75.88 Resnet50 188.0 20.5 69.85 VGG16 1070.0 27.71 65.83 VGG19 1120.0 30.31 68.34 DenseNet 56.2 19.26 59.29 EfficientNet 42.6 30.31 59.80

[0064] It should be noted that the Resnet14 model is a pre-built model, which is a model that further adjusts the structure of the existing Resnet18 model.

[0065] In a specific embodiment, the Resnet14 model is based on the existing Resnet18 model, and the model parameters of each submodule in the Resnet18 model are adjusted in combination with the characteristics of the pattern in the diffraction image, and the last submodule is deleted to form a model; for more details, please refer to Figure 3 , which is a structural diagram of the Resnet14 model, wherein the parameters of each module and the structure of each model are as follows Figure 3 shown.

[0066] In step S102, a configuration scheme of image bit depth parameters is pre-set, as shown in Table 2. The Resnet14 model is used to test the recognition accuracy of the sample data set based on the image parameter configuration schemes shown in Table 2. The highest image parameter configuration scheme is obtained as: 8-bit depth. Based on this parameter configuration scheme, the diffraction images to be recognized are pre-processed, that is, each diffraction image is processed into an 8-bit depth diffraction image.

[0067] Table 2 Information on various image parameter configuration schemes and corresponding image recognition performance

[0068] Parameter configuration scheme Accuracy (%) 16-bit depth 54.49 8-bit depth 71.72

[0069] Based on the image recognition model adjusted in step S104, image recognition is performed on the diffraction image preprocessed to 8-bit depth to obtain the corresponding spot class activation map. The effect after the thermal area display of the class activation map is as follows: Figure 4 and Figure 5 In the diffraction images of each display thermal area, the portion of the display thermal area in the screenshot is saved; for example, Figure 5 In the diffraction pattern shown, only the upper Figure 5 The image in the box is saved. The original image ( Figure 4 ) is 720*720, and the size of the cropped image is 374*348. The size of the cropped diffraction image is about 25% of the original image, that is, about 75% of the storage space is saved during the storage of the diffraction image.

[0070] See also Figure 6 The present invention also provides a diffraction image recognition system based on a class activation map, comprising:

[0071] A model selection module 21 is used to determine, based on the sample data set, a model with the highest image recognition performance from among a number of pre-selected models as the image recognition model for the diffraction image;

[0072] Optionally, the pre-selected model includes a Resnet14 model, which is a new model formed by adjusting the model parameters of each sub-module in the Resnet18 model based on the existing Resnet18 model in combination with the characteristics of the spots in the diffraction image, and deleting the last sub-module.

[0073] An image parameter configuration module 22 determines a parameter configuration mode of the diffraction image based on the sample data set and the image recognition model;

[0074] An image preprocessing module 23, configured to preprocess the diffraction image according to the parameter configuration mode to obtain a preprocessed diffraction image;

[0075] The image parameters include but are not limited to the number of image channels, the bit depth of each channel, or other existing image intrinsic parameter information.

[0076] Optionally, the image parameters include that the bit depth of each channel is 8 bits.

[0077] A model adjustment module 24 adjusts the structure of the image recognition model using the global pooling layer in the class activation map to obtain an adjusted image recognition model;

[0078] The GAP layer (Global Average Pooling) in the class activation map CAM replaces the fully connected layer in the image recognition model, that is, after the last convolution layer in the first image model, the obtained feature map is globally pooled, and the convolution feature vector of each pixel in the feature map is combined with the weight matrix (W1, W2, ..., Wn in the figure) to obtain a neural network model including the GAP layer as the adjusted image recognition model.

[0079] Among them, the weight matrix is ​​randomly initialized before model training;

[0080] Based on this model, each feature map can be calculated to obtain the contribution of each feature map to the discrimination result, that is, the contribution of each sub-region in the feature map to the discrimination result can be obtained.

[0081] Specifically, for the input image, for a given point (x, y), use f k (x, y) represents the convolution feature vector of the point. After global pooling, let F k =∑ x,y fk(x,y); for a single pixel k, use As each F in class c k The contribution of is:

[0082]

[0083]

[0084] Among them, S c The score for each category; M c It is the category activation map of each category c, which is used to reflect the contribution of each pixel to each category in the judgment process.

[0085] The recognition processing module 25 uses the adjusted image recognition model to perform image recognition on the pre-processed diffraction image to obtain a category activation map of the pattern.

[0086] Optionally, the diffraction image recognition system based on class activation maps also includes: a post-processing unit; used to screen out class activation maps containing thermal areas in each of the class activation maps as selected diffraction images; and, in each of the selected diffraction images, intercept a portion containing the thermal area as the final diffraction image.

[0087] The present invention provides an electronic device, comprising: a processor, a memory, a communication interface, and a system bus; the memory and the communication interface are connected to the processor via the system bus and communicate with each other; the memory is used to store at least one instruction, which enables the processor to execute each step of the above-mentioned diffraction image recognition method based on the class activation map.

[0088] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0089] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When invoked by a processor, the program implements the above-described diffraction pattern recognition method based on a class activation map. The computer-readable storage medium may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0090] In summary, the diffraction image recognition method, system, device and computer storage medium based on the class activation map proposed in the present invention determine the model with the highest recognition performance in the pre-selected models as the image recognition model of the diffraction image, and determine the parameter configuration method of the diffraction image based on the image recognition model, and adjust and improve the image recognition model based on the class activation map technology, so as to perform recognition based on the improved image recognition model to obtain the class activation map of the pattern; and screen the diffraction image and extract the thermal area based on the class activation map; not only can the model prediction accuracy be significantly improved, and the invalid diffraction images can be screened with a higher accuracy, but also the storage size of the diffraction image can be greatly reduced through screening, which improves the convenience for subsequent image processing, information extraction, etc.; in addition, the adjustment and improvement of the image recognition model based on the class activation map technology can also achieve the lightweight model, so as to facilitate the further expansion of application scenarios.

[0091] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

[0092] [1]FRMaia, "The coherent x-ray imaging data bank," Nature methods, vol.9, no.9, pp.854–855, 2012.

Claims

1. A diffraction image recognition method based on class activation map, characterized in that: include: Based on the sample data set, a model with the highest image recognition performance is determined from among several pre-selected models to be used as the image recognition model for the diffraction image; wherein one or more recognition performance parameters are selected as evaluation indicators to evaluate the recognition performance of each pre-selected model to obtain the model with the best performance among the pre-selected models; Determining a parameter configuration mode of the diffraction image based on the sample data set and the image recognition model; preprocessing the diffraction image according to the parameter configuration method to obtain a preprocessed diffraction image; Using the global pooling layer in the class activation map to adjust the structure of the image recognition model to obtain an adjusted image recognition model; using the global pooling layer in the class activation map to adjust the structure of the image recognition model includes: replacing the fully connected layer after the last convolutional layer in the image recognition model structure with the global pooling layer in the class activation map; The adjusted image recognition model is used to perform image recognition on the preprocessed diffraction image to obtain a category activation map of the pattern.

2. The diffraction image recognition method based on class activation map according to claim 1, characterized in that: Also includes: In each of the class activation maps, a class activation map including a thermal region is selected as a selected diffraction image; In each of the selected diffraction images, a portion including the thermal region is cut out to serve as a final diffraction image.

3. The diffraction image recognition method based on class activation map according to claim 1, characterized in that: The determining of the parameter configuration mode of the diffraction image based on the sample data set and the image recognition model includes: Setting a plurality of preset schemes for configuring the diffraction image parameters; each of the parameter configuration schemes includes a different setting for the same type of parameters in the diffraction image; Based on the image recognition model, and according to each of the preset schemes, the image recognition performance of the sample data sets is tested respectively; Among the parameter configuration schemes, the parameter configuration scheme with the highest image recognition performance is obtained.

4. The diffraction image recognition method based on class activation map according to claim 3, characterized in that: The parameter configuration scheme includes: setting the bit depth of a single channel of the image to 8 bits.

5. The diffraction image recognition method based on class activation map according to any one of claims 1 to 4, characterized in that: The preselected model includes the Resnet14 model, which is a new model obtained by adjusting the module structure and module parameters of the Resnet18 model.

6. The diffraction image recognition method based on class activation map according to claim 5, characterized in that: The module structure adjustment of the Resnet18 model includes: deleting the last submodule of the Resnet18 model in the module sequence.

7. A diffraction image recognition system based on class activation map, characterized in that: include: A model selection module is used to determine, based on a sample data set, a model with the highest image recognition performance from among a number of pre-selected models, as the image recognition model for the diffraction image; wherein one or more recognition performance parameters are selected as evaluation indicators to evaluate the recognition performance of each pre-selected model to obtain the model with the best performance among the pre-selected models; An image parameter configuration module, which determines a parameter configuration mode of the diffraction image based on the sample data set and the image recognition model; an image preprocessing module, configured to preprocess the diffraction image according to the parameter configuration mode to obtain a preprocessed diffraction image; A model adjustment module, using the global pooling layer in the class activation map to adjust the structure of the image recognition model to obtain an adjusted image recognition model; the adjustment of the structure of the image recognition model using the global pooling layer in the class activation map includes: replacing the fully connected layer after the last convolutional layer in the image recognition model structure with the global pooling layer in the class activation map; The recognition processing module uses the adjusted image recognition model to perform image recognition on the preprocessed diffraction image to obtain a category activation map of the pattern.

8. The diffraction image recognition system based on class activation map according to claim 7, characterized in that: The preselected model includes the Resnet14 model, which is a new model obtained by adjusting the module structure and module parameters of the Resnet18 model.

9. The diffraction pattern recognition system based on class activation map according to claim 7, characterized in that: Also includes: Post-processing unit; for screening out a class activation map containing a thermal region from among the class activation maps as a selected diffraction image; Furthermore, in each of the selected diffraction images, a portion including the thermal region is cut out to serve as a final diffraction image.

10. An electronic device, characterized in that: include: Processor, communication interface, memory and communication bus; The processor, the communication interface and the memory communicate with each other via the communication bus; The memory is used to store at least one instruction; the instruction enables the processor to execute the diffraction image recognition method based on class activation map according to any one of claims 1 to 6.

11. A computer storage medium storing a computer program, wherein: When the computer program is executed by a processor, the diffraction image recognition method based on the class activation map according to any one of claims 1 to 6 is implemented.

Images