Image internal texture classification method and training method of texture classification model

Abstract

The invention provides an image internal texture classification method and a model training method, which are used for the technical field of image processing and computer vision. The classification method comprises the following steps: performing feature extraction on a target texture image to obtain a first multi-scale feature map; performing point multiplication operation on the initial mask image template and the first multi-scale feature map to obtain a target multi-scale feature map; generating a target mask image according to the target multi-scale feature map; performing dynamic feature fusion on every two adjacent target multi-scale feature maps to obtain a fused target multi-scale feature map; performing spatial sampling on the fused target multi-scale feature map by using the target mask image to obtain a target sampling multi-scale feature map; performing cross attention calculation on the target sampling multi-scale feature map to obtain a second multi-scale feature map; and performing texture classification on the second multi-scale feature map by using a classifier to obtain a texture classification result of the target texture image.

Description

Technical Field

[0001] The present disclosure relates to the technical fields of image processing and computer vision, and in particular to a method for classifying internal textures of an image and a method for training a texture classification model. Background Art

[0002] Texture classification is widely present on natural and artificial physical surfaces and plays an important role in many application scenarios, such as 3D modeling, medical image processing, material science or terrain recognition. Texture classification involves extracting spatial structures, patterns and regularities from images to describe the characteristics of the surface of an object.

[0003] Early texture classification methods relied on traditional image processing techniques and hand-designed feature extraction algorithms. These methods extract texture features by statistical analysis of local regions of the image, such as gray-level co-occurrence matrix and local binary pattern. However, such methods cannot fully capture the global information in the image, resulting in poor classification performance in complex scenes.

[0004] With the development of deep learning technology, especially the introduction of convolutional neural networks and visual transformers, texture classification methods have made significant progress. However, when processing images of different resolutions, this method may suffer from poor texture classification performance due to information loss or texture pattern changes. Summary of the invention

[0005] In view of the above problems, the present disclosure provides a method for classifying internal textures of an image and a method for training a texture classification model.

[0006] According to one aspect of the present disclosure, a method for classifying internal texture of an image is provided, comprising: extracting features from a target texture image to obtain N first multi-scale feature maps, wherein each first multi-scale feature map has different semantic information; for the i-th first multi-scale feature map in the N first multi-scale feature maps, performing a dot multiplication operation on an initial mask image template and the i-th first multi-scale feature map to obtain the i-th target multi-scale feature map, thereby obtaining N target multi-scale feature maps; for the i-th target multi-scale feature map in the N target multi-scale feature maps, based on an attention mechanism, generating the i-th target mask image according to the i-th target multi-scale feature map, thereby obtaining N target mask images; for every two of the N target multi-scale feature maps, performing a dot multiplication operation on the initial mask image template and the i-th first multi-scale feature map to obtain the i-th target multi-scale feature map, thereby obtaining N target multi-scale feature maps. Adjacent target multi-scale feature maps are dynamically fused to obtain N fused target multi-scale feature maps; the i-th target mask image in the N target mask images is used to spatially sample the i-th fused target multi-scale feature map in the N fused target multi-scale feature maps to obtain the i-th target sampling multi-scale feature map, and N target sampling multi-scale feature maps are obtained; a densely connected attention mechanism is used to perform cross-attention calculations between the N target sampling multi-scale feature maps to obtain N second multi-scale feature maps; a classifier is used to perform texture classification on the target texture image according to the N second multi-scale feature maps to obtain a texture classification result of the target texture image, wherein N is an even number ≥2, and 0≤i≤N-1.

[0007] Another aspect of the present disclosure provides a training method for an image internal texture classification model, comprising: extracting features from a target sample texture image to obtain N first sample multi-scale feature maps, wherein each first sample multi-scale feature map has different semantic information; for the i-th first sample multi-scale feature map in the N first sample multi-scale feature maps, performing a point multiplication operation on an initial sample mask image template and the i-th first sample multi-scale feature map to obtain the i-th target sample multi-scale feature map, thereby obtaining N target sample multi-scale feature maps; for the i-th target sample multi-scale feature map in the N target sample multi-scale feature maps, based on an attention mechanism, generating the i-th target sample mask image according to the i-th target sample multi-scale feature map to obtain N target sample mask images; performing dynamic feature fusion on every two adjacent target sample multi-scale feature maps in the N target sample multi-scale feature maps to obtain N fused target samples; sample multi-scale feature map; using the i-th target sample mask image in the N target sample mask images, spatially sampling the i-th fused target sample multi-scale feature map in the N fused target sample multi-scale feature maps to obtain the i-th target sample sampling multi-scale feature map, and obtaining N target sample sampling multi-scale feature maps; using the densely connected attention mechanism, cross attention calculation is performed between the N target sample sampling multi-scale feature maps to obtain N second sample multi-scale feature maps; using a classifier, according to the N second sample multi-scale feature maps, texture classification is performed on the target sample texture image to obtain a texture classification prediction result of the target sample texture image, wherein N is an even number ≥2, and 0≤i≤N-1; a loss value is obtained according to the texture classification prediction result and the texture classification true result, and the parameters of the image internal texture classification model are adjusted based on the loss value to obtain a trained image internal texture classification model.

[0008] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0009] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0010] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor.

[0011] According to an embodiment of the present disclosure, by performing feature extraction on a target texture image, N first multi-scale feature maps are obtained, N target multi-scale feature maps are obtained based on the N first multi-scale feature maps and an initial mask image template, N target mask images are generated respectively according to the N target multi-scale feature maps, dynamic feature fusion is performed on the N target multi-scale feature maps to obtain N fused target multi-scale feature maps, based on the N target mask images, the N fused target multi-scale feature maps are spatially sampled, and cross-attention calculation is performed on the obtained N target sampled multi-scale feature pronunciations to obtain N second multi-scale feature maps, and a classifier is used to perform texture classification on the target texture image according to the N second multi-scale feature maps to obtain a classification result of the target texture image. By adopting the technical means of dynamic feature fusion of N multi-scale feature maps, mask processing of multi-scale feature maps and spatial sampling, texture classification of the target texture image is achieved, which at least partially solves the technical problem that the existing methods may cause information loss and texture pattern changes during image scaling, thereby reducing the accuracy of texture classification results. The technical effect of effectively fusing feature maps of different scales while retaining the original resolution, thereby reducing information loss and texture pattern changes and improving the accuracy of texture classification results is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0013] Figure 1 An exemplary system architecture to which an image internal texture classification method and a training method can be applied according to an embodiment of the present disclosure is schematically shown;

[0014] Figure 2 A flowchart of a method for classifying internal texture of an image according to an embodiment of the present disclosure is schematically shown;

[0015] Figure 3 A schematic diagram of determining a target mask image according to an embodiment of the present disclosure is schematically shown;

[0016] Figure 4 A flowchart of a method for obtaining a fused target multi-scale feature map according to an embodiment of the present disclosure is schematically shown;

[0017] Figure 5 A schematic diagram of obtaining N fused target multi-scale feature maps according to an embodiment of the present disclosure is schematically shown;

[0018] Figure 6 A flowchart of a method for training an image internal texture classification model according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 7 A schematic diagram schematically shows a method for classifying internal texture of an image according to an embodiment of the present disclosure;

[0020] Figure 8 A block diagram of an image internal texture classification device according to an embodiment of the present disclosure is schematically shown;

[0021] Fig. 9 A block diagram schematically shows a training device for an image internal texture classification model according to an embodiment of the present disclosure; and

[0022] Fig.10 A block diagram of an electronic device suitable for implementing an image internal texture classification method and a model training method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0024] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0025] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0026] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0027] In the embodiments of the present disclosure, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage, etc. of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information and network security.

[0028] In the embodiments of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0029] The embodiment of the present disclosure provides an image internal texture classification method, comprising: extracting features from a target texture image to obtain N first multi-scale feature maps, wherein each first multi-scale feature map has different semantic information; for the i-th first multi-scale feature map in the N first multi-scale feature maps, performing a point multiplication operation on an initial mask image template and the i-th first multi-scale feature map to obtain the i-th target multi-scale feature map, thereby obtaining N target multi-scale feature maps; for the i-th target multi-scale feature map in the N target multi-scale feature maps, based on an attention mechanism, generating the i-th target mask image according to the i-th target multi-scale feature map, thereby obtaining N target mask images; for every two adjacent first multi-scale feature maps in the N target multi-scale feature maps, performing a point multiplication operation on an initial mask image template and the i-th first multi-scale feature map to obtain the i-th target multi-scale feature map, thereby obtaining N target multi-scale feature maps; The target multi-scale feature maps are dynamically fused to obtain N fused target multi-scale feature maps; the i-th target mask image in the N target mask images is used to spatially sample the i-th fused target multi-scale feature map in the N fused target multi-scale feature maps to obtain the i-th target sampling multi-scale feature map, and N target sampling multi-scale feature maps are obtained; the densely connected attention mechanism is used to perform cross attention calculation on the N target sampling multi-scale feature maps to obtain N second multi-scale feature maps; the classifier is used to perform texture classification on the target texture image according to the N second multi-scale feature maps to obtain the texture classification result of the target texture image, wherein N is an even number ≥2, and 0≤i≤N-1.

[0030] Figure 1 The exemplary system architecture to which the image internal texture classification method and training method can be applied according to the embodiment of the present disclosure is schematically shown. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0031] like Figure 1As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0032] The user may use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only for example).

[0033] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0034] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0035] It should be noted that the image internal texture classification method and model training method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the image internal texture classification device and model training device provided in the embodiment of the present disclosure can generally be set in the server 105. The image internal texture classification method and model training method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the image internal texture classification device and model training device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Alternatively, the image internal texture classification method and model training method provided in the embodiment of the present disclosure can also be performed by the first terminal device 101, the second terminal device 102 or the third terminal device 103, or can also be performed by other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103. Correspondingly, the image internal texture classification device and model training device provided in the embodiments of the present disclosure may also be arranged in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0036] For example, the target texture image may be originally stored in any one of the first terminal device 101, the second terminal device 102 or the third terminal device 103 (for example, the first terminal device 101, but not limited thereto), or stored in an external storage device and may be imported into the first terminal device 101. Then, the first terminal device 101 may locally execute the image internal texture classification method provided by the embodiment of the present disclosure, or send the target texture image to other terminal devices, servers, or server clusters, and the other terminal devices, servers, or server clusters that receive the target texture image may execute the image internal texture classification method provided by the embodiment of the present disclosure.

[0037] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0038] Figure 2 The flowchart of the method for classifying internal texture of an image according to an embodiment of the present disclosure is schematically shown.

[0039] like Figure 2 As shown, the method includes operations S201 to S207.

[0040] In operation S201, feature extraction is performed on a target texture image to obtain N first multi-scale feature maps, wherein each first multi-scale feature map has different semantic information.

[0041] According to an embodiment of the present disclosure, the first multi-scale feature map may be obtained by extracting features from a target texture image using a convolutional neural network, which serves as a feature extraction backbone network.

[0042] According to an embodiment of the present disclosure, the N first multi-scale feature maps are feature maps of different scales extracted by N convolution modules of a convolutional neural network. Each first multi-scale feature map has different semantic information. For example, the multi-scale feature maps extracted by the convolution modules of the first few layers represent the shallow features of the target texture image; the multi-scale feature maps extracted by the convolution modules of the latter few layers represent the deep features of the target texture image. The shallow features can be the fine-grained information of the target texture image, such as color, texture, edges and corners; the deep features can be the coarse-grained information of the target texture image, that is, more abstract semantic information.

[0043] In operation S202, for the ith first multi-scale feature map among the N first multi-scale feature maps, a dot multiplication operation is performed on the initial mask image template and the ith first multi-scale feature map to obtain the ith target multi-scale feature map, thereby obtaining N target multi-scale feature maps.

[0044] According to an embodiment of the present disclosure, the initial mask image template is obtained by masking the preprocessed target texture image to mask the original area in the preprocessed target texture image, where the original area represents the area where the original image information of the target texture image is located in the initial mask image.

[0045] According to an embodiment of the present disclosure, for the i-th first multi-scale feature map of N first multi-scale feature maps, the initial mask image can be used to perform a dot multiplication operation with the i-th first multi-scale feature map to obtain the i-th target multi-scale feature map. Based on the above method, N target multi-scale feature maps can be obtained.

[0046] In operation S203, for the i-th target multi-scale feature map among the N target multi-scale feature maps, based on the attention mechanism, an i-th target mask image is generated according to the i-th target multi-scale feature map to obtain N target mask images.

[0047] According to an embodiment of the present disclosure, an attention mask generation module can be used to determine a learnable weight matrix based on the i-th target multi-scale feature map, and based on the calculation of the learnable weight matrix and the feature matrix of the i-th target multi-scale feature map, the i-th target mask image can be masked to generate N target mask images, so that N target mask images can be masked to generate.

[0048] According to an embodiment of the present disclosure, unimportant feature areas can be masked during the generation of the target mask image, and only feature areas that have important contributions in the texture classification task are retained to guide subsequent spatial sampling operations to reduce the impact of irrelevant feature areas on classification.

[0049] In operation S204, dynamic feature fusion is performed on every two adjacent target multi-scale feature maps in the N target multi-scale feature maps to obtain N fused target multi-scale feature maps.

[0050] According to an embodiment of the present disclosure, a multi-scale feature mixing module can be used to dynamically fuse every two adjacent target multi-scale feature maps in N target multi-scale feature maps based on a maximum feature mapping algorithm to obtain a fused target multi-scale feature map for each target multi-scale feature map.

[0051] According to an embodiment of the present disclosure, a deep feature maximum value in a target multi-scale feature map having deep features may be mapped to an adjacent target multi-scale feature map so as to extract global texture feature information in a target texture image.

[0052] In operation S205, the i-th target mask image among the N target mask images is used to perform spatial sampling on the i-th fused target multi-scale feature map among the N fused target multi-scale feature maps to obtain the i-th target sampling multi-scale feature map, thereby obtaining N target sampling multi-scale feature maps.

[0053] According to an embodiment of the present disclosure, the N target mask images obtained can be used to perform spatial sampling on the N fused target multi-scale feature maps. Spatial sampling is based on the features of the important areas retained in the target mask image, and the features of the corresponding important new areas are extracted from the fused target multi-scale feature map to obtain a sampled multi-scale feature map, that is, a target sampled multi-scale feature map.

[0054] According to an embodiment of the present disclosure, the dimension of the i-th fused target multi-scale feature map can be adjusted, and then the i-th fused target scale feature map can be sampled with a fixed length, and based on the i-th target mask image, the area to be retained marked by the target mask image can be sampled for the i-th fused target multi-scale feature map to obtain the i-th target sampled multi-scale feature map, thereby obtaining N target sampled multi-scale feature maps.

[0055] According to the embodiments of the present disclosure, the target sampling multi-scale feature map retains the important regional features in the texture classification task of the target texture image, which reduces the amount of calculation for the subsequent texture classification task and reduces the calculation burden.

[0056] In operation S206, a densely connected attention mechanism is used to perform cross-attention calculations on the N target sampled multi-scale feature maps to obtain N second multi-scale feature maps.

[0057] In operation S207, a classifier is used to perform texture classification on the target texture image according to the N second multi-scale feature maps to obtain a texture classification result of the target texture image, where N is an even number ≥2, and 0≤i≤N-1.

[0058] According to an embodiment of the present disclosure, the target sampling multi-scale feature map can be calculated through a masked self-attention mechanism to obtain a corresponding self-attention mask image; then the target sampling multi-scale feature map corresponding to the shallow feature is sequentially densely connected with the self-attention mask image through a cross-attention mechanism calculation to obtain N second multi-scale feature maps.

[0059] According to the embodiments of the present disclosure, the introduction of a densely connected attention mechanism can correlate shallow features with deep features in N target sampling multi-scale feature maps, reflecting the correlation between features of different scales between the N target multi-scale feature maps; features of different scales can also be fused with each other, thereby enhancing the understanding of texture feature information in a global context.

[0060] According to an embodiment of the present disclosure, a classifier may be used to perform texture probability calculation on the texture in the target texture image based on N second multi-scale feature maps to obtain a texture classification result of the target texture image.

[0061] According to an embodiment of the present disclosure, by performing feature extraction on a target texture image, N first multi-scale feature maps are obtained, N target multi-scale feature maps are obtained based on the N first multi-scale feature maps and an initial mask image template, N target mask images are generated respectively according to the N target multi-scale feature maps, dynamic feature fusion is performed on the N target multi-scale feature maps to obtain N fused target multi-scale feature maps, based on the N target mask images, the N fused target multi-scale feature maps are spatially sampled, and cross-attention calculation is performed on the obtained N target sampled multi-scale feature pronunciations to obtain N second multi-scale feature maps, and a classifier is used to perform texture classification on the target texture image according to the N second multi-scale feature maps to obtain a classification result of the target texture image. By adopting the technical means of dynamic feature fusion of N multi-scale feature maps, mask processing of multi-scale feature maps and spatial sampling, texture classification of the target texture image is achieved, which at least partially solves the technical problem that the existing methods may cause information loss and texture pattern changes during image scaling, thereby reducing the accuracy of texture classification results. The technical effect of effectively fusing feature maps of different scales while retaining the original resolution, thereby reducing information loss and texture pattern changes and improving the accuracy of texture classification results is achieved.

[0062] According to an embodiment of the present disclosure, feature extraction is performed on a target texture image to obtain N first multi-scale feature maps, including:

[0063] Input the target texture image into the convolutional neural network, extract the first intermediate multi-scale feature map output by each convolution module in the M convolutional neural network, and obtain M first intermediate multi-scale feature maps, where M is a positive integer ≥ 2;

[0064] Based on the pre-set N target convolution modules, N second intermediate multi-scale feature maps are determined from the M first intermediate multi-scale feature maps, wherein the N second intermediate convolution modules are respectively output by convolution operations through the N target convolution modules, wherein 2≤N≤M; for the i-th second intermediate multi-scale feature map among the N second intermediate multi-scale feature maps, a convolution operation is performed on the i-th second intermediate multi-scale feature map to obtain the i-th first multi-scale feature map, thereby obtaining N first multi-scale feature maps.

[0065] According to an embodiment of the present disclosure, a convolutional neural network may include M convolution modules, and the first intermediate multi-scale feature map may be a feature map obtained after convolution processing by each convolution module.

[0066] According to an embodiment of the present disclosure, the target texture image is input into the 0th convolution module (denoted as ) is convolved to obtain the 0th first intermediate multi-scale feature map corresponding to the 0th convolution module; then the 0th first intermediate multi-scale feature map is input into the 1st convolution module of the convolutional neural network (denoted as ) is convolved to obtain the first first intermediate multi-scale feature map corresponding to the first convolution module, and the first first intermediate multi-scale feature map is input into the second convolution module of the convolutional neural network (denoted as ) is convolved to obtain the second first intermediate scale feature map corresponding to the second convolution module, ..., the m-1th first intermediate multi-scale feature map is input into the mth convolution module of the convolutional neural network (denoted as ) is convolved to obtain the mth first intermediate multi-scale feature map corresponding to the mth convolution module, …, and the M-2th first intermediate multi-scale map is input into the M-1th convolution module of the convolutional neural network (denoted as ) is convolved to obtain the M-1th first intermediate multi-scale feature map corresponding to the M-1th convolution module, thereby obtaining M first intermediate multi-scale feature maps. Where 0≤m≤M-1.

[0067] According to an embodiment of the present disclosure, the convolution processing from the 0th convolution module to the M-1th convolution module of the convolutional neural network can be a feature extraction process from shallow features to deep features of the target texture image. Each of the M first intermediate multi-scale feature maps has different semantic information.

[0068] According to an embodiment of the present disclosure, the N target convolution modules may be determined from the M convolution modules included in the convolutional neural network. Based on the N convolution modules, the N first intermediate multi-scale feature maps corresponding to the N convolution modules may be determined from the M first intermediate multi-scale feature maps as the second intermediate multi-scale feature maps.

[0069] According to an embodiment of the present disclosure, for example, the convolutional neural network includes the 0th to the 4th convolutional modules, which are 5 convolutional modules in total. The target texture image is input into the convolutional neural network, and the first intermediate multi-scale feature map output by each of the 5 convolutional modules in the convolutional neural network is extracted respectively, so as to obtain 5 first intermediate multi-scale feature maps. The 1st to the 4th convolutional modules are determined as the target convolutional modules (i.e., ~ ), the first first intermediate multi-scale feature map corresponding to the first convolution module to the fourth first intermediate multi-scale feature map corresponding to the fourth convolution module are used as the four second intermediate multi-scale feature maps.

[0070] According to an embodiment of the present disclosure, for the i-th second intermediate multi-scale feature map among the N second intermediate multi-scale feature maps, a convolution operation can be performed on the i-th second intermediate multi-scale feature map to obtain the i-th first multi-scale feature map. Based on the above method, N first multi-scale feature maps can be obtained.

[0071] According to an embodiment of the present disclosure, for example, each of the four second intermediate multi-scale feature maps obtained above can be subjected to a convolution operation to obtain a first multi-scale feature map for each second intermediate multi-scale feature map, that is, to obtain four first multi-scale feature maps.

[0072] According to an embodiment of the present disclosure, a convolution operation is performed on the N second intermediate multi-scale feature maps in order to make the dimensions of the obtained N first multi-scale feature maps consistent.

[0073] According to an embodiment of the present disclosure, based on the attention mechanism, an i-th target mask image is generated according to the i-th target multi-scale feature map to obtain N target mask images, including: encoding the i-th target multi-scale feature map to obtain the i-th importance weight matrix corresponding to the i-th target multi-scale feature map; obtaining the i-th target mask image according to the i-th target multi-scale feature map and the i-th importance weight matrix to obtain N target mask images.

[0074] According to an embodiment of the present disclosure, an autoencoder can be used to encode the i-th target multi-scale feature map, extract the most representative features, and then decode the extracted most representative features. In this process, the autoencoder can generate an i-th importance weight matrix corresponding to the i-th target multi-scale feature map.

[0075] According to an embodiment of the present disclosure, the dimension size of the importance weight matrix is ​​consistent with the dimension size of the target multi-scale feature map, and is used to measure the importance of each feature area in the target multi-scale map.

[0076] According to an embodiment of the present disclosure, the i-th importance weight matrix and the i-th target multi-scale feature map can be used for weighted calculation to obtain the i-th target mask image. Based on the above method, N target mask images can be obtained.

[0077] According to an embodiment of the present disclosure, the i-th target multi-scale feature map and the i-th importance weight matrix both include C channels.

[0078] According to an embodiment of the present disclosure, according to the i-th target multi-scale feature map and the importance weight matrix, the i-th target mask image is obtained, and N target mask images are obtained, including:

[0079] For the cth channel among C channels, the i-th target mask image is obtained according to the feature matrix of the c-th channel of the i-th target multi-scale feature map and the feature matrix of the c-th channel of the i-th importance weight matrix, and N target mask images are obtained, where C is a positive integer ≥, 1≤c≤C.

[0080] According to an embodiment of the present disclosure, according to the feature matrix of the cth channel of the i-th target multi-scale feature map and the feature matrix of the cth channel of the i-th importance weight matrix, an i-th target mask image is obtained, and N target mask images are obtained, including: performing a dot multiplication operation on the feature matrix of the c-th channel of the i-th target multi-scale feature map and the feature matrix of the c-th channel of the i-th importance weight matrix to obtain a mask matrix of the c-th channel, and obtaining a mask matrix of C channels; performing a sum operation on the mask matrices of the C channels to obtain the i-th target mask image, and obtaining N target mask images.

[0081] According to an embodiment of the present disclosure, the i-th target multi-scale feature map may include C channels, and the i-th importance weight matrix may also include C channels. The feature matrix of each channel in the i-th target multi-scale feature map and the feature matrix of each channel in the i-th importance weight matrix may be obtained.

[0082] According to an embodiment of the present disclosure, for the cth channel among C channels, the feature matrix of the cth channel in the i-th target multi-scale feature map can be dot-multiplied with the feature matrix of the cth channel in the i-th importance weight matrix to obtain a mask matrix of the cth channel.

[0083] According to an embodiment of the present disclosure, the above method may be used to calculate the mask matrix of each channel, thereby obtaining the mask matrices of C channels.

[0084] According to an embodiment of the present disclosure, the mask matrices of the obtained C channels can be summed and normalized using a softmax activation function to obtain the i-th target intermediate mask image, and the i-th target intermediate mask image can be binarized to obtain the i-th target mask image.

[0085] According to an embodiment of the present disclosure, the above method can be used to calculate the target mask image corresponding to each target multi-scale feature map, thereby obtaining N target mask images.

[0086] According to an embodiment of the present disclosure, the target mask image It can be obtained through formula (1-1)~(1-2), that is:

[0087] (1-1);

[0088] , (1-2);

[0089] Where f(h) is the threshold function, for The eigenvalue of the hth element in the output target intermediate mask image matrix, 1≤h≤H, H is the number of all elements, H is a positive integer ≥1; C is a positive integer ≥1, c∈{1,2,…,C}, E(·) represents the averaging operation, μ is The average of the eigenvalues ​​of all elements in the output target intermediate mask image matrix.

[0090] According to an embodiment of the present disclosure, for example, the activation function in the above formula (1-1) can be used The feature matrix corresponding to the i-th target intermediate mask image corresponding to the i-th target multi-scale feature map is obtained. The feature matrix corresponding to the i-th target intermediate mask image contains H elements. For the feature value of the h-th element among the H elements, the feature value f(h) of the h-th element is compared with the average value μ. When the feature value f(h) of the h-th element ≥ the average value μ, the h-th element is binarized to 1; when the feature value f(h) of the h-th element < the average value μ, the h-th element is binarized to 0. In this way, the H elements in the feature matrix corresponding to the i-th target intermediate mask image can be binarized to obtain the i-th target mask image. The above method can be used to obtain N target mask images corresponding to N target multi-scale feature maps.

[0091] The target mask corresponding to the jth element in the feature matrix of the i-th target multi-scale feature map can be calculated in sequence, that is, when calculating When the value of is greater than or equal to the value of μ, The value is 1, that is, the mask value corresponding to the target mask of the jth element of the i-th target mask image is 1; when calculating When the value of is less than the value of μ, The value is 0, that is, the mask value corresponding to the target mask of the jth element of the i-th target mask image is 0; the target masks corresponding to the J elements are calculated in sequence to obtain the i-th target mask image corresponding to the i-th target multi-scale feature image, and obtain N target mask images.

[0092] According to the embodiments of the present disclosure, based on the above method, N target mask images can be obtained. The present disclosure obtains the target mask image by performing mask processing on the target multi-scale feature image to include important feature areas in the target multi-scale feature image and reduce the interference of irrelevant areas on the texture classification task.

[0093] Figure 3 A schematic diagram of determining a target mask image according to an embodiment of the present disclosure is schematically shown.

[0094] like Figure 3As shown in schematic diagram 300, the target multi-scale feature map can be input into the autoencoder for encoding to obtain an importance weight matrix corresponding to the feature matrix of the target multi-scale feature map, and a dot multiplication operation is performed based on the eigenvalue of each channel in the target multi-scale feature map and the eigenvalue of each channel in the importance weight matrix, and the results of the dot multiplication operation of each channel value are summed and normalized to obtain a target mask image corresponding to the target multi-scale feature map.

[0095] Figure 4 A flowchart of a method for obtaining a fused target multi-scale feature map according to an embodiment of the present disclosure is schematically shown.

[0096] like Figure 4 As shown, the method 400 may include: operations S401-S402.

[0097] In operation S401, the kth target multi-scale feature map is used to perform maximum feature mapping on the k-1th target multi-scale feature map to obtain the k-1th fused target multi-scale feature map.

[0098] In operation S402, the k-1th fused target multi-scale feature map is used to perform maximum feature mapping on the k-2th target multi-scale feature map to obtain the k-2th fused target multi-scale feature map, and N fused target multi-scale feature maps are obtained, where 0<k≤N-1. When k is N-1, the N-1th fused target multi-scale feature map is the kth target multi-scale feature map.

[0099] According to an embodiment of the present disclosure, the maximum feature map can be to compare the sizes of feature values ​​at the same position in two adjacent target multi-scale feature maps, map the larger feature value in the target multi-scale feature map representing the deep features to the same position in another target multi-scale feature map, and cover the smaller value in the other target multi-scale feature map.

[0100] According to an embodiment of the present disclosure, layer normalization may be performed on N target multi-scale feature maps respectively to smooth the distribution of the target multi-scale feature maps.

[0101] According to an embodiment of the present disclosure, for example, for the kth target multi-scale feature map and the k-1th target multi-scale feature map, the kth target multi-scale feature map represents a feature map of deep features relative to the k-1th target multi-scale feature map, and the eigenvalue sizes of the elements at the same position in the feature matrix of the k-1th target multi-scale feature map can be compared. If the eigenvalue of the feature matrix of the kth target multi-scale feature map is larger, the smaller eigenvalue at the position of the feature matrix of the k-1th target multi-scale feature map is replaced with the larger eigenvalue of the feature matrix of the k-1th target multi-scale feature map; if the eigenvalue of the feature matrix of the k-1th target multi-scale feature map is larger, the larger eigenvalue of the feature matrix of the k-1th target multi-scale feature map is retained, so that the k-1th fused target multi-scale feature map can be obtained.

[0102] According to an embodiment of the present disclosure, the eigenvalues ​​of the elements at the same position in the feature matrix of the k-1th fused target multi-scale feature map and the feature matrix of the k-2th target multi-scale feature map are compared. If the eigenvalue of the feature matrix of the k-1th fused target multi-scale feature map is larger, the smaller eigenvalue at the position of the feature matrix of the k-2th target multi-scale feature map is replaced with the larger eigenvalue of the feature matrix of the k-1th fused target multi-scale feature map; if the eigenvalue of the feature matrix of the k-2th target multi-scale feature map is larger, the larger eigenvalue of the feature matrix of the k-2th target multi-scale feature map is retained, so that the k-2th fused target multi-scale feature map can be obtained.

[0103] According to an embodiment of the present disclosure, based on the above method, the eigenvalues ​​of the elements at the same position in the feature matrix of the first fused target multi-scale feature map and the feature matrix of the 0th target multi-scale feature map can be compared. If the eigenvalue of the feature matrix of the first fused target multi-scale feature map is larger, the smaller eigenvalue at that position of the feature matrix of the 0th target multi-scale feature map is replaced with the larger eigenvalue of the feature matrix of the first fused target multi-scale feature map; if the eigenvalue of the feature matrix of the 0th target multi-scale feature map is larger, the larger eigenvalue of the feature matrix of the 0th target multi-scale feature map is retained, so that the 0th fused target multi-scale feature map can be obtained.

[0104] According to the embodiments of the present disclosure, when k is N-1, the obtained N-1th fused target multi-scale feature map is the kth target multi-scale feature map. Based on the above-obtained 0th fused target multi-scale feature map to the N-th fused target multi-scale feature map, N fused target multi-scale feature maps are obtained.

[0105] Figure 5A schematic diagram of obtaining N fused target multi-scale feature maps according to an embodiment of the present disclosure is schematically shown.

[0106] like Figure 5 As shown in the schematic diagram 500, for example, the target multi-scale feature map can be represented in matrix form from representing deep features to representing shallow features as follows: , , …, , , …, , After normalizing the above target feature maps, the maximum feature map is used to convert and Perform the maximum feature mapping and and

[0107] Compare the eigenvalues ​​at the same position in The eigenvalue of this position is larger,

[0108] The smaller eigenvalue at that position in is replaced by If The eigenvalue of this position is larger, so it is retained The eigenvalue of the position in the The maximum feature map of the target is obtained after the N-2 fusion of the multi-scale feature map At this time, the target multi-scale feature map after the N-1 fusion for itself.

[0109] Based on the above method, the target multi-scale feature map after N-2 fusion is and Perform the maximum feature mapping to obtain the target multi-scale feature map after the N-3 fusion , and so on, the k-th fused target multi-scale feature map and Perform the maximum feature mapping to obtain the target multi-scale feature map after the k-1 fusion ,…, the target multi-scale feature map after the first fusion and Perform the maximum feature mapping to obtain the target multi-scale feature map after the 0th fusion , thus, N fused target multi-scale feature maps are obtained.

[0110] According to an embodiment of the present disclosure, a densely connected attention mechanism is used to perform cross attention calculations between N target sampled multi-scale feature maps to obtain N second multi-scale feature maps, including:

[0111] For the i-th target sampling multi-scale feature map of N target sampling multi-scale feature maps, linear projection is performed on the i-th target sampling multi-scale feature map to obtain a query matrix, a key matrix and a value matrix corresponding to the i-th target sampling multi-scale feature map; based on the query matrix, the key matrix, the value matrix and the initial mask image template, the i-th intermediate second multi-scale feature map corresponding to the i-th target sampling multi-scale feature map is obtained, and N intermediate second multi-scale feature maps are obtained; for the i-th intermediate second multi-scale feature map of the N intermediate second multi-scale feature maps, dense connection attention calculation is performed on the first i-1 target sampling multi-scale feature maps and the i-th intermediate second multi-scale feature map to obtain the i-th second multi-scale feature map, and N second multi-scale feature maps are obtained, where, when i is 0, the i-th intermediate second multi-scale feature map is the i-th second multi-scale feature map, and 0≤i≤N-1.

[0112] According to an embodiment of the present disclosure, the query matrix for the i-th target sampling multi-scale feature map can be obtained by linearly projecting the i-th target sampling multi-scale feature map for the N target sampling multi-scale feature maps. , key matrix Sum Matrix .

[0113] According to an embodiment of the present disclosure, the feature dimension of the i-th target sampling multi-scale feature map is determined , the query matrix of the multi-scale feature map of the i-th target sampling can be obtained , key matrix , value matrix , feature dimension And the initial mask image template, get the i-th intermediate second multi-scale feature map , Based on the above method, N intermediate second multi-scale feature maps can be obtained.

[0114] According to an embodiment of the present disclosure, the i-th intermediate second multi-scale feature map It can be obtained by the following formula (2):

[0115] (2);

[0116] in, is the key matrix The transpose of is the initial mask image template.

[0117] According to an embodiment of the present disclosure, N target sampling multi-scale feature maps can be expressed as , , …, …、 , , and the corresponding N intermediate second multi-scale feature maps can be expressed as , , …, , …, , .

[0118] According to an embodiment of the present disclosure, for the i-th intermediate second multi-scale feature map among the N intermediate second multi-scale feature maps, for example , we can use densely connected attention calculation to sample the multi-scale feature maps of the first i-1 targets (i.e., , , …, ) and the i-th intermediate second multi-scale feature map Performing densely connected cross attention calculations, we can obtain the second multi-scale feature map with the i-th intermediate The corresponding second multi-scale feature map .

[0119] According to an embodiment of the present disclosure, for example, and Performing dense connection attention calculations, we can get the same The corresponding second multi-scale feature map ; , and Performing dense connection attention calculations, we can get the same The corresponding second multi-scale feature map , and so on, we can , , …, …、 and Performing dense connection attention calculations, we can get the same The corresponding second multi-scale feature map , and obtain N second multi-scale feature maps.

[0120] According to an embodiment of the present disclosure, when i is 0, that is, for the middle second multi-scale feature map , for which no densely connected attention calculation is performed, the 0th second multi-scale feature map It is the middle second multi-scale feature map itself.

[0121] According to an embodiment of the present disclosure, the second multi-scale feature map It can be calculated by formula (3), namely:

[0122] (3);

[0123] in, Represented as a query matrix; Represented as a key matrix and a value matrix.

[0124] According to an embodiment of the present disclosure, N second multi-scale feature maps can be obtained based on the above formula (3).

[0125] According to an embodiment of the present disclosure, the classifier includes G; wherein, according to the N second multi-scale feature maps, the target texture image is texture classified to obtain the texture classification result of the target texture image, including:

[0126] Based on preset rules, two second multi-scale feature maps are determined from N second multi-scale feature maps; a g-th classifier among G classifiers is used to perform texture classification on a target texture image according to the two second multi-scale feature maps, and a texture classification result calculated by the g-th classifier is obtained, and a texture classification result calculated by the G classifiers is obtained, wherein 1≤G≤N / 2, 1≤g≤G; a texture classification result of the target texture image is obtained according to the texture classification results calculated by the G classifiers.

[0127] According to an embodiment of the present disclosure, two second multi-scale feature maps can be determined from N second multi-scale feature maps based on the preset rule of Ni-1. For example, if the feature map is determined to be the i-th one from the N second multi-scale feature maps, another second multi-scale feature map is determined according to the rule of Ni-1.

[0128] According to an embodiment of the present disclosure, for the i-th second multi-scale feature map of N second multi-scale feature maps, the global feature information in the i-th second multi-scale feature map can be extracted to obtain the global feature information of the i-th second multi-scale feature map. , based on the above method, the global feature information of N second multi-scale feature maps is obtained.

[0129] According to an embodiment of the present disclosure, for example, when i is 0, based on the above preset rules, the first classifier among the G classifiers can be used to classify the global feature information of the 0th second multi-scale feature map. and the global feature information of the N-0-1th second multi-scale feature map , the texture type appearing inside the target texture image is calculated, and the texture classification result of the target texture image can be obtained ;

[0130] According to an embodiment of the present disclosure, when i is 1, the second classifier among the G classifiers can be used to obtain the global feature information of the first second multi-scale feature map. And the global feature information of the N-1-1th second multi-scale feature map , the texture type appearing inside the target texture image is calculated, and the texture classification result of the target texture image can be obtained .

[0131] According to an embodiment of the present disclosure, based on the above method, the g-th classifier among the G classifiers can be used to obtain the global feature information of the i-th second multi-scale feature map. and the global feature information of the Ni-1th second multi-scale feature map , the texture type appearing inside the target texture image is calculated, and the texture classification result of the target texture image can be obtained .

[0132] According to an embodiment of the present disclosure, the texture classification result of the target texture image It can be calculated by formula (4), namely:

[0133] (4);

[0134] Among them, 1≤g≤N / 2; classifier is a linear classifier; CLS is the global feature information.

[0135] According to an embodiment of the present disclosure, based on the above method, texture classification results calculated by G classifiers can be obtained, and each texture classification result can include the predicted probability values ​​of various texture types inside the target image. For example, each texture classification result includes probability values ​​for three types A / B / C. The probability values ​​of each of the three types A / B / C in the G texture classification results can be weighted and averaged, and the probability values ​​of the three types A / B / C after weighted averaging can be obtained respectively. The type corresponding to the largest probability value in the probability value is taken as the texture classification result of the target texture image.

[0136] According to an embodiment of the present disclosure, the method further includes: performing scale normalization preprocessing on the target texture image.

[0137] According to the embodiments of the present disclosure, since the resolution of the target texture image may vary greatly in different scenes, simple image scaling may result in the loss of important texture information. Therefore, before inputting the target texture image into the convolutional neural network for feature extraction, the target texture image needs to be preprocessed by scale normalization.

[0138] According to an embodiment of the present disclosure, the target texture image may be scaled normalized by filling with "0". Specifically, the image size may be unified by filling "0" pixels in the lower right corner of each input target texture image.

[0139] According to an embodiment of the present disclosure, the initial mask image It is also obtained by filling with "0", and its size is the same as the target texture image obtained after filling with "0". The initial mask image is used to mark which pixel areas are "0"-filled areas, thereby avoiding the interference of these "0"-filled areas on the texture classification task.

[0140] According to an embodiment of the present disclosure, by padding the target texture image with "0", not only the resolution information of the original target texture image can be retained, but also a uniform size can be provided for subsequent feature extraction and attention calculation.

[0141] Figure 6 The flowchart of the training method of the image internal texture classification model according to the embodiment of the present disclosure is schematically shown.

[0142] like Figure 6 As shown, the method 600 may include: operations S601 to S608.

[0143] In operation S601, feature extraction is performed on a target sample texture image to obtain N first sample multi-scale feature maps, wherein each first sample multi-scale feature map has different semantic information.

[0144] In operation S602, for the i-th first sample multi-scale feature map of the N first sample multi-scale feature maps, a dot multiplication operation is performed on the initial sample mask image template and the i-th first sample multi-scale feature map to obtain the i-th target sample multi-scale feature map, thereby obtaining N target sample multi-scale feature maps.

[0145] In operation S603, for the i-th target sample multi-scale feature map among the N target sample multi-scale feature maps, based on the attention mechanism, an i-th target sample mask image is generated according to the i-th target sample multi-scale feature map to obtain N target sample mask images.

[0146] In operation S604, dynamic feature fusion is performed on every two adjacent target sample multi-scale feature maps in the N target sample multi-scale feature maps to obtain N fused target sample multi-scale feature maps.

[0147] In operation S605, the i-th target sample mask image among the N target sample mask images is used to perform spatial sampling on the i-th fused target sample multi-scale feature map among the N fused target sample multi-scale feature maps to obtain the i-th target sample sampling multi-scale feature map, and obtain N target sample sampling multi-scale feature maps.

[0148] In operation S606, a densely connected attention mechanism is used to perform cross-attention calculations on the N target sample multi-scale feature maps to obtain N second sample multi-scale feature maps.

[0149] In operation S607, a classifier is used to perform texture classification on the target sample texture image according to the N second sample multi-scale feature maps to obtain a texture classification prediction result of the target sample texture image, where N is an even number ≥2, and 0≤i≤N-1.

[0150] In operation S608, a loss value is obtained according to the texture classification prediction result and the texture classification true result, and the parameters of the image internal texture classification model are adjusted based on the loss value to obtain a trained image internal texture classification model.

[0151] According to an embodiment of the present disclosure, the specific description of operations S601~S607 for training the image internal texture classification model corresponds to the description of operations S201~S201 in the above-mentioned image internal texture classification method, and the description of its training method can refer to the description of the image internal texture classification method.

[0152] According to the embodiments of the present disclosure, through the description of operations S601~S607, the texture classification prediction result can be obtained, and then based on the loss function, the loss value is obtained according to the texture classification prediction result and the texture true result, and the parameters of the image internal texture analysis model are adjusted based on the loss value, and then the next round of training is carried out to obtain the corresponding loss value of the next round, until the preset conditions are met, the training is stopped, and the trained image internal texture analysis model is obtained.

[0153] According to an embodiment of the present disclosure, the preset condition may be a preset loss threshold. When the loss value meets the preset loss threshold, training is stopped. The preset condition may also be the number of training iterations. When the number of training iterations meets the preset number of iterations, training is stopped.

[0154] According to an embodiment of the present disclosure, the loss function may be a cross entropy loss function L, and the loss value may be obtained based on the following formula (5), namely:

[0155] (5);

[0156] in, is the predicted label of the rth texture classification; T is the true label of the texture classification; The cross entropy loss between the predicted label and the true label for the rth texture classification.

[0157] According to an embodiment of the present disclosure, the training loss value of the image internal texture classification model can be obtained by the above formula (5), and the parameters of the model are adjusted based on the loss value. When the preset conditions are met, the training is stopped to obtain a trained image internal texture classification model.

[0158] Figure 7 A schematic diagram of an image internal texture classification method according to an embodiment of the present disclosure is schematically shown.

[0159] like Figure 7 As shown in the schematic diagram 700, the target texture image is preprocessed by filling "0" to obtain a preprocessed target texture image 701; at the same time, an initial mask image template 702 is generated. The preprocessed target texture image 701 is input into a backbone neural network 703 having M convolution modules to obtain M first intermediate multi-scale feature maps 704; based on the pre-set N target convolution modules, N second intermediate multi-scale feature maps 705 are determined from the M first intermediate multi-scale feature maps 704; convolution operations are performed on the N second intermediate multi-scale feature maps 705 respectively to obtain N first multi-scale feature maps 706; the N first multi-scale feature maps 706 are respectively multiplied by the initial mask image template 702 to obtain N target multi-scale feature maps 707; and the mask image template 707 is respectively generated according to the N target multi-scale feature maps 707. The code generates N target mask images 708; the N target multi-scale feature maps 707 are dynamically fused to obtain N fused target multi-scale feature maps 709; the N target mask images 708 are used to perform spatial sampling on the N fused target multi-scale feature maps 709 to obtain N target sampled multi-scale feature maps 710; based on the dense connection attention mechanism, the N target sampled multi-scale feature maps 710 are densely connected and cross-attention is calculated to obtain N second multi-scale feature maps 711; the classifier is used to obtain the texture classification result 712 of the target texture image according to the N second multi-scale feature maps 711.

[0160] According to an embodiment of the present invention, Table 1 is a quantitative comparison table of the classification results of the method provided by the present invention and the DeepNet, MAPNet, DSRNet, CLASSNet, DFAEN and RADAM methods on four different data sets, and the classification accuracy is expressed in percentage form, as shown in Table 1:

[0161] Table 1

[0162]

[0163] As can be seen from Table 1, the performance of the method provided by the present invention on the four data sets is better than that of the existing methods. Compared with the existing methods, the accuracy of the present invention is significantly improved, especially on data sets 1 and 2, the accuracy reaches 97.1% and 95.3%. And whether the method uses the RestNet18 network or the ConvNeXt in ImageNet-21K network, the accuracy of the present invention is higher than that of other methods. In other words, even if the method provided by the present invention uses different backbone networks, its accuracy is better than that of other methods.

[0164] Figure 8 The block diagram of the image internal texture classification device according to an embodiment of the present disclosure is schematically shown.

[0165] like Figure 8 As shown, the device 800 includes: a first image feature extraction module 810, a first feature operation module 820, a first feature mask module 830, a first dynamic feature fusion module 840, a first feature sampling module 850, a first feature cross-attention calculation module 860 and a first texture classification module 870.

[0166] The first image feature extraction module 810 is used to extract features from the target texture image to obtain N first multi-scale feature maps, wherein each first multi-scale feature map has different semantic information.

[0167] The first feature operation module 820 is used to perform a point multiplication operation on the initial mask image template and the i-th first multi-scale feature map among the N first multi-scale feature maps to obtain the i-th target multi-scale feature map, thereby obtaining N target multi-scale feature maps.

[0168] The first feature mask module 830 is used to generate an i-th target mask image according to the i-th target multi-scale feature map among the N target multi-scale feature maps based on the attention mechanism, so as to obtain N target mask images.

[0169] The first dynamic feature fusion module 840 is used to perform dynamic feature fusion on every two adjacent target multi-scale feature maps in the N target multi-scale feature maps to obtain N fused target multi-scale feature maps.

[0170] The first feature sampling module 850 is used to use the i-th target mask image in the N target mask images to perform spatial sampling on the i-th fused target multi-scale feature map in the N fused target multi-scale feature maps to obtain the i-th target sampling multi-scale feature map, and obtain N target sampling multi-scale feature maps.

[0171] The first feature cross-attention calculation module 860 is used to utilize a densely connected attention mechanism to perform cross-attention calculations on N target sampled multi-scale feature maps to obtain N second multi-scale feature maps.

[0172] The first texture classification module 870 is used to use a classifier pair to perform texture classification on the target texture image according to N second multi-scale feature maps to obtain a texture classification result of the target texture image, wherein N is an even number ≥ 2, and 0≤i≤N-1.

[0173] Fig. 9 A block diagram of a training device for an image internal texture classification model according to an embodiment of the present disclosure is schematically shown.

[0174] like Fig. 9 As shown, the device 900 includes: a second image feature extraction module 910, a second feature operation module 920, a second feature mask module 930, a second dynamic feature fusion module 940, a second feature sampling module 950, a second feature cross-attention calculation module 960, a second texture classification module 970 and a parameter adjustment module 980.

[0175] The second image feature extraction module 910 is used to extract features from the target sample texture image to obtain N first sample multi-scale feature maps, wherein each first sample multi-scale feature map has different semantic information.

[0176] The second feature operation module 920 is used to perform a point multiplication operation on the initial sample mask image template and the i-th first sample multi-scale feature map among the N first sample multi-scale feature maps to obtain the i-th target sample multi-scale feature map, thereby obtaining N target sample multi-scale feature maps.

[0177] The second feature mask module 930 is used to generate an i-th target sample mask image according to the i-th target sample multi-scale feature map among the N target sample multi-scale feature maps based on the attention mechanism, so as to obtain N target sample mask images.

[0178] The second dynamic feature fusion module 940 is used to perform dynamic feature fusion on every two adjacent target sample multi-scale feature maps in the N target sample multi-scale feature maps to obtain N fused target sample multi-scale feature maps.

[0179] The second feature sampling module 950 is used to use the i-th target sample mask image in the N target sample mask images to perform spatial sampling on the i-th fused target sample multi-scale feature map in the N fused target sample multi-scale feature maps to obtain the i-th target sample sampling multi-scale feature map and obtain N target sample sampling multi-scale feature maps.

[0180] The second feature cross-attention calculation module 960 is used to utilize the densely connected attention mechanism to perform cross-attention calculations on the N target sample sampling multi-scale feature maps to obtain N second sample multi-scale feature maps.

[0181] The second texture classification module 970 is used to use a classifier to perform texture classification on the target sample texture image according to N second sample multi-scale feature maps to obtain a texture classification prediction result of the target sample texture image, wherein N is an even number ≥ 2, and 0≤i≤N-1.

[0182] The parameter adjustment module 980 is used to obtain a loss value according to the texture classification prediction result and the texture classification actual result, and adjust the parameters of the image internal texture classification model based on the loss value to obtain a trained image internal texture classification model.

[0183] According to an embodiment of the present disclosure, any multiple of the first image feature extraction module 810, the first feature operation module 820, the first feature mask module 830, the first dynamic feature fusion module 840, the first feature sampling module 850, the first feature cross attention calculation module 860 and the first texture classification module 870 or the second image feature extraction module 910, the second feature operation module 920, the second feature mask module 930, the second dynamic feature fusion module 940, the second feature sampling module 950, the second feature cross attention calculation module 960, the second texture classification module 970 and the parameter adjustment module 980 can be combined in one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present disclosure, at least one of the first image feature extraction module 810, the first feature operation module 820, the first feature mask module 830, the first dynamic feature fusion module 840, the first feature sampling module 850, the first feature cross-attention calculation module 860 and the first texture classification module 870 or the second image feature extraction module 910, the second feature operation module 920, the second feature mask module 930, the second dynamic feature fusion module 940, the second feature sampling module 950, the second feature cross-attention calculation module 960, the second texture classification module 970 and the parameter adjustment module 980 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the first image feature extraction module 810, the first feature operation module 820, the first feature mask module 830, the first dynamic feature fusion module 840, the first feature sampling module 850, the first feature cross-attention calculation module 860 and the first texture classification module 870 or the second image feature extraction module 910, the second feature operation module 920, the second feature mask module 930, the second dynamic feature fusion module 940, the second feature sampling module 950, the second feature cross-attention calculation module 960, the second texture classification module 970 and the parameter adjustment module 980 can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is executed.

[0184] It should be noted that the image internal texture classification device and the model training device part in the embodiments of the present disclosure correspond to the image internal texture classification method and the model training method part in the embodiments of the present disclosure. The description of the image internal texture classification device and the model training device part specifically refers to the image internal texture classification method and the model training method part, which will not be repeated here.

[0185] Fig.10 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present disclosure is schematically shown. Fig.10 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0186] like Fig.10 As shown, the electronic device 1000 according to an embodiment of the present disclosure includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 to a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include an onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0187] In RAM 1003, various programs and data required for the operation of electronic device 1000 are stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Processor 1001 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1002 and / or RAM 1003. It should be noted that the program can also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0188] According to an embodiment of the present disclosure, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to the bus 1004. The electronic device 1000 may further include one or more of the following components connected to the input / output (I / O) interface 1005: an input portion 1006 including a keyboard, a mouse, etc.; an output portion 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1008 including a hard disk, etc.; and a communication portion 1009 including a network interface card such as a LAN card, a modem, etc. The communication portion 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the input / output (I / O) interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage portion 1008 as needed.

[0189] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0190] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0191] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0192] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 1002 and / or the RAM 1003 described above and / or one or more memories other than the ROM 1002 and the RAM 1003 .

[0193] The embodiments of the present disclosure also include a computer program product, which includes a computer program, and the computer program contains program code for executing the method provided by the embodiments of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the method provided by the embodiments of the present disclosure.

[0194] When the computer program is executed by the processor 1001, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0195] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1009, and / or installed from the removable medium 1011. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0196] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0197] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0198] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for classifying internal texture of an image, characterized in that: The method comprises: Performing feature extraction on the target texture image to obtain N first multi-scale feature maps, wherein each of the first multi-scale feature maps has different semantic information; For an i-th first multi-scale feature map among the N first multi-scale feature maps, perform a dot multiplication operation on an initial mask image template and the i-th first multi-scale feature map to obtain an i-th target multi-scale feature map, thereby obtaining N target multi-scale feature maps; For the i-th target multi-scale feature map in the N target multi-scale feature maps, based on the attention mechanism, generate an i-th target mask image according to the i-th target multi-scale feature map to obtain N target mask images; Performing dynamic feature fusion on every two adjacent target multi-scale feature maps in the N target multi-scale feature maps to obtain N fused target multi-scale feature maps; Using the i-th target mask image in the N target mask images, spatially sampling the i-th fused target multi-scale feature map in the N fused target multi-scale feature maps to obtain the i-th target sampling multi-scale feature map, and obtaining N target sampling multi-scale feature maps; Using a densely connected attention mechanism, cross attention calculation is performed between the N target sampled multi-scale feature maps to obtain N second multi-scale feature maps; Using a classifier, the target texture image is texture classified according to the N second multi-scale feature maps to obtain a texture classification result of the target texture image, wherein N is an even number ≥ 2, and 0≤i≤N-1.

2. The method according to claim 1, characterized in that: The step of extracting features from the target texture image to obtain N first multi-scale feature maps includes: Inputting the target texture image into a convolutional neural network, respectively extracting a first intermediate multi-scale feature map output by each of the M convolutional modules in the convolutional neural network, to obtain M first intermediate multi-scale feature maps, where M is a positive integer ≥ 2; Based on the preset N target convolution modules, determining N second intermediate multi-scale feature maps from the M first intermediate multi-scale feature maps, wherein the N second intermediate multi-scale feature maps are respectively obtained by performing convolution operations on the N target convolution modules, wherein 2≤N≤M; For the i-th second intermediate multi-scale feature map among the N second intermediate multi-scale feature maps, a convolution operation is performed on the i-th second intermediate multi-scale feature map to obtain the i-th first multi-scale feature map, thereby obtaining N first multi-scale feature maps.

3. The method according to claim 1, characterized in that The method of generating an i-th target mask image based on the i-th target multi-scale feature map based on the attention mechanism to obtain N target mask images includes: Performing encoding processing on the i-th target multi-scale feature map to obtain an i-th importance weight matrix corresponding to the i-th target multi-scale feature map; According to the i-th target multi-scale feature map and the i-th importance weight matrix, an i-th target mask image is obtained, and N target mask images are obtained.

4. The method according to claim 3, characterized in that The i-th target multi-scale feature map and the i-th importance weight matrix each include C channels; Wherein, obtaining the i-th target mask image and obtaining N target mask images according to the i-th target multi-scale feature map and the importance weight matrix includes: For the cth channel among C channels, the i-th target mask image is obtained according to the feature matrix of the c-th channel of the i-th target multi-scale feature map and the feature matrix of the c-th channel of the i-th importance weight matrix, and N target mask images are obtained, where C is a positive integer ≥, 1≤c≤C.

5. The method according to claim 4, characterized in that The step of obtaining an i-th target mask image according to the feature matrix of the c-th channel of the i-th target multi-scale feature map and the feature matrix of the c-th channel of the i-th importance weight matrix, and obtaining N target mask images, comprises: Performing a dot multiplication operation on the feature matrix of the cth channel of the i-th target multi-scale feature map and the feature matrix of the cth channel of the i-th importance weight matrix to obtain a mask matrix of the cth channel, thereby obtaining a mask matrix of C channels; The mask matrices of the C channels are summed to obtain the ith target mask image, thereby obtaining N target mask images.

6. The method according to claim 1, characterized in that The method of performing dynamic feature fusion on every two adjacent target multi-scale feature maps in the N target multi-scale feature maps based on the multi-scale feature mixing module to obtain N fused target multi-scale feature maps includes: Using the k-th target multi-scale feature map, perform maximum feature mapping on the k-1-th target multi-scale feature map to obtain the k-1-th fused target multi-scale feature map; Using the k-1th fused target multi-scale feature map, perform maximum feature mapping on the k-2th target multi-scale feature map to obtain the k-2th fused target multi-scale feature map, and obtain N fused target multi-scale feature maps, where 0<k≤N-1, and when k is N-1, the N-1th fused target multi-scale feature map is the kth target multi-scale feature map.

7. The method according to claim 1, characterized in that The densely connected attention mechanism is used to perform cross attention calculation on the N target sampled multi-scale feature maps to obtain N second multi-scale feature maps, including: For an i-th target sampling multi-scale feature map of the N target sampling multi-scale feature maps, linearly project the i-th target sampling multi-scale feature map to obtain a query matrix, a key matrix, and a value matrix corresponding to the i-th target sampling multi-scale feature map; Based on the query matrix, the key matrix, the value matrix and the initial mask image template, obtaining an i-th intermediate second multi-scale feature map corresponding to the i-th target sampling multi-scale feature map, and obtaining N intermediate second multi-scale feature maps; For the i-th intermediate second multi-scale feature map of the N intermediate second multi-scale feature maps, dense connection attention calculation is performed on the first i-1 target sampling multi-scale feature maps and the i-th intermediate second multi-scale feature map to obtain the i-th second multi-scale feature map, and N second multi-scale feature maps are obtained, where, when i is 0, the i-th intermediate second multi-scale feature map is the i-th second multi-scale feature map, and 0≤i≤N-1.

8. The method according to claim 1, characterized in that: The classifiers include G; The step of performing texture classification on the target texture image according to the N second multi-scale feature maps to obtain a texture classification result of the target texture image includes: Based on a preset rule, determining two second multi-scale feature maps from the N second multi-scale feature maps; Using a g-th classifier among the G classifiers, and according to the two second multi-scale feature maps, performing texture classification on the target texture image, obtaining a texture classification result calculated by the g-th classifier, and obtaining texture classification results calculated by G classifiers, wherein 1≤G≤N / 2, 1≤g≤G; The texture classification result of the target texture image is obtained according to the texture classification results calculated by the G classifiers.

9. The method according to claim 1, characterized in that: The method further comprises: The target texture image is preprocessed by scale normalization.

10. A training method for an image internal texture classification model, characterized in that: The method comprises: Performing feature extraction on the target sample texture image to obtain N first sample multi-scale feature maps, wherein each of the first sample multi-scale feature maps has different semantic information; For an i-th first sample multi-scale feature map among the N first sample multi-scale feature maps, perform a dot multiplication operation on the initial sample mask image template and the i-th first sample multi-scale feature map to obtain an i-th target sample multi-scale feature map, thereby obtaining N target sample multi-scale feature maps; For the i-th target sample multi-scale feature map among the N target sample multi-scale feature maps, based on the attention mechanism, generate the i-th target sample mask image according to the i-th target sample multi-scale feature map to obtain N target sample mask images; Performing dynamic feature fusion on every two adjacent target sample multi-scale feature maps in the N target sample multi-scale feature maps to obtain N fused target sample multi-scale feature maps; Using the i-th target sample mask image among the N target sample mask images, spatially sampling the i-th fused target sample multi-scale feature map among the N fused target sample multi-scale feature maps to obtain the i-th target sample sampling multi-scale feature map, and obtaining N target sample sampling multi-scale feature maps; Using a densely connected attention mechanism, cross attention calculation is performed between the N target sample multi-scale feature maps to obtain N second sample multi-scale feature maps; Using a classifier, according to the N second sample multi-scale feature maps, the target sample texture image is texture classified to obtain a texture classification prediction result of the target sample texture image, wherein N is an even number ≥ 2, and 0≤i≤N-1; A loss value is obtained according to the texture classification prediction result and the texture classification true result, and the parameters of the image internal texture classification model are adjusted based on the loss value to obtain a trained image internal texture classification model.

Images