Image internal texture classification method and training method of texture classification model

Abstract

The disclosure provides an image internal texture classification method and a model training method, which are used in the fields of image processing and computer vision technology. The classification method comprises: performing feature extraction on a target texture image to obtain a first multi-scale feature map; performing point multiplication operation on an initial mask image template and the first multi-scale feature map to obtain a target multi-scale feature map; generating a target mask image according to the target multi-scale feature map; performing dynamic feature fusion on every two adjacent target multi-scale feature maps to obtain a fused target multi-scale feature map; performing spatial sampling on the fused target multi-scale feature map by using the target mask image to obtain a target sampled multi-scale feature map; performing cross-attention calculation on the target sampled multi-scale feature map to obtain a second multi-scale feature map; and performing texture classification on the second multi-scale feature map by using a classifier to obtain a texture classification result of the target texture image.

Description

Technical Field

[0001] This disclosure relates to the fields of image processing and computer vision technology, and specifically to an image internal texture classification method and a training method for a texture classification model. Background Technology

[0002] Texture classification is widely present on both natural and man-made physical surfaces and plays an important role in many applications, such as 3D modeling, medical image processing, materials science, and terrain recognition. Texture classification involves extracting spatial structure, patterns, and regularities from images to describe the properties of object surfaces.

[0003] Early texture classification methods relied on traditional image processing techniques and hand-designed feature extraction algorithms. These methods extracted texture features, such as gray-level co-occurrence matrices and local binary patterns, through statistical analysis of local regions of the image. However, these methods failed to fully capture global information in the image, resulting in poor classification performance in complex scenes.

[0004] With the development of deep learning technology, especially the introduction of convolutional neural networks and visual transformers, texture classification methods have made significant progress. However, when processing images of different resolutions, this method may suffer from poor texture classification performance due to information loss or changes in texture patterns. Summary of the Invention

[0005] In view of the above problems, this disclosure provides an image internal texture classification method and a training method for the texture classification model.

[0006] According to one aspect of this disclosure, an image internal texture classification method is provided, comprising: extracting features from a target texture image to obtain N first multi-scale feature maps, wherein each first multi-scale feature map has different semantic information; performing a dot product operation on an initial mask image template and the i-th first multi-scale feature map in the N first multi-scale feature maps to obtain an i-th target multi-scale feature map, thereby obtaining N target multi-scale feature maps; generating an i-th target mask image based on an attention mechanism in the i-th target multi-scale feature map in the N target multi-scale feature maps, thereby obtaining N target mask images; and performing a dot product operation on every two of the N target multi-scale feature maps. Adjacent target multi-scale feature maps are dynamically fused to obtain N fused target multi-scale feature maps. Using the i-th target mask image from the N target mask images, spatial sampling is performed on the i-th fused target multi-scale feature map among the N fused target multi-scale feature maps to obtain the i-th target sampled multi-scale feature map, resulting in N target sampled multi-scale feature maps. Using a densely connected attention mechanism, cross-attention calculation is performed among the N target sampled multi-scale feature maps to obtain N second multi-scale feature maps. Using a classifier, texture classification is performed on the target texture image based on the N second multi-scale feature maps to obtain the texture classification result of the target texture image, where N is an even number ≥ 2, and 0 ≤ i ≤ N-1.

[0007] Another aspect of this disclosure provides a training method for an image internal texture classification model, comprising: extracting features from a target sample texture image to obtain N first sample multi-scale feature maps, wherein each first sample multi-scale feature map has different semantic information; performing a dot product operation on an initial sample mask image template and the i-th first sample multi-scale feature map in the N first sample multi-scale feature maps to obtain an i-th target sample multi-scale feature map, thus obtaining N target sample multi-scale feature maps; generating an i-th target sample mask image based on an attention mechanism, thus obtaining N target sample mask images; and performing dynamic feature fusion on every two adjacent target sample multi-scale feature maps in the N target sample multi-scale feature maps to obtain N fused target samples. Multi-scale feature maps of samples are generated. Using the i-th target sample mask image from N target sample mask images, spatial sampling is performed on the i-th fused multi-scale feature map of the N fused target sample multi-scale feature maps to obtain the i-th target sample sample multi-scale feature map, resulting in N target sample sample multi-scale feature maps. A densely connected attention mechanism is used to perform cross-attention calculation among the N target sample sample multi-scale feature maps to obtain N second-sample multi-scale feature maps. A classifier is used to perform texture classification on the target sample texture image based on the N second-sample multi-scale feature maps, obtaining the texture classification prediction result of the target sample texture image, where N is an even number ≥ 2, and 0 ≤ i ≤ N-1. A loss value is obtained based on the texture classification prediction result and the true texture classification result, and the parameters of the image internal texture classification model are adjusted based on this loss value to obtain the trained image internal texture classification model.

[0008] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0009] A fourth aspect of this disclosure also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0010] The fifth aspect of this disclosure also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0011] According to embodiments of this disclosure, N first multi-scale feature maps are obtained by extracting features from the target texture image. Based on the N first multi-scale feature maps and an initial mask image template, N target multi-scale feature maps are obtained. Based on the N target multi-scale feature maps, N target mask images are generated respectively. Dynamic feature fusion is performed on the N target multi-scale feature maps to obtain N fused target multi-scale feature maps. Based on the N target mask images, spatial sampling is performed on the N fused target multi-scale feature maps, and cross-attention calculation is performed on the pronunciation of the obtained N target sampled multi-scale features to obtain N second multi-scale feature maps. Using a classifier, texture classification is performed on the target texture image based on the N second multi-scale feature maps to obtain the classification result of the target texture image. By employing techniques such as dynamic feature fusion of N multi-scale feature maps, masking of multi-scale feature maps, and spatial sampling, texture classification of target texture images is achieved. This at least partially solves the technical problem in existing methods where information loss and texture pattern changes may occur during image scaling, thereby reducing the accuracy of texture classification results. As a result, it achieves the technical effect of effectively fusing feature maps of different scales while preserving the original resolution, thereby reducing information loss and texture pattern changes and improving the accuracy of texture classification results. Attached Figure Description

[0012] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0013] Figure 1 The illustration schematically shows an exemplary system architecture to which image internal texture classification and training methods can be applied according to embodiments of the present disclosure;

[0014] Figure 2 A flowchart illustrating an image internal texture classification method according to an embodiment of the present disclosure is shown schematically.

[0015] Figure 3 A schematic diagram illustrating the determination of a target mask image according to an embodiment of the present disclosure is shown.

[0016] Figure 4 A flowchart illustrating a method for obtaining a fused multi-scale feature map of a target according to an embodiment of the present disclosure is shown schematically.

[0017] Figure 5 The illustration shows a schematic diagram of obtaining N fused target multi-scale feature maps according to an embodiment of the present disclosure;

[0018] Figure 6 A flowchart illustrating a training method for an image internal texture classification model according to an embodiment of the present disclosure is shown schematically.

[0019] Figure 7 A schematic diagram of an image internal texture classification method according to an embodiment of the present disclosure is shown;

[0020] Figure 8 A block diagram of an image internal texture classification apparatus according to an embodiment of the present disclosure is shown schematically;

[0021] Figure 9 A block diagram illustrating a training apparatus for an image internal texture classification model according to an embodiment of the present disclosure is shown schematically; and

[0022] Figure 10 A block diagram of an electronic device suitable for implementing an image internal texture classification method and a model training method according to embodiments of the present disclosure is shown schematically. Detailed Implementation

[0023] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0026] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0027] In the embodiments disclosed herein, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security and network security.

[0028] In the embodiments disclosed herein, user authorization or consent is obtained before acquiring or collecting user personal information.

[0029] Embodiments of this disclosure provide a method for classifying internal textures in an image, comprising: extracting features from a target texture image to obtain N first multi-scale feature maps, wherein each first multi-scale feature map has different semantic information; performing a dot product operation on an initial mask image template and the i-th first multi-scale feature map in the N first multi-scale feature maps to obtain an i-th target multi-scale feature map, thereby obtaining N target multi-scale feature maps; generating an i-th target mask image based on an attention mechanism in the i-th target multi-scale feature map in the N target multi-scale feature maps, thereby obtaining N target mask images; and processing every two adjacent elements in the N target multi-scale feature maps. Dynamic feature fusion is performed on the target multi-scale feature maps to obtain N fused target multi-scale feature maps. Using the i-th target mask image among the N target mask images, spatial sampling is performed on the i-th fused target multi-scale feature map among the N fused target multi-scale feature maps to obtain the i-th target sampled multi-scale feature map, resulting in N target sampled multi-scale feature maps. Using a dense connection attention mechanism, cross-attention calculation is performed among the N target sampled multi-scale feature maps to obtain N second multi-scale feature maps. Using a classifier, texture classification is performed on the target texture image based on the N second multi-scale feature maps to obtain the texture classification result of the target texture image, where N is an even number ≥ 2, and 0 ≤ i ≤ N-1.

[0030] Figure 1 This illustration schematically depicts an exemplary system architecture to which image internal texture classification and training methods can be applied according to embodiments of this disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0031] like Figure 1As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0032] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social media platform software, etc. (for example only).

[0033] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0034] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0035] It should be noted that the image internal texture classification method and model training method provided in this disclosure embodiment can generally be executed by server 105. Correspondingly, the image internal texture classification device and model training device provided in this disclosure embodiment can generally be located in server 105. The image internal texture classification method and model training method provided in this disclosure embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the image internal texture classification device and model training device provided in this disclosure embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Alternatively, the image internal texture classification method and model training method provided in this disclosure embodiment can also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the image internal texture classification device and model training device provided in the embodiments of this disclosure may also be disposed in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or disposed in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0036] For example, the target texture image may originally be stored in any one of the first terminal device 101, the second terminal device 102, or the third terminal device 103 (e.g., the first terminal device 101, but not limited thereto), or it may be stored on an external storage device and imported into the first terminal device 101. Then, the first terminal device 101 may locally execute the image internal texture classification method provided in the embodiments of this disclosure, or send the target texture image to other terminal devices, servers, or server clusters, and have the other terminal devices, servers, or server clusters that receive the target texture image execute the image internal texture classification method provided in the embodiments of this disclosure.

[0037] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0038] Figure 2 A flowchart illustrating an image internal texture classification method according to an embodiment of the present disclosure is shown schematically.

[0039] like Figure 2 As shown, the method includes operations S201 to S207.

[0040] In operation S201, feature extraction is performed on the target texture image to obtain N first multi-scale feature maps, where each first multi-scale feature map has different semantic information.

[0041] According to embodiments of this disclosure, the first multi-scale feature map can be obtained by feature extraction from the target texture image using a convolutional neural network. The convolutional neural network serves as the backbone network for feature extraction.

[0042] According to embodiments of this disclosure, the N first multi-scale feature maps are feature maps of different scales extracted by N convolutional modules of a convolutional neural network. Each first multi-scale feature map has different semantic information. For example, the multi-scale feature maps extracted by the convolutional modules in the first few layers represent the shallow features of the target texture image; the multi-scale feature maps extracted by the convolutional modules in the later layers represent the deep features of the target texture image. Shallow features can be fine-grained information of the target texture image, such as color, texture, and edges; deep features can be coarse-grained information of the target texture image, that is, more abstract semantic information.

[0043] In operation S202, for the i-th first multi-scale feature map among the N first multi-scale feature maps, a dot product operation is performed on the initial mask image template and the i-th first multi-scale feature map to obtain the i-th target multi-scale feature map, thus obtaining the N target multi-scale feature maps.

[0044] According to embodiments of this disclosure, the initial mask image template is obtained by masking a preprocessed target texture image to mask the original region in the preprocessed target texture image, wherein the original region represents the region where the original image information of the target texture image is located in the initial mask image.

[0045] According to embodiments of this disclosure, for the i-th first multi-scale feature map among N first multi-scale feature maps, a dot product operation can be performed between the initial mask image and the i-th first multi-scale feature map to obtain the i-th target multi-scale feature map. Based on the above method, N target multi-scale feature maps can be obtained.

[0046] In operation S203, for the i-th target multi-scale feature map in the N target multi-scale feature maps, based on the attention mechanism, the i-th target mask image is generated, resulting in N target mask images.

[0047] According to embodiments of this disclosure, an attention mask generation module can be used to determine a learnable weight matrix based on the i-th target multi-scale feature map, and based on the calculation of the learnable weight matrix and the feature matrix of the i-th target multi-scale feature map, a mask image of the i-th target can be generated, thereby generating N target mask images.

[0048] According to embodiments of this disclosure, during the generation of the target mask image, unimportant feature regions can be masked, retaining only the feature regions that make significant contributions to the texture classification task, so as to guide subsequent spatial sampling operations and reduce the impact of irrelevant feature regions on classification.

[0049] In operation S204, dynamic feature fusion is performed on every two adjacent target multi-scale feature maps in the N target multi-scale feature maps to obtain N fused target multi-scale feature maps.

[0050] According to embodiments of this disclosure, a multi-scale feature fusion module can be used to dynamically fuse every two adjacent target multi-scale feature maps in N target multi-scale feature maps based on the maximum feature mapping algorithm, so as to obtain a fused target multi-scale feature map for each target multi-scale feature map.

[0051] According to embodiments of this disclosure, the maximum value of deep features in a multi-scale feature map of a target with deep features can be mapped to an adjacent multi-scale feature map of the target, so as to extract global texture feature information in the target texture image.

[0052] In operation S205, using the i-th target mask image from N target mask images, spatial sampling is performed on the i-th fused target multi-scale feature map in the N fused target multi-scale feature maps to obtain the i-th target sampled multi-scale feature map, thus obtaining the N target sampled multi-scale feature maps.

[0053] According to embodiments of this disclosure, spatial sampling can be performed on N fused target multi-scale feature maps using the obtained N target mask images. Spatial sampling is based on the features of important regions retained in the target mask images. Corresponding important new regions are extracted from the fused target multi-scale feature maps to obtain the sampled multi-scale feature maps, i.e., the target sampled multi-scale feature maps.

[0054] According to embodiments of this disclosure, the dimensions of the i-th fused target multi-scale feature map can be adjusted, and then the i-th fused target scale feature map can be sampled for a fixed length. Based on the i-th target mask image, the region to be retained marked in the target mask image is sampled from the i-th fused target multi-scale feature map to obtain the i-th target sampled multi-scale feature map, thereby obtaining N target sampled multi-scale feature maps.

[0055] According to embodiments of this disclosure, the target sampling multi-scale feature map retains important regional features in the texture classification task of the target texture image, reducing the computational load for subsequent texture classification tasks.

[0056] In operation S206, a densely connected attention mechanism is used to perform cross-attention calculations between N target multi-scale feature maps to obtain N second multi-scale feature maps.

[0057] In operation S207, a classifier is used to classify the texture based on N second multi-scale feature maps to obtain the texture classification result of the target texture image, where N is an even number ≥ 2 and 0 ≤ i ≤ N-1.

[0058] According to embodiments of this disclosure, the target sampling multi-scale feature map can be calculated using a mask self-attention mechanism to obtain the corresponding self-attention mask image; then, the target sampling multi-scale feature map corresponding to the shallow features can be sequentially connected with the self-attention mask image using a cross-attention mechanism to calculate N second multi-scale feature maps.

[0059] According to embodiments of this disclosure, the introduction of a densely connected attention mechanism can correlate shallow and deep features in multi-scale feature maps of N targets, reflecting the correlation between features of different scales among the multi-scale feature maps of N targets; it can also fuse features of different scales, thereby enhancing the understanding of texture feature information in the global context.

[0060] According to embodiments of this disclosure, a classifier can be used to calculate the texture probability of a target texture image based on N second multi-scale feature maps, thereby obtaining the texture classification result of the target texture image.

[0061] According to embodiments of this disclosure, N first multi-scale feature maps are obtained by extracting features from the target texture image. Based on the N first multi-scale feature maps and an initial mask image template, N target multi-scale feature maps are obtained. Based on the N target multi-scale feature maps, N target mask images are generated respectively. Dynamic feature fusion is performed on the N target multi-scale feature maps to obtain N fused target multi-scale feature maps. Based on the N target mask images, spatial sampling is performed on the N fused target multi-scale feature maps, and cross-attention calculation is performed on the pronunciation of the obtained N target sampled multi-scale features to obtain N second multi-scale feature maps. Using a classifier, texture classification is performed on the target texture image based on the N second multi-scale feature maps to obtain the classification result of the target texture image. By employing techniques such as dynamic feature fusion of N multi-scale feature maps, masking of multi-scale feature maps, and spatial sampling, texture classification of target texture images is achieved. This at least partially solves the technical problem in existing methods where information loss and texture pattern changes may occur during image scaling, thereby reducing the accuracy of texture classification results. As a result, it achieves the technical effect of effectively fusing feature maps of different scales while preserving the original resolution, thereby reducing information loss and texture pattern changes and improving the accuracy of texture classification results.

[0062] According to embodiments of this disclosure, feature extraction is performed on the target texture image to obtain N first multi-scale feature maps, including:

[0063] The target texture image is input into a convolutional neural network, and the first intermediate multi-scale feature map output by each of the M convolutional modules in the convolutional neural network is extracted to obtain M first intermediate multi-scale feature maps, where M is a positive integer ≥2;

[0064] Based on N pre-set target convolutional modules, N second intermediate multi-scale feature maps are determined from M first intermediate multi-scale feature maps. The N second intermediate convolutional modules are output by performing convolution operations on the N target convolutional modules respectively, where 2≤N≤M. For the i-th second intermediate multi-scale feature map among the N second intermediate multi-scale feature maps, a convolution operation is performed on the i-th second intermediate multi-scale feature map to obtain the i-th first multi-scale feature map, resulting in N first multi-scale feature maps.

[0065] According to embodiments of this disclosure, a convolutional neural network may include M convolutional modules, and the first intermediate multi-scale feature map may be a feature map obtained after convolution processing by each convolutional module.

[0066] According to embodiments of this disclosure, the target texture image is input into the 0th convolutional module of the convolutional network (denoted as...). By performing convolution processing, the 0th first intermediate multi-scale feature map corresponding to the 0th convolutional module can be obtained; then the 0th first intermediate multi-scale feature map is input into the 1st convolutional module of the convolutional neural network (denoted as ). The first convolutional module performs convolutional processing to obtain the first intermediate multi-scale feature map corresponding to the first convolutional module. This first intermediate multi-scale feature map is then input into the second convolutional module of the convolutional neural network (denoted as...). The convolution process is performed to obtain the second first intermediate scale feature map corresponding to the second convolution module, ..., and the (m-1)th first intermediate multi-scale feature map is input into the m-th convolution module of the convolutional neural network (denoted as ...). The convolution process is performed to obtain the m-th first intermediate multi-scale feature map corresponding to the m-th convolutional module, ..., and the (M-2)-th first intermediate multi-scale feature map is input into the (M-1)-th convolutional module of the convolutional neural network (denoted as ...). The convolution process is performed to obtain the (M-1)th first intermediate multi-scale feature map corresponding to the (M-1)th convolution module, thus obtaining M first intermediate multi-scale feature maps. Where 0≤m≤M-1.

[0067] According to embodiments of this disclosure, the convolutional processing of the 0th to M-1th convolutional modules of a convolutional neural network can be a feature extraction process from shallow features to deep features of a target texture image. Each of the M first intermediate multi-scale feature maps has different semantic information.

[0068] According to embodiments of this disclosure, the N target convolutional modules can be determined from the M convolutional modules included in the convolutional neural network. Based on these N convolutional modules, N first intermediate multi-scale feature maps corresponding to the N convolutional modules can be determined from the M first intermediate multi-scale feature maps as second intermediate multi-scale feature maps.

[0069] According to embodiments of this disclosure, for example, a convolutional neural network includes five convolutional modules, from the 0th to the 4th. The target texture image is input into the convolutional neural network, and the first intermediate multi-scale feature map output from each of the five convolutional modules is extracted, resulting in five first intermediate multi-scale feature maps. The 1st to 4th convolutional modules are then identified as the target convolutional modules (i.e., ...). ~ If the first intermediate multi-scale feature map corresponding to the first convolutional module is used as the first intermediate multi-scale feature map, then the fourth intermediate multi-scale feature map corresponding to the fourth convolutional module will be used as the four second intermediate multi-scale feature maps.

[0070] According to embodiments of this disclosure, for the i-th second intermediate multi-scale feature map among N second intermediate multi-scale feature maps, a convolution operation can be performed on the i-th second intermediate multi-scale feature map to obtain the i-th first multi-scale feature map. Based on the above method, N first multi-scale feature maps can be obtained.

[0071] According to embodiments of this disclosure, for example, each of the four second intermediate multi-scale feature maps obtained above can be convolved to obtain a first multi-scale feature map for each second intermediate multi-scale feature map, that is, four first multi-scale feature maps are obtained.

[0072] According to embodiments of this disclosure, convolution operations are performed on N second intermediate multi-scale feature maps to ensure that the resulting N first multi-scale feature maps have the same dimensionality.

[0073] According to an embodiment of this disclosure, based on an attention mechanism, an i-th target mask image is generated based on the i-th target multi-scale feature map to obtain N target mask images, including: encoding the i-th target multi-scale feature map to obtain the i-th importance weight matrix corresponding to the i-th target multi-scale feature map; obtaining the i-th target mask image based on the i-th target multi-scale feature map and the i-th importance weight matrix to obtain N target mask images.

[0074] According to embodiments of this disclosure, an autoencoder can be used to encode the i-th target multi-scale feature map, extract the most representative features, and then decode these extracted most representative features. In this process, the autoencoder can generate an i-th importance weight matrix corresponding to the i-th target multi-scale feature map.

[0075] According to embodiments of this disclosure, the dimension of the importance weight matrix is ​​the same as the dimension of the target multi-scale feature map, and is used to measure the importance of each feature region in the target multi-scale map.

[0076] According to embodiments of this disclosure, the i-th importance weight matrix and the i-th target multi-scale feature map can be used for weighted calculation to obtain the i-th target mask image. Based on the above method, N target mask images can be obtained.

[0077] According to embodiments of this disclosure, both the i-th target multi-scale feature map and the i-th importance weight matrix include C channels.

[0078] According to embodiments of this disclosure, based on the multi-scale feature map of the i-th target and the importance weight matrix, the i-th target mask image is obtained, resulting in N target mask images, including:

[0079] For the c-th channel among the C channels, based on the feature matrix of the c-th channel of the i-th target multi-scale feature map and the feature matrix of the c-th channel of the i-th importance weight matrix, the i-th target mask image is obtained, resulting in N target mask images, where C is a positive integer ≥ 1, and 1 ≤ c ≤ C.

[0080] According to an embodiment of this disclosure, obtaining an i-th target mask image and N target mask images based on the feature matrix of the c-th channel of the i-th target multi-scale feature map and the feature matrix of the c-th channel of the i-th importance weight matrix includes: performing a dot product operation on the feature matrix of the c-th channel of the i-th target multi-scale feature map and the feature matrix of the c-th channel of the i-th importance weight matrix to obtain a mask matrix of the c-th channel, and obtaining a mask matrix of C channels; performing a summation operation on the mask matrices of the C channels to obtain the i-th target mask image and N target mask images.

[0081] According to embodiments of this disclosure, the i-th target multi-scale feature map may contain C channels, and the i-th importance weight matrix may also contain C channels. The feature matrix of each channel in the i-th target multi-scale feature map, and the feature matrix of each channel in the i-th importance weight matrix can be obtained.

[0082] According to an embodiment of this disclosure, for the c-th channel among C channels, the feature matrix of the c-th channel in the i-th target multi-scale feature map can be multiplied by the feature matrix of the c-th channel in the i-th importance weight matrix to obtain the mask matrix of the c-th channel.

[0083] According to embodiments of this disclosure, the mask matrix for each channel can be calculated using the above method, thereby obtaining a mask matrix for C channels.

[0084] According to embodiments of this disclosure, the obtained mask matrices of C channels can be summed and normalized using the softmax activation function to obtain the i-th target intermediate mask image. The i-th target intermediate mask image can then be binarized to obtain the i-th target mask image.

[0085] According to embodiments of this disclosure, the above method can be used to calculate the target mask image corresponding to the multi-scale feature map of each target, thereby obtaining N target mask images.

[0086] According to embodiments of this disclosure, the target mask image This can be obtained through equations (1-1) to (1-2), that is:

[0087] (1-1);

[0088] , (1-2);

[0089] Where f(h) is the threshold function, for The h-th element in the output target intermediate mask image matrix is ​​the eigenvalue, 1≤h≤H, where H is the number of all elements and is a positive integer ≥1; C is a positive integer ≥1, c∈{1,2,…,C}, E(·) represents the mean operation, and μ is... The average of the eigenvalues ​​of all elements in the output target intermediate mask image matrix.

[0090] According to embodiments of this disclosure, for example, the activation function in equation (1-1) above can be utilized. The feature matrix corresponding to the intermediate mask image of the i-th target, corresponding to the multi-scale feature map of the i-th target, is obtained. This feature matrix contains H elements. For the eigenvalue of the h-th element among the H elements, the eigenvalue f(h) of the h-th element is compared with the average value μ. If the eigenvalue f(h) of the h-th element is greater than or equal to the average value μ, then the h-th element is binarized to 1; if the eigenvalue f(h) of the h-th element is less than the average value μ, then the h-th element is binarized to 0. In this way, the H elements in the feature matrix corresponding to the intermediate mask image of the i-th target can be binarized to obtain the mask image of the i-th target. Using the above method, N target mask images corresponding to N multi-scale feature maps of N targets can be obtained respectively.

[0091] We can sequentially calculate the target mask corresponding to the j-th element in the feature matrix of the i-th target multi-scale feature map, that is, when calculating... When the value of is greater than or equal to the value of μ, The value is 1, meaning the mask value corresponding to the j-th element of the i-th target mask image is 1; when calculating When the value is less than the value of μ, The value is 0, which means that the mask value corresponding to the target mask of the j-th element of the i-th target mask image is 0; calculate the target mask corresponding to the J elements in sequence to obtain the i-th target mask image corresponding to the i-th target multi-scale feature image, and obtain N target mask images.

[0092] According to embodiments of this disclosure, N target mask images can be obtained based on the above method. This disclosure obtains target mask images by performing masking processing on multi-scale feature images of the targets, thereby including important feature regions in the multi-scale feature images of the targets and reducing the interference of irrelevant regions on texture classification tasks.

[0093] Figure 3 A schematic diagram illustrating the determination of a target mask image according to an embodiment of the present disclosure is shown.

[0094] like Figure 3As shown in the schematic diagram 300, the target multi-scale feature map can be input into the autoencoder for encoding to obtain the importance weight matrix corresponding to the feature matrix of the target multi-scale feature map. Based on the feature value of each channel in the target multi-scale feature map and the feature value of each channel in the importance weight matrix, a dot product operation is performed. The result of the dot product operation of each channel value is summed and normalized to obtain the target mask image corresponding to the target multi-scale feature map.

[0095] Figure 4 A flowchart illustrating a method for obtaining a fused multi-scale feature map of a target according to an embodiment of the present disclosure is shown.

[0096] like Figure 4 As shown, the method 400 may include operations S401 to S402.

[0097] In operation S401, the k-th target multi-scale feature map is used to perform maximum feature mapping on the (k-1)-th target multi-scale feature map to obtain the (k-1)-th fused target multi-scale feature map.

[0098] In operation S402, using the (k-1)th fused target multi-scale feature map, the (k-2)th target multi-scale feature map is subjected to maximum feature mapping to obtain the (k-2)th fused target multi-scale feature map, resulting in N fused target multi-scale feature maps, where 0 < k ≤ N-1. When k is N-1, the (N-1)th fused target multi-scale feature map is the kth target multi-scale feature map.

[0099] According to embodiments of this disclosure, maximum feature mapping can be achieved by comparing the size of feature values ​​at the same position in two adjacent target multi-scale feature maps, mapping the larger feature value in the target multi-scale feature map representing deep features to the same position in the other target multi-scale feature map, thus covering the smaller value in the other target multi-scale feature map.

[0100] According to embodiments of this disclosure, layer normalization can be performed on N target multi-scale feature maps respectively to smooth the distribution of the target multi-scale feature maps.

[0101] According to embodiments of this disclosure, for example, for the k-th target multi-scale feature map and the (k-1)-th target multi-scale feature map, the k-th target multi-scale feature map represents a feature map of deep features relative to the (k-1)-th target multi-scale feature map. The eigenvalues ​​of the elements at the same position in the feature matrix of the k-th target multi-scale feature map and the (k-1)-th target multi-scale feature map can be compared. If the eigenvalue of the feature matrix of the k-th target multi-scale feature map is larger, the smaller eigenvalue at that position in the feature matrix of the (k-1)-th target multi-scale feature map is replaced with the larger eigenvalue of the feature matrix of the k-th target multi-scale feature map; if the eigenvalue of the feature matrix of the (k-1)-th target multi-scale feature map is larger, the larger eigenvalue of the feature matrix of the (k-1)-th target multi-scale feature map is retained, thereby obtaining the (k-1)-th fused target multi-scale feature map.

[0102] According to an embodiment of this disclosure, the eigenvalues ​​of the elements at the same position in the feature matrix of the (k-1)th fused target multi-scale feature map are compared with those of the (k-2)th target multi-scale feature map. If the eigenvalue of the feature matrix of the (k-1)th fused target multi-scale feature map is larger, the smaller eigenvalue at that position in the feature matrix of the (k-2)th target multi-scale feature map is replaced with the larger eigenvalue of the feature matrix of the (k-1)th fused target multi-scale feature map. If the eigenvalue of the feature matrix of the (k-2)th target multi-scale feature map is larger, the larger eigenvalue is retained, thereby obtaining the (k-2)th fused target multi-scale feature map.

[0103] According to embodiments of this disclosure, based on the above method, the eigenvalues ​​of elements at the same position in the feature matrix of the first fused target multi-scale feature map and the feature matrix of the 0th target multi-scale feature map can be compared. If the eigenvalue of the feature matrix of the first fused target multi-scale feature map is larger, the smaller eigenvalue at that position in the feature matrix of the 0th target multi-scale feature map is replaced with the larger eigenvalue of the feature matrix of the first fused target multi-scale feature map. If the eigenvalue of the feature matrix of the 0th target multi-scale feature map is larger, the larger eigenvalue of the feature matrix of the 0th target multi-scale feature map is retained, thereby obtaining the 0th fused target multi-scale feature map.

[0104] According to an embodiment of this disclosure, when k is N-1, the (N-1)th fused target multi-scale feature map is the kth target multi-scale feature map. Based on the above-obtained 0th fused target multi-scale feature map to the N-th fused target multi-scale feature map, N fused target multi-scale feature maps are obtained.

[0105] Figure 5The illustration shows a schematic diagram of obtaining N fused target multi-scale feature maps according to an embodiment of the present disclosure.

[0106] like Figure 5 As shown in schematic diagram 500, for example, the target multi-scale feature map, from representing deep features to representing shallow features, can be represented in matrix form as follows: , … , … , After performing layer normalization on the above target feature maps respectively, the maximum feature mapping is used to... and Perform maximum feature mapping, and

[0107] Compare the feature values ​​at the same position in the middle, if The eigenvalue at that position is relatively large,

[0108] Replace the smaller feature value at that position with Larger eigenvalues ​​in; if If the eigenvalue at that position is large, then retain it. The feature value at that position is used to sequentially achieve the following: The maximum feature mapping yields the (N-2)th fused target multi-scale feature map. At this point, the (N-1)th fused target multi-scale feature map for itself.

[0109] Based on the above method, the target multi-scale feature map after fusion for the (N-2)th time is then... and Perform maximum feature mapping to obtain the (N-3)th fused target multi-scale feature map. And so on, the k-th fused target multi-scale feature map and Perform maximum feature mapping to obtain the (k-1)th fused target multi-scale feature map. ..., the first fused multi-scale feature map of the target and Perform maximum feature mapping to obtain the target multi-scale feature map after fusion at the 0th fusion level. Thus, N fused target multi-scale feature maps are obtained.

[0110] According to embodiments of this disclosure, a densely connected attention mechanism is used to perform cross-attention calculations among N target sampled multi-scale feature maps to obtain N second multi-scale feature maps, including:

[0111] For the i-th target sampled multi-scale feature map among N target sampled multi-scale feature maps, perform linear projection on the i-th target sampled multi-scale feature map to obtain the query matrix, key matrix, and value matrix corresponding to the i-th target sampled multi-scale feature map; based on the query matrix, key matrix, value matrix, and initial mask image template, obtain the i-th intermediate second multi-scale feature map corresponding to the i-th target sampled multi-scale feature map, and obtain N intermediate second multi-scale feature maps; for the i-th intermediate second multi-scale feature map among the N intermediate second multi-scale feature maps, perform dense connection attention calculation on the first i-1 target sampled multi-scale feature maps and the i-th intermediate second multi-scale feature map to obtain the i-th second multi-scale feature map, and obtain N second multi-scale feature maps, where, when i is 0, the i-th intermediate second multi-scale feature map is the i-th second multi-scale feature map, 0≤i≤N-1.

[0112] According to embodiments of this disclosure, for the i-th target sampled multi-scale feature map among N target sampled multi-scale feature maps, a query matrix for the i-th target sampled multi-scale feature map can be obtained by performing a linear projection on the i-th target sampled multi-scale feature map. Key matrix Sum matrix .

[0113] According to embodiments of this disclosure, the feature dimension of the i-th target sampling multi-scale feature map is determined. The query matrix can be based on the obtained multi-scale feature map of the i-th target. Key matrix Value matrix Feature Dimension And the initial mask image template, to obtain the i-th intermediate second multi-scale feature map. Based on the above method, N intermediate second multi-scale feature maps can be obtained.

[0114] According to embodiments of this disclosure, the i-th intermediate second multi-scale feature map It can be obtained through the following formula (2), that is:

[0115] (2);

[0116] in, Key matrix Transpose of; This is the initial mask image template.

[0117] According to embodiments of this disclosure, N target sampling multi-scale feature maps can be represented as follows: , … …、 , The corresponding N intermediate second-scale feature maps generated can be represented as follows: , … … , .

[0118] According to embodiments of this disclosure, for the i-th intermediate second multi-scale feature map among N intermediate second multi-scale feature maps, for example... This can be achieved by utilizing densely connected attention computation to sample multi-scale feature maps of the first i-1 targets (i.e., , … ) and the i-th intermediate second multi-scale feature map By performing cross-attention calculations on densely connected features, we can obtain the second multi-scale feature map of the i-th intermediate feature map. The corresponding second multi-scale feature map .

[0119] According to embodiments of this disclosure, for example, and By performing dense connection attention calculations, we can obtain the same result as... The corresponding second multi-scale feature map ; , and By performing dense connection attention calculations, we can obtain the same result as... The corresponding second multi-scale feature map And so on, we can , … …、 and By performing dense connection attention calculations, we can obtain the same result as... The corresponding second multi-scale feature map This yields N second-scale feature maps.

[0120] According to embodiments of this disclosure, when i is 0, that is, for the intermediate second multi-scale feature map... For it, dense connection attention calculation is not required; the 0th second multi-scale feature map It is the second multi-scale feature map in the middle. itself.

[0121] According to embodiments of this disclosure, the second multi-scale feature map It can be calculated using equation (3), that is:

[0122] (3);

[0123] in, Represented as a query matrix; It is represented as a key matrix and a value matrix.

[0124] According to embodiments of this disclosure, N second multi-scale feature maps can be obtained based on the above formula (3).

[0125] According to embodiments of this disclosure, the classifier comprises G classes; wherein, based on N second multi-scale feature maps, texture classification is performed on the target texture image to obtain the texture classification result of the target texture image, including:

[0126] Based on preset rules, two second multi-scale feature maps are determined from N second multi-scale feature maps; using the g-th classifier among G classifiers, texture classification is performed on the target texture image based on the two second multi-scale feature maps, and the texture classification result calculated by the g-th classifier is obtained, and the texture classification result calculated by G classifiers is obtained, where 1≤G≤N / 2, 1≤g≤G; based on the texture classification result calculated by G classifiers, the texture classification result of the target texture image is obtained.

[0127] According to embodiments of this disclosure, two second multi-scale feature maps can be determined from N second multi-scale feature maps based on a preset rule of Ni-1. For example, if the i-th feature map is determined from N second multi-scale feature maps, then another second multi-scale feature map is determined according to the rule of Ni-1.

[0128] According to embodiments of this disclosure, for the i-th second multi-scale feature map among N second multi-scale feature maps, global feature information in the i-th second multi-scale feature map can be extracted to obtain the global feature information of the i-th second multi-scale feature map. Based on the above method, global feature information of N second multi-scale feature maps is obtained.

[0129] According to embodiments of this disclosure, for example, when i is 0, based on the aforementioned preset rules, the first classifier among the G classifiers can be used, based on the global feature information of the 0th second multi-scale feature map. Global feature information of the N-0-1th second multi-scale feature map By calculating the texture types appearing within the target texture image, the texture classification result of the target texture image can be obtained. ;

[0130] According to embodiments of this disclosure, when i is 1, the second classifier out of G classifiers can be used based on the global feature information of the first second multi-scale feature map. Global feature information of the N-1-1th second multi-scale feature map By calculating the texture types appearing within the target texture image, the texture classification result of the target texture image can be obtained. .

[0131] According to embodiments of this disclosure, based on the above method, the g-th classifier out of G classifiers can be used to obtain global feature information from the i-th second multi-scale feature map. Global feature information of the Ni-1th second multi-scale feature map By calculating the texture types appearing within the target texture image, the texture classification result of the target texture image can be obtained. .

[0132] According to embodiments of this disclosure, the texture classification result of the target texture image It can be calculated using equation (4), that is:

[0133] (4);

[0134] Where 1≤g≤N / 2; classifier is a linear classifier; CLS is global feature information.

[0135] According to embodiments of this disclosure, based on the above method, texture classification results calculated by G classifiers can be obtained. Each texture classification result can include probability values ​​for various texture types within the predicted target image. For example, each texture classification result includes probability values ​​for three types: A, B, and C. The probability values ​​for each of the three types A, B, and C in the G texture classification results can be weighted and averaged to obtain the weighted average probability values ​​for each of the three types A, B, and C. The type corresponding to the highest probability value is taken as the texture classification result of the target texture image.

[0136] According to embodiments of this disclosure, the method further includes: performing scale normalization preprocessing on the target texture image.

[0137] According to embodiments of this disclosure, since the resolution of the target texture image may vary significantly in different scenes, simple image scaling may lead to the loss of important texture information. Therefore, before inputting the target texture image into a convolutional neural network for feature extraction, it is necessary to perform scale normalization preprocessing on the target texture image.

[0138] According to embodiments of this disclosure, target texture images can be scaled by padding with "0" pixels. Specifically, for each input target texture image, the image size can be standardized by padding its lower right corner with "0" pixels.

[0139] According to embodiments of this disclosure, the initial mask image It is also obtained by padding with "0". It has the same size as the target texture image obtained after padding with "0". This initial mask image is used to mark which pixel areas are padded with "0", thereby avoiding these padded "0" areas from interfering with the texture classification task.

[0140] According to embodiments of this disclosure, padding the target texture image with "0"s not only preserves the resolution information of the original target texture image, but also provides a uniform size for subsequent feature extraction and attention calculation.

[0141] Figure 6 A flowchart illustrating a training method for an image internal texture classification model according to an embodiment of the present disclosure is shown.

[0142] like Figure 6 As shown, the method 600 may include operations S601 to S608.

[0143] In operation S601, feature extraction is performed on the texture image of the target sample to obtain N first sample multi-scale feature maps, where each first sample multi-scale feature map has different semantic information.

[0144] In operation S602, for the i-th first sample multi-scale feature map of N first sample multi-scale feature maps, a dot product operation is performed on the initial sample mask image template and the i-th first sample multi-scale feature map to obtain the i-th target sample multi-scale feature map, and thus obtain N target sample multi-scale feature maps.

[0145] In operation S603, for the i-th multi-scale feature map of N target samples, based on the attention mechanism, the i-th target sample mask image is generated according to the i-th multi-scale feature map of the i-th target sample, thus obtaining N target sample mask images.

[0146] In operation S604, dynamic feature fusion is performed on every two adjacent multi-scale feature maps of N target samples to obtain N fused multi-scale feature maps of target samples.

[0147] In operation S605, using the i-th target sample mask image from N target sample mask images, spatial sampling is performed on the i-th fused target sample multi-scale feature map in the N fused target sample multi-scale feature map to obtain the i-th target sample sampling multi-scale feature map, thus obtaining the N target sample sampling multi-scale feature map.

[0148] In the S606 operation, a densely connected attention mechanism is used to perform cross-attention calculations between the multi-scale feature maps of N target samples to obtain N second sample multi-scale feature maps.

[0149] In operation S607, a classifier is used to perform texture classification on the target sample texture image based on N second sample multi-scale feature maps, and the texture classification prediction result of the target sample texture image is obtained, where N is an even number ≥ 2 and 0 ≤ i ≤ N-1.

[0150] In operation S608, the loss value is obtained based on the texture classification prediction result and the texture classification true result. The parameters of the image internal texture classification model are adjusted based on the loss value to obtain the trained image internal texture classification model.

[0151] According to the embodiments of this disclosure, the specific descriptions of operations S601 to S607 in training the image internal texture classification model correspond to the descriptions of operations S201 to S207 in the above-described image internal texture classification method, and the description of the training method can be referred to the description of the image internal texture classification method.

[0152] According to the embodiments of this disclosure, by operating S601~S607, a texture classification prediction result can be obtained. Then, based on the loss function, a loss value is obtained according to the texture classification prediction result and the actual texture result. Based on the loss value, the parameters of the image internal texture analysis model are adjusted, and then the next round of training is performed to obtain the corresponding loss value for the next round. Training is stopped when the preset conditions are met, and the trained image internal texture analysis model is obtained.

[0153] According to embodiments of this disclosure, the preset condition can be a preset loss threshold, and training stops when the loss value meets the preset loss threshold; the preset condition can also be the number of training iterations, and training stops when the number of training iterations meets the preset number of iterations.

[0154] According to embodiments of this disclosure, the loss function can be the cross-entropy loss function L, and the loss value can be obtained based on the following equation (5), namely:

[0155] (5);

[0156] in, Let T be the predicted label for the r-th texture classification; T is the true label for the texture classification. The cross-entropy loss is used to classify the r-th texture and predict its label and the true label.

[0157] According to the embodiments of this disclosure, the training loss value of the image internal texture classification model can be obtained by the above formula (5). Based on the loss value, the model parameters are adjusted. When the preset conditions are met, the training is stopped, and the trained image internal texture classification model is obtained.

[0158] Figure 7 A schematic diagram of an image internal texture classification method according to an embodiment of the present disclosure is shown.

[0159] like Figure 7 As shown in the schematic diagram 700, the target texture image is preprocessed by padding with "0"s to obtain a preprocessed target texture image 701; simultaneously, an initial mask image template 702 is generated. The preprocessed target texture image 701 is input into a backbone neural network 703 with M convolutional modules to obtain M first intermediate multi-scale feature maps 704; based on pre-set N target convolutional modules, N second intermediate multi-scale feature maps 705 are determined from the M first intermediate multi-scale feature maps 704; convolution operations are performed on the N second intermediate multi-scale feature maps 705 respectively to obtain N first multi-scale feature maps 706; dot multiplication operations are performed on the N first multi-scale feature maps 706 and the initial mask image template 702 respectively to obtain N target multi-scale feature maps 707; and a mask is generated based on the N target multi-scale feature maps 707. N target mask images 708 are generated; dynamic feature fusion is performed on the N target multi-scale feature maps 707 respectively to obtain N fused target multi-scale feature maps 709; spatial sampling is performed on the N fused target multi-scale feature maps 709 using the N target mask images 708 to obtain N target sampled multi-scale feature maps 710; based on the dense connection attention mechanism, dense connection cross-attention calculation is performed on the N target sampled multi-scale feature maps 710 to obtain N second multi-scale feature maps 711; using a classifier, the texture classification result 712 of the target texture image is obtained based on the N second multi-scale feature maps 711.

[0160] According to an embodiment of the present invention, Table 1 is a quantitative comparison table of the classification results of the method provided by the present invention with DeepNet, MAPNet, DSRNet, CLASSNet, DFAEN and RADAM methods on four different datasets. The classification accuracy is expressed as a percentage, as shown in Table 1:

[0161] Table 1

[0162]

[0163] As shown in Table 1, the method provided by this invention outperforms existing methods on all four datasets. Compared to existing methods, the accuracy of this invention is significantly improved, especially on datasets 1 and 2, where the accuracy reaches 97.1% and 95.3%, respectively. Furthermore, regardless of whether the RestNet18 network or the ConvNeXt in ImageNet-21K network is used, the accuracy of this invention is higher than that of other methods. In other words, the method provided by this invention outperforms other methods even when using different backbone networks.

[0164] Figure 8 A block diagram of an image internal texture classification apparatus according to an embodiment of the present disclosure is shown schematically.

[0165] like Figure 8 As shown, the device 800 includes: a first image feature extraction module 810, a first feature operation module 820, a first feature mask module 830, a first dynamic feature fusion module 840, a first feature sampling module 850, a first feature cross-attention calculation module 860, and a first texture classification module 870.

[0166] The first image feature extraction module 810 is used to extract features from the target texture image to obtain N first multi-scale feature maps, wherein each first multi-scale feature map has different semantic information.

[0167] The first feature operation module 820 is used to perform a dot product operation on the initial mask image template and the i-th first multi-scale feature map for the i-th first multi-scale feature map in N first multi-scale feature maps, so as to obtain the i-th target multi-scale feature map and thus obtain N target multi-scale feature maps.

[0168] The first feature mask module 830 is used to generate the i-th target mask image based on the attention mechanism for the i-th target multi-scale feature map in N target multi-scale feature maps, thereby obtaining N target mask images.

[0169] The first dynamic feature fusion module 840 is used to perform dynamic feature fusion on every two adjacent target multi-scale feature maps in N target multi-scale feature maps to obtain N fused target multi-scale feature maps.

[0170] The first feature sampling module 850 is used to spatially sample the i-th fused target multi-scale feature map from the N fused target multi-scale feature maps using the i-th target mask image from the N target mask images, to obtain the i-th target sampled multi-scale feature map, and thus obtain the N target sampled multi-scale feature maps.

[0171] The first feature cross-attention calculation module 860 is used to perform cross-attention calculation between N target multi-scale feature maps using a densely connected attention mechanism to obtain N second multi-scale feature maps.

[0172] The first texture classification module 870 is used to classify the target texture image using a classifier pair based on N second multi-scale feature maps, and obtain the texture classification result of the target texture image, where N is an even number ≥ 2 and 0 ≤ i ≤ N-1.

[0173] Figure 9 A block diagram of a training apparatus for an image internal texture classification model according to an embodiment of the present disclosure is shown schematically.

[0174] like Figure 9 As shown, the device 900 includes: a second image feature extraction module 910, a second feature operation module 920, a second feature mask module 930, a second dynamic feature fusion module 940, a second feature sampling module 950, a second feature cross-attention calculation module 960, a second texture classification module 970, and a parameter adjustment module 980.

[0175] The second image feature extraction module 910 is used to extract features from the texture image of the target sample to obtain N first sample multi-scale feature maps, wherein each first sample multi-scale feature map has different semantic information.

[0176] The second feature operation module 920 is used to perform a dot product operation on the initial sample mask image template and the i-th first sample multi-scale feature map for the i-th first sample multi-scale feature map in the N first sample multi-scale feature maps, so as to obtain the i-th target sample multi-scale feature map and thus obtain the N target sample multi-scale feature maps.

[0177] The second feature mask module 930 is used to generate the i-th target sample mask image based on the attention mechanism for the i-th target sample multi-scale feature map in the N target sample multi-scale feature maps, thereby obtaining N target sample mask images.

[0178] The second dynamic feature fusion module 940 is used to dynamically fuse the multi-scale feature maps of each pair of adjacent target samples in the multi-scale feature maps of N target samples to obtain N fused multi-scale feature maps of target samples.

[0179] The second feature sampling module 950 is used to spatially sample the i-th fused target sample multi-scale feature map in the N fused target sample multi-scale feature maps using the i-th target sample mask image in the N target sample mask images, so as to obtain the i-th target sample sampling multi-scale feature map and thus obtain the N target sample sampling multi-scale feature maps.

[0180] The second feature cross-attention calculation module 960 is used to perform cross-attention calculation between the multi-scale feature maps of N target samples using a densely connected attention mechanism to obtain N second sample multi-scale feature maps.

[0181] The second texture classification module 970 is used to classify the texture image of the target sample based on N second sample multi-scale feature maps using a classifier, and obtain the texture classification prediction result of the target sample texture image, where N is an even number ≥ 2 and 0 ≤ i ≤ N-1.

[0182] The parameter adjustment module 980 is used to obtain a loss value based on the texture classification prediction result and the texture classification real result, and to adjust the parameters of the image internal texture classification model based on the loss value to obtain the trained image internal texture classification model.

[0183] According to embodiments of this disclosure, any multiple of the following modules can be implemented in one module / unit / subunit: a first image feature extraction module 810, a first feature operation module 820, a first feature mask module 830, a first dynamic feature fusion module 840, a first feature sampling module 850, a first feature cross-attention calculation module 860, and a first texture classification module 870; or any one of the following modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least some of the functions of one or more of these modules / units / subunits can be combined with at least some of the functions of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the following components—the first image feature extraction module 810, the first feature operation module 820, the first feature mask module 830, the first dynamic feature fusion module 840, the first feature sampling module 850, the first feature cross-attention calculation module 860, and the first texture classification module 870—or the second image feature extraction module 910, the second feature operation module 920, the second feature mask module 930, the second dynamic feature fusion module 940, the second feature sampling module 950, the second feature cross-attention calculation module 960, the second texture classification module 970, and the parameter adjustment module 980—can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable method of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the following modules can be implemented, either the first image feature extraction module 810, the first feature operation module 820, the first feature mask module 830, the first dynamic feature fusion module 840, the first feature sampling module 850, the first feature cross-attention calculation module 860, and the first texture classification module 870, or the second image feature extraction module 910, the second feature operation module 920, the second feature mask module 930, the second dynamic feature fusion module 940, the second feature sampling module 950, the second feature cross-attention calculation module 960, the second texture classification module 970, and the parameter adjustment module 980, as at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0184] It should be noted that the image internal texture classification device and model training device in the embodiments of this disclosure correspond to the image internal texture classification method and model training method in the embodiments of this disclosure. For a detailed description of the image internal texture classification device and model training device, please refer to the image internal texture classification method and model training method section, which will not be repeated here.

[0185] Figure 10 A block diagram of an electronic device suitable for implementing the methods described above, according to embodiments of the present disclosure, is illustrated schematically. Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0186] like Figure 10 As shown, an electronic device 1000 according to an embodiment of the present disclosure includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage portion 1008 into a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0187] RAM 1003 stores various programs and data required for the operation of electronic device 1000. Processor 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Processor 1001 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 1002 and / or RAM 1003. It should be noted that programs may also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0188] According to embodiments of this disclosure, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to a bus 1004. The electronic device 1000 may also include one or more of the following components connected to the input / output (I / O) interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the input / output (I / O) interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1010 as needed so that computer programs read from it can be installed into the storage section 1008 as needed.

[0189] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by processor 1001, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0190] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0191] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0192] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 1002 and / or RAM 1003 described above and / or one or more memories other than ROM 1002 and RAM 1003.

[0193] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the methods provided in the embodiments of this disclosure.

[0194] When the computer program is executed by the processor 1001, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0195] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1009, and / or installed from a removable medium 1011. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0196] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0197] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present disclosure can be combined and / or combined in various ways, even if such combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0198] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for classifying the internal texture of an image, characterized in that, The method includes: Feature extraction is performed on the target texture image to obtain N first multi-scale feature maps, where each first multi-scale feature map has different semantic information; For the i-th first multi-scale feature map among the N first multi-scale feature maps, perform a dot product operation on the initial mask image template and the i-th first multi-scale feature map to obtain the i-th target multi-scale feature map, and thus obtain N target multi-scale feature maps. For the i-th target multi-scale feature map among the N target multi-scale feature maps, based on the attention mechanism, the i-th target mask image is generated according to the i-th target multi-scale feature map, resulting in N target mask images; Dynamic feature fusion is performed on every two adjacent target multi-scale feature maps in the N target multi-scale feature maps to obtain N fused target multi-scale feature maps, including: Using the k-th target multi-scale feature map, perform maximum feature mapping on the (k-1)-th target multi-scale feature map to obtain the (k-1)-th fused target multi-scale feature map; Using the (k-1)th fused target multi-scale feature map, the (k-2)th target multi-scale feature map is subjected to maximum feature mapping to obtain the (k-2)th fused target multi-scale feature map, resulting in N fused target multi-scale feature maps, where 0 < k ≤ N-1. When k is N-1, the (N-1)th fused target multi-scale feature map is the kth target multi-scale feature map. The maximum feature mapping compares the feature values ​​at the same position in two adjacent target multi-scale feature maps, maps the larger feature value in the target multi-scale feature map representing deep features to the same position in the other target multi-scale feature map, and covers the smaller value in the other target multi-scale feature map. Using the i-th target mask image among the N target mask images, spatial sampling is performed on the i-th fused target multi-scale feature map among the N fused target multi-scale feature maps to obtain the i-th target sampled multi-scale feature map, thus obtaining the N target sampled multi-scale feature maps. By utilizing a densely connected attention mechanism, cross-attention calculation is performed among the N target multi-scale feature maps to obtain N second multi-scale feature maps. Using a classifier, the target texture image is classified according to the N second multi-scale feature maps to obtain the texture classification result of the target texture image, where N is an even number ≥ 2 and 0 ≤ i ≤ N-1.

2. The method according to claim 1, characterized in that, The feature extraction of the target texture image yields N first multi-scale feature maps, including: The target texture image is input into a convolutional neural network, and the first intermediate multi-scale feature map output by each of the M convolutional modules in the convolutional neural network is extracted to obtain M first intermediate multi-scale feature maps, where M is a positive integer ≥2; Based on N pre-set target convolutional modules, N second intermediate multi-scale feature maps are determined from the M first intermediate multi-scale feature maps. The N second intermediate multi-scale feature maps are obtained by performing convolution operations on the N target convolutional modules respectively, where 2≤N≤M. For the i-th second intermediate multi-scale feature map among the N second intermediate multi-scale feature maps, perform a convolution operation on the i-th second intermediate multi-scale feature map to obtain the i-th first multi-scale feature map, and obtain N first multi-scale feature maps.

3. The method according to claim 1, characterized in that, The attention-based mechanism generates an i-th target mask image based on the i-th target multi-scale feature map, resulting in N target mask images, including: The i-th target multi-scale feature map is encoded to obtain the i-th importance weight matrix corresponding to the i-th target multi-scale feature map; Based on the i-th target multi-scale feature map and the i-th importance weight matrix, the i-th target mask image is obtained, and N target mask images are obtained.

4. The method according to claim 3, characterized in that, The i-th target multi-scale feature map and the i-th importance weight matrix both include C channels; The step of obtaining the i-th target mask image based on the i-th target multi-scale feature map and the importance weight matrix, and obtaining N target mask images, includes: For the c-th channel among the C channels, based on the feature matrix of the c-th channel of the i-th target multi-scale feature map and the feature matrix of the c-th channel of the i-th importance weight matrix, the i-th target mask image is obtained, resulting in N target mask images, where C is a positive integer ≥ 1, and 1 ≤ c ≤ C.

5. The method according to claim 4, characterized in that, The process of obtaining the i-th target mask image based on the feature matrix of the c-th channel of the i-th target multi-scale feature map and the feature matrix of the c-th channel of the i-th importance weight matrix, and obtaining N target mask images, includes: Perform a dot product operation on the feature matrix of the c-th channel of the i-th target multi-scale feature map and the feature matrix of the c-th channel of the i-th importance weight matrix to obtain the mask matrix of the c-th channel, and thus obtain the mask matrix of C channels. The mask matrices of the C channels are summed to obtain the i-th target mask image, and then N target mask images are obtained.

6. The method according to claim 1, characterized in that, The densely connected attention mechanism performs cross-attention calculations on the N target sampled multi-scale feature maps to obtain N second multi-scale feature maps, including: For the i-th target sampling multi-scale feature map among the N target sampling multi-scale feature maps, perform linear projection on the i-th target sampling multi-scale feature map to obtain the query matrix, key matrix, and value matrix corresponding to the i-th target sampling multi-scale feature map; Based on the query matrix, the key matrix, the value matrix, and the initial mask image template, the i-th intermediate second multi-scale feature map corresponding to the i-th target sampling multi-scale feature map is obtained, and N intermediate second multi-scale feature maps are obtained. For the i-th intermediate second multi-scale feature map among the N intermediate second multi-scale feature maps, dense connection attention is calculated between the first i-1 target sampling multi-scale feature maps and the i-th intermediate second multi-scale feature map to obtain the i-th second multi-scale feature map, resulting in N second multi-scale feature maps. Where, when i is 0, the i-th intermediate second multi-scale feature map is the i-th second multi-scale feature map, and 0≤i≤N-1.

7. The method according to claim 1, characterized in that, The classifiers include G types; Specifically, based on the N second multi-scale feature maps, texture classification is performed on the target texture image to obtain the texture classification result of the target texture image, including: Based on preset rules, two second multi-scale feature maps are determined from the N second multi-scale feature maps; Using the g-th classifier among the G classifiers, the target texture image is classified according to the two second multi-scale feature maps to obtain the texture classification result calculated by the g-th classifier, and the texture classification result calculated by the G classifiers is obtained, where 1≤G≤N / 2, 1≤g≤G; The texture classification result of the target texture image is obtained based on the texture classification results calculated by the G classifiers.

8. The method according to claim 1, characterized in that, The method further includes: The target texture image is preprocessed using scale normalization.

9. A training method for an image internal texture classification model, characterized in that, The method includes: Feature extraction is performed on the texture image of the target sample to obtain N first sample multi-scale feature maps, wherein each first sample multi-scale feature map has different semantic information; For the i-th first sample multi-scale feature map in the N first sample multi-scale feature maps, perform a dot product operation on the initial sample mask image template and the i-th first sample multi-scale feature map to obtain the i-th target sample multi-scale feature map, and obtain N target sample multi-scale feature maps. For the i-th target sample multi-scale feature map in the N target sample multi-scale feature maps, based on the attention mechanism, the i-th target sample mask image is generated according to the i-th target sample multi-scale feature map, thus obtaining N target sample mask images; Dynamic feature fusion is performed on every two adjacent multi-scale feature maps of the N target samples to obtain N fused multi-scale feature maps of the target samples. This includes: using the k-th multi-scale feature map of the target sample to perform maximum feature mapping on the (k-1)-th multi-scale feature map of the target sample to obtain the (k-1)-th fused multi-scale feature map of the target sample; and using the (k-1)-th fused multi-scale feature map of the target sample to perform maximum feature mapping on the (k-2)-th multi-scale feature map of the target sample to obtain the (k-2)-th fused multi-scale feature map of the target sample. Figure shows that N fused target sample multi-scale feature maps are obtained, where 0 < k ≤ N-1. When k is N-1, the (N-1)th fused target sample multi-scale feature map is the kth target sample multi-scale feature map. The maximum feature mapping is to compare the feature values ​​at the same position in two adjacent target sample multi-scale feature maps, and map the larger feature value in the target sample multi-scale feature map representing deep features to the same position in another target sample multi-scale feature map, covering the smaller value in the other target sample multi-scale feature map. Using the i-th target sample mask image from the N target sample mask images, spatial sampling is performed on the i-th fused target sample multi-scale feature map in the N fused target sample multi-scale feature maps to obtain the i-th target sample sampling multi-scale feature map, thus obtaining the N target sample sampling multi-scale feature maps. Using a densely connected attention mechanism, cross-attention calculation is performed between the multi-scale feature maps of the N target samples to obtain N second sample multi-scale feature maps. Using a classifier, based on the N second sample multi-scale feature maps, the texture image of the target sample is classified to obtain the texture classification prediction result of the texture image of the target sample, where N is an even number ≥ 2, and 0 ≤ i ≤ N-1; The loss value is obtained based on the texture classification prediction result and the texture classification true result. The parameters of the image internal texture classification model are adjusted based on the loss value to obtain the trained image internal texture classification model.

Images