Visual image classification method and device, equipment, storage medium and program product

Abstract

The invention relates to a visual image classification method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring a to-be-processed image, and inputting the to-be-processed image into a pre-trained visual image classification model; segmenting the to-be-processed image through the visual image classification model to obtain a plurality of image blocks and respective position codes of the plurality of image blocks; for each image block, performing original feature extraction and saliency feature extraction on the image block, splicing the original feature of the image block and the corresponding saliency feature, and performing vectorization processing on the spliced feature of the image block to obtain a feature vector of the image block; and performing coding processing and classification processing according to the respective feature vectors and the respective position codes of the plurality of image blocks to obtain a visual image classification result. By adopting the method, the visual image classification accuracy can be improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a visual image classification method, apparatus, device, storage medium, and program product. Background Technology

[0002] Traditional visual image classification typically relies on low-level visual features such as color, texture, edges, and contrast; however, it may fail to capture more complex salient features. The challenge of salient feature detection lies in accurately mimicking human visual attention mechanisms and handling noise and complex backgrounds in images. Furthermore, differences in attention mechanisms among individuals further complicate the detection process. Therefore, traditional methods suffer from poor accuracy in visual image classification. Summary of the Invention

[0003] Therefore, it is necessary to provide a visual image classification method, apparatus, computer device, computer-readable storage medium, and computer program product to address the aforementioned technical problems and improve the accuracy of visual image classification.

[0004] In a first aspect, this application provides a visual image classification method, including:

[0005] Acquire the image to be processed and input it into a pre-trained visual image classification model;

[0006] The image to be processed is segmented using a visual image classification model to obtain multiple image patches and their respective position codes. For each image patch, original features and salient features are extracted. The original features and corresponding salient features of the image patch are concatenated, and the concatenated features are vectorized to obtain the feature vector of the image patch. Based on the feature vectors and position codes of the multiple image patches, encoding and classification processes are performed to obtain the visual image classification result.

[0007] In one embodiment, the visual image classification model includes a salient feature extraction module; salient feature extraction of image patches includes:

[0008] The original features of the image patch are processed by the saliency feature extraction module to perform channel attention and spatial attention processing, resulting in a spatial attention weight map.

[0009] Spatial attention weight maps are used as saliency features for image patches.

[0010] In one embodiment, the original features of the image patch are processed by a saliency feature extraction module through channel attention and spatial attention processing to obtain a spatial attention weight map, including:

[0011] The original features of the image patch are processed by the saliency feature extraction module to obtain a channel attention weight map. The original features of the image patch are then weighted according to the channel attention weight map, and spatial attention is applied to the weighted features to obtain a spatial attention weight map.

[0012] In one embodiment, spatial attention processing is applied to the weighted features to obtain a spatial attention weight map, including:

[0013] The weighted features are then subjected to average pooling and max pooling respectively.

[0014] The average pooling result and the max pooling result are concatenated;

[0015] The splicing result is activated to obtain a spatial attention weight map.

[0016] In one embodiment, the original features of the image patch are processed by a saliency feature extraction module to obtain a channel attention weight map, including:

[0017] The original features of the image patch are processed by average pooling and max pooling through the saliency feature extraction module.

[0018] The average pooling and max pooling results are processed by multi-layer perception to obtain the channel attention weight map.

[0019] In one embodiment, the visual image classification model includes an encoder and a classification head; encoding and classification are performed based on the feature vectors and positional codes of multiple image patches to obtain a visual image classification result, including:

[0020] The encoder sums the feature vectors and position codes of multiple image patches to obtain the summation results of multiple image patches. Attention processing is then applied to the summation results of multiple image patches to obtain the attention weights of each image patch. The feature vectors of each image patch are then weighted according to the attention weights.

[0021] The weighted processing results are classified using a classification head to obtain the visual image classification result.

[0022] Secondly, this application also provides a visual image classification device, comprising:

[0023] The acquisition module is used to acquire the image to be processed and input the image to be processed into a pre-trained visual image classification model.

[0024] The processing module is used to segment the image to be processed using a visual image classification model, obtaining multiple image blocks and their respective position codes. For each image block, the module performs original feature extraction and salient feature extraction, concatenates the original features of the image block with the corresponding salient features, and vectorizes the concatenated features to obtain the feature vector of the image block. Based on the feature vectors and position codes of the multiple image blocks, the module performs encoding and classification processing to obtain the visual image classification result.

[0025] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0026] Acquire the image to be processed and input it into a pre-trained visual image classification model;

[0027] The image to be processed is segmented using a visual image classification model to obtain multiple image patches and their respective position codes. For each image patch, original features and salient features are extracted. The original features and corresponding salient features of the image patch are concatenated, and the concatenated features are vectorized to obtain the feature vector of the image patch. Based on the feature vectors and position codes of the multiple image patches, encoding and classification processes are performed to obtain the visual image classification result.

[0028] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0029] Acquire the image to be processed and input it into a pre-trained visual image classification model;

[0030] The image to be processed is segmented using a visual image classification model to obtain multiple image patches and their respective position codes. For each image patch, original features and salient features are extracted. The original features and corresponding salient features of the image patch are concatenated, and the concatenated features are vectorized to obtain the feature vector of the image patch. Based on the feature vectors and position codes of the multiple image patches, encoding and classification processes are performed to obtain the visual image classification result.

[0031] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0032] Acquire the image to be processed and input it into a pre-trained visual image classification model;

[0033] The image to be processed is segmented using a visual image classification model to obtain multiple image patches and their respective position codes. For each image patch, original features and salient features are extracted. The original features and corresponding salient features of the image patch are concatenated, and the concatenated features are vectorized to obtain the feature vector of the image patch. Based on the feature vectors and position codes of the multiple image patches, encoding and classification processes are performed to obtain the visual image classification result.

[0034] The aforementioned visual image classification methods, devices, computer equipment, computer-readable storage media, and computer program products classify images to be processed using a pre-trained visual image classification model. This model can extract original features and salient features from each image block of the image to be processed. The concatenation of the original and salient features allows the model to prioritize salient visual target regions in the image, improving the model's visual image classification ability and significantly reducing background noise interference with detection. In the model's encoding stage, positional encoding of each image block is introduced, compensating for the lack of spatial information in the feature vectors, enabling the model to accurately understand the spatial layout of the target and improving the accuracy of visual image classification. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a diagram illustrating the application environment of a visual image classification method in one embodiment.

[0037] Figure 2 This is a flowchart illustrating a visual image classification method in one embodiment;

[0038] Figure 3 This is a schematic diagram of the composition of a visual image classification model in one embodiment;

[0039] Figure 4 This is a schematic diagram of the overall process of a visual image classification method in one embodiment;

[0040] Figure 5 This is a structural block diagram of a visual image classification device in one embodiment;

[0041] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0043] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0044] The visual image classification method provided in this application can be applied to, for example... Figure 1 The application environment shown is illustrated. Terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. This embodiment uses the method applied to a terminal as an example; it is understood that this method can also be applied to a server, and to a system including both a terminal and a server, and implemented through the interaction between the terminal and the server. Terminal 102 acquires the image to be processed and inputs it into a pre-trained visual image classification model. The visual image classification model segments the image to be processed, obtaining multiple image blocks and their respective position codes. For each image block, original feature extraction and salient feature extraction are performed. The original features of the image block are concatenated with the corresponding salient features, and the concatenated features are vectorized to obtain the feature vector of the image block. Encoding and classification processing are performed based on the feature vectors and position codes of the multiple image blocks to obtain the visual image classification result. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, and projection equipment. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0045] In one exemplary embodiment, such as Figure 2 As shown, a visual image classification method is provided, which can be applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps 202 to 204. Wherein:

[0046] Step 202: Obtain the image to be processed and input it into the pre-trained visual image classification model.

[0047] The images to be processed are the raw image data for the visual image classification task. They can be obtained from real-time camera acquisition, local image files, or image streams transmitted over the network. The size and resolution can be flexibly adjusted according to the task requirements.

[0048] The visual image classification model is a deep learning model trained on a large-scale dataset, used to classify input images. For example, the key components of a visual image classification model include an image patch embedding module, a location encoding module, a raw feature extraction module, a salient feature extraction module, a fusion module, convolutional layers, an encoder, and a classification head.

[0049] Step 204: The image to be processed is segmented using a visual image classification model to obtain multiple image blocks and their respective position codes. For each image block, original features and salient features are extracted. The original features and corresponding salient features of the image block are concatenated. The concatenated features of the image block are vectorized to obtain the feature vector of the image block. Encoding and classification processing are performed based on the feature vectors and position codes of the multiple image blocks to obtain the visual image classification result.

[0050] Among them, such as Figure 3 The diagram illustrates the composition of a visual image classification model in some embodiments. After the image to be processed is input into the visual image classification model, it is segmented into multiple image patches using an image patch embedding module. Each image patch retains local visual information (such as edges, textures, and local target contours) of the corresponding region of the image to be processed. For example, a fixed-size patch segmentation method is used, dividing the image to be processed into 4×4 patches. If the image size is not divisible by the patch size, edge padding can be used to fill in the gaps, and overlapping areas between image patches can be preserved to prevent feature loss due to target segmentation. Converting large-size images into a sequence of small-size image patches reduces the computational complexity of single-step feature extraction while allowing the model to focus on detailed features of local regions, facilitating subsequent patch-by-patch feature processing and global correlation modeling.

[0051] The location encoding module encodes the positions of multiple image patches within the image to be processed, resulting in location codes. A location code is a vector representing the spatial position of an image patch within the image to be processed, with dimensions identical to the feature vector of the image patch. Location encoding preserves spatial location information within the image.

[0052] The original features of each image patch are extracted using the original feature extraction module. For example, the original features are the RGB three-channel features of the image patch, which can be transformed into a high-dimensional feature map through multiple convolution and pooling operations.

[0053] The salient features of each image patch are extracted using a salient feature extraction module. Salient features are regions or targets in an image patch that automatically attract human visual attention; they are prominent parts in visual perception. For example, brightly colored objects, moving objects, and areas with strong contrast to their surroundings.

[0054] The fusion module fuses the original features of each image patch with the corresponding saliency features along the channel dimension to obtain the stitched features of the corresponding image patch.

[0055] Convolutional layers vectorize the stitched features of each image patch to obtain the corresponding feature vector. For example, convolutional layers perform convolution on the stitched features to obtain local features of the image patch, and downsampling of these local features yields the feature vector of the image patch. Downsampling can be achieved using global average pooling (GAP) or global max pooling (GMP). The feature vector of the image patch integrates original visual information and saliency information.

[0056] The encoder performs encoding processing based on the feature vectors and positional codes of multiple image patches to obtain the encoding result. For example, the encoder uses a multi-head attention mechanism to fuse the feature vectors of each image patch with information from other image patches in the sequence, and combines this with positional codes to form an enhanced feature vector with global correlation as the encoding result.

[0057] The encoding results are classified using a classification head to obtain the visual image classification result. For example, the classification head uses a fully connected layer to determine the target category of an image patch. The visual image classification result can include the visual target category and a confidence score.

[0058] In some embodiments, the training steps of the visual image classification model include: acquiring image samples and visual category labels for the image samples, and inputting the image samples into an initial model; segmenting the image samples using the initial model to obtain multiple image patch samples and their respective location encoding samples; for each image patch sample, performing original feature extraction and salient feature extraction, concatenating the original feature samples of the image patch sample with the corresponding salient feature samples, and vectorizing the concatenated feature samples of the image patch sample to obtain feature vector samples of the image patch sample; performing encoding and classification processing based on the feature vector samples and their respective location encoding samples of the multiple image patch samples to obtain visual category prediction results; calculating the model loss based on the visual category prediction results, visual category labels, and loss function, adjusting the parameters of the initial model using the model loss, and stopping when a preset stopping condition is reached to obtain the visual image classification model.

[0059] In the aforementioned visual image classification method, a pre-trained visual image classification model is used to classify the image to be processed. The visual image classification model can extract the original features and salient features for each image block of the image to be processed. After concatenating the original features and salient features, the model can prioritize the visual target regions in the image, improve the model's visual image classification ability, and significantly reduce the interference of background noise on detection. In the model's encoding stage, positional encoding of each image block is introduced to make up for the lack of spatial information in the feature vector, allowing the model to accurately understand the spatial layout of the target and improve the accuracy of visual image classification.

[0060] In an exemplary embodiment, the visual image classification model includes a salient feature extraction module; salient feature extraction of an image patch includes: performing channel attention processing and spatial attention processing on the original features of the image patch through the salient feature extraction module to obtain a spatial attention weight map; and using the spatial attention weight map as the salient feature of the image patch.

[0061] The salient feature extraction module incorporates a two-layer processing logic of channel attention and spatial attention. First, it filters key semantic channels through channel attention, and then it locates salient spatial regions within the channels through spatial attention, ultimately outputting pixel-level salient features.

[0062] Channel attention processing refers to the attention weighting operation performed on the channel dimension of the original features of an image patch. It is used to select channels in the original features that are valuable for visual image classification, filter redundant channel information, and lay a high-quality channel feature foundation for subsequent spatial attention processing.

[0063] Spatial attention processing refers to the attention weighting operation performed on the spatial dimension based on the features after channel attention weighting. It is used to locate salient spatial regions (i.e., the pixel positions of visual targets) within image blocks, weaken the feature responses of background regions, and directly output salient features that can characterize the position of visual targets.

[0064] In this embodiment, key semantic channels representing target edges and contours are first filtered out from the original features through channel attention processing, while redundant background channels are filtered out. Then, salient spatial regions within the channels are located through spatial attention processing. The generated spatial attention weight map can intuitively identify the target location within the image patch, allowing the model to prioritize the target region and significantly reduce the interference of background noise on detection. Especially in complex scenes, it can effectively reduce missed detections and false detections, and improve the detection accuracy of small targets and occluded targets.

[0065] In an exemplary embodiment, the original features of an image patch are processed by a saliency feature extraction module to perform channel attention processing and spatial attention processing to obtain a spatial attention weight map. This includes: performing channel attention processing on the original features of the image patch through the saliency feature extraction module to obtain a channel attention weight map; performing weighted processing on the original features of the image patch based on the channel attention weight map; and performing spatial attention processing on the weighted features to obtain a spatial attention weight map.

[0066] The channel attention processing mainly involves performing global average pooling and global max pooling on the original features in the spatial dimension to obtain the global statistical features of each channel; then, these two types of pooled features are input into a shared multilayer perceptron (MLP) to learn the dependencies between channels; finally, a channel attention weight map is generated through sigmoid activation.

[0067] Channel attention weight maps are used to weight each channel of the original feature, strengthening key semantic channels and suppressing ineffective channels.

[0068] Spatial attention processing mainly involves performing average pooling and max pooling on the channel-weighted features along the channel dimension to obtain two single-channel spatial feature maps. These two maps are then concatenated and input into a convolutional layer to learn the dependencies between spatial regions. Finally, a pixel-level spatial attention weight map is generated through Sigmoid activation, which can locate salient spatial regions within an image patch (i.e., the pixel location of the target) and weaken the feature response of the background region, thereby achieving pixel-level target region focusing.

[0069] In this embodiment, redundant channels are first filtered through channel attention, and then the salient regions of the key channels are located in the spatial dimension. This enables the model to accurately capture the core features of the target and significantly reduce the interference of background noise on detection. Especially in complex scenarios with cluttered backgrounds, small targets, or target occlusion, it can effectively reduce missed detections and false detections. Weighted processing strengthens the features of the key channels and improves the semantic recognition of the features.

[0070] In an exemplary embodiment, spatial attention processing is performed on the weighted features to obtain a spatial attention weight map, including: performing average pooling and max pooling on the weighted features respectively; concatenating the average pooling result and the max pooling result; and performing activation processing on the concatenated result to obtain the spatial attention weight map.

[0071] The salient feature extraction module performs a global averaging operation on the weighted features along the channel dimension. Specifically, for each spatial location, it calculates the pixel mean across all channels as the average pooling result. The average pooling result is a single-channel feature map that smoothly preserves the global channel statistics of the spatial location, reflecting the average semantic response of that location across all channels.

[0072] The salient feature extraction module performs a global max operation on the weighted features along the channel dimension. That is, for each spatial location, it calculates the maximum pixel value across all channels as the result of max pooling. The max pooling result is a single-channel feature map that captures the channel extrema information at a spatial location, highlighting the most representative semantic features at that location.

[0073] The saliency feature extraction module merges the results of average pooling and max pooling along the channel dimension, resulting in a concatenated dual-channel feature map. A non-linear transformation is then applied to the concatenated result, mapping the values ​​to the 0-1 range. The resulting image serves as the spatial attention weight map. For example, the Sigmoid activation function can be used for this non-linear transformation.

[0074] In this embodiment, average pooling preserves the global channel statistics of spatial location, while max pooling captures the local salient features of channel extrema. The combination of the two avoids the loss of information from single pooling and provides a more comprehensive basis for spatial attention, making the localization of visual target areas more accurate. The combination result is transformed into a spatial attention weight map, which can quantify the salience of each spatial location.

[0075] In an exemplary embodiment, the original features of the image patch are processed by the saliency feature extraction module to obtain a channel attention weight map, including: performing average pooling and max pooling on the original features of the image patch by the saliency feature extraction module; and performing multilayer perceptron processing on the average pooling and max pooling results to obtain the channel attention weight map.

[0076] The salient feature extraction module performs a global averaging operation on the original features of the image patch in the spatial dimension. That is, it calculates the mean of all spatial pixels in each channel to obtain the average pooling result, which is a single-channel feature vector. This can smoothly retain the global spatial statistical information of each channel and reflect the overall semantic response strength of the channel.

[0077] The salient feature extraction module performs a global maximum operation on the original features of the image patch in the spatial dimension, that is, it calculates the maximum value for all spatial pixels of each channel to obtain the max pooling result, which is a single-channel feature vector. This can capture the local extreme value features of each channel and highlight the most representative semantic features within the channel.

[0078] Multilayer perceptron processing refers to the operation of learning channel dependencies by inputting the results of two pooling operations into a shared multilayer perceptron (MLP).

[0079] The two vectors output by the MLP are added pixel by pixel, and then activated by an activation function to map the final value to between 0 and 1, thus obtaining the channel attention weight map.

[0080] In this embodiment, average pooling reflects the global semantic strength of the channel, which can smooth out noise interference; max pooling captures the local extreme features of the channel, which can highlight the core semantic response of the target; when the two are combined, MLP can learn more comprehensive channel semantic information, and the generated weight map can more accurately distinguish between key channels and redundant channels.

[0081] In an exemplary embodiment, the visual image classification model includes an encoder and a classification head. Encoding and classification processing are performed on the feature vectors and positional codes of multiple image patches to obtain a visual image classification result. This includes: summing the feature vectors and positional codes of multiple image patches using the encoder to obtain summation results for each image patch; performing attention processing on the summation results to obtain attention weights for each image patch; weighting the feature vectors of multiple image patches according to the attention weights; and classifying the weighted results using the classification head to obtain the visual image classification result.

[0082] In the visual image classification model, the encoder can be an encoding module based on the Transformer architecture, which integrates two core logics: positional encoding fusion and multi-head attention.

[0083] The feature vectors and corresponding position codes of each image patch are added element by element to obtain the summation result of the image patch, establishing global semantic associations between image patches and generating enhanced features with spatial location and global dependencies.

[0084] The summation results of multiple image patches are subjected to multi-head attention processing, allowing the summation results of each image patch to interact with the summation results of other image patches and calculate their relevance weights. The attention weights are the probability distributions output after attention processing, used to characterize the semantic relevance between image patches.

[0085] By using attention weights to sum the feature vectors of multiple image patches in a weighted manner, a weighted processing result is obtained. This allows the image patch features to integrate globally highly correlated information, generating enhanced features with global semantics and improving the semantic recognition of visual image classification.

[0086] The classification head includes a classification branch and a regression branch. The classification head is mostly composed of fully connected layers or convolutional layers. The classification branch outputs the visual target category and confidence score, while the regression branch outputs the target bounding box coordinates.

[0087] For example, the encoder can be composed of L identical encoder layers stacked together (L can be 12, 24, etc.). The data processing flow for each encoder layer includes:

[0088] Step 1: Layer Normalization (Norm): Perform layer normalization on the input feature vector sequence to stabilize the training process.

[0089] Step 2: Multi-Head Attention: Self-Attention: Each feature vector calculates attention weights with all feature vectors in the feature vector sequence (including itself), and aggregates information from other feature vectors based on these weights. Multi-Head: The feature vector is split into multiple "heads," and self-attention is performed in parallel to capture different dependencies. Output: A new token sequence after attention-weighted aggregation.

[0090] Step 3: Residual Connection (Add): The input from Step 1 is directly added to the output from Step 2 to alleviate gradient vanishing and preserve the original information.

[0091] Step 4: Layer Normalization (Norm): Perform layer normalization again on the sum of the residuals.

[0092] Step 5: Feedforward Network (MLP): Each token independently passes through a two-layer fully connected network (with an activation function in the middle, such as GELU) for nonlinear transformation and feature enhancement.

[0093] Step 6: Residual Connection (Add): Add the output of Step 4 to the output of Step 5 to obtain the final output of the encoder layer.

[0094] The encoder's attention mechanism can focus more on important areas, improving the accuracy and efficiency of classification and detection.

[0095] In this embodiment, by summing the feature vectors of image blocks with their positional codes, the model can understand both the semantic content of the image blocks and their spatial layout, thus solving the problem of the encoder's insensitivity to position and improving the accuracy of visual target localization. Attention processing allows the features of each image block to be integrated with the contextual information of the entire sequence, enabling the establishment of cross-regional semantic associations, which significantly improves the accuracy of visual image classification and reduces false negatives and false positives in complex scenes.

[0096] To illustrate the visual image classification method and its effects in this solution in detail, a specific embodiment is described below:

[0097] The visual image classification method in this application can be applied to various scenarios. For example, it can be used in visual classification, visual object detection, object segmentation, and subsequent multimodal large-scale modeling, including visual image understanding, image-text question answering, and video understanding. Figure 4 The diagram shown is a schematic representation of the overall process of visual image classification in some embodiments.

[0098] 1. Input: A batch of images to be processed is given as input.

[0099] 2. Initialize network parameters: Based on the network structure, initialize the weights and biases of the Visual Image Classification (VIT) model, including: initializing the parameters of the Transformer encoder, such as embedding dimension, number of heads, and hidden layer size; initializing the filter size and number of channels of the convolutional and pooling layers. The image to be processed is segmented into multiple image patches, where each patch has three RGB channels.

[0100] 3. Saliency feature extraction: Extract saliency features for each patch of image blocks to obtain the saliency feature S of the fourth channel.

[0101] 4. Image Channel Fusion and Feature Extraction: For each image patch, the extracted salient features are concatenated with the RGB three channels. Simultaneously, a convolutional neural network is connected after each patch to extract features from the four-channel patch. Local feature representations can be extracted through convolution and pooling operations. For example, a series of convolutional layers and activation functions are applied to the input four-channel patch to extract local image features; pooling layers are used to downsample the local features, reducing the spatial dimensionality of the features, and obtaining the feature vectors of each image patch, i.e., the embedding features required by the encoder.

[0102] 5. Visual image classification: The extracted embedding is combined with the position code and input into the subsequent encoder. After encoding processing, it is input into the classification head to obtain the visual image classification result.

[0103] The aforementioned visual image classification methods, devices, computer equipment, computer-readable storage media, and computer program products classify images to be processed using a pre-trained visual image classification model. This model can extract original features and salient features from each image block of the image to be processed. The concatenation of the original and salient features allows the model to prioritize salient visual target regions in the image, improving its ability to detect visual targets and significantly reducing background noise interference. Furthermore, during the model's encoding stage, positional encoding of each image block is introduced, compensating for the lack of spatial information in the feature vectors and enabling the model to accurately understand the spatial layout of the target, thus improving the accuracy of visual image classification.

[0104] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0105] Based on the same inventive concept, this application also provides a visual image classification apparatus for implementing the visual image classification method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more embodiments of the visual image classification apparatus provided below can be found in the limitations of the visual image classification method described above, and will not be repeated here.

[0106] In one exemplary embodiment, such as Figure 5 As shown, a visual image classification device 500 is provided, including: an acquisition module 520 and a processing module 540, wherein:

[0107] The acquisition module 520 is used to acquire the image to be processed and input the image to be processed into a pre-trained visual image classification model.

[0108] The processing module 540 is used to segment the image to be processed using a visual image classification model to obtain multiple image blocks and their respective position codes; for each image block, it performs original feature extraction and salient feature extraction, concatenates the original features of the image block with the corresponding salient features, and vectorizes the concatenated features of the image block to obtain the feature vector of the image block; and performs encoding and classification processing based on the feature vectors and position codes of the multiple image blocks to obtain the visual image classification result.

[0109] The aforementioned visual image classification device classifies images to be processed using a pre-trained visual image classification model. This model can extract original features and salient features from each image block of the image to be processed. After concatenating the original features and salient features, the model can prioritize the visual target regions in the image, improving the model's ability to detect visual targets and significantly reducing the interference of background noise on detection. In the model's encoding stage, positional encoding of each image block is introduced to compensate for the lack of spatial information in the feature vectors, allowing the model to accurately understand the spatial layout of the target and improve the accuracy of visual image classification.

[0110] In one embodiment, the visual image classification model includes a salient feature extraction module; the processing module 540 is further configured to: perform channel attention processing and spatial attention processing on the original features of the image patch through the salient feature extraction module to obtain a spatial attention weight map; and use the spatial attention weight map as the salient feature of the image patch.

[0111] In one embodiment, the original features of the image patch are processed by the saliency feature extraction module to perform channel attention processing and spatial attention processing to obtain a spatial attention weight map. The processing module 540 is further configured to: perform channel attention processing on the original features of the image patch by the saliency feature extraction module to obtain a channel attention weight map; perform weighted processing on the original features of the image patch according to the channel attention weight map, and perform spatial attention processing on the weighted features to obtain a spatial attention weight map.

[0112] In one embodiment, spatial attention processing is performed on the weighted features to obtain a spatial attention weight map. The processing module 540 is further configured to: perform average pooling and max pooling processing on the weighted features respectively; concatenate the average pooling processing result and the max pooling processing result; and perform activation processing on the concatenated result to obtain a spatial attention weight map.

[0113] In one embodiment, the original features of the image patch are processed by the saliency feature extraction module to obtain a channel attention weight map. The processing module 540 is further configured to: perform average pooling and max pooling on the original features of the image patch by the saliency feature extraction module; and perform multilayer perceptual processing on the average pooling and max pooling results to obtain a channel attention weight map.

[0114] In one embodiment, the visual image classification model includes an encoder and a classification head; encoding and classification are performed on the feature vectors and position codes of multiple image patches to obtain a visual image classification result. The processing module 540 is further configured to: sum the feature vectors and position codes of multiple image patches using the encoder to obtain summation results for multiple image patches; perform attention processing on the summation results of multiple image patches to obtain attention weights for multiple image patches; perform weighted processing on the feature vectors of multiple image patches according to the attention weights; and perform classification processing on the weighted processing results using the classification head to obtain a visual image classification result.

[0115] Each module in the aforementioned visual image classification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0116] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a visual image classification method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0117] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0118] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0119] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0120] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0121] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0122] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0123] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0124] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A visual image classification method, characterized in that, The method includes: Obtain the image to be processed and input the image to be processed into a pre-trained visual image classification model; The image to be processed is segmented using the visual image classification model to obtain multiple image blocks and their respective position codes. For each image block, original features and salient features are extracted. The original features and corresponding salient features of the image block are concatenated, and the concatenated features are vectorized to obtain the feature vector of the image block. Encoding and classification processing are performed based on the feature vectors and position codes of the multiple image blocks to obtain the visual image classification result.

2. The method according to claim 1, characterized in that, The visual image classification model includes a salient feature extraction module; salient feature extraction of the image patch includes: The saliency feature extraction module performs channel attention processing and spatial attention processing on the original features of the image patch to obtain a spatial attention weight map. The spatial attention weight map is used as the saliency feature of the image patch.

3. The method according to claim 2, characterized in that, The step of performing channel attention processing and spatial attention processing on the original features of the image patch through the saliency feature extraction module to obtain a spatial attention weight map includes: The original features of the image patch are processed by the saliency feature extraction module to obtain a channel attention weight map; the original features of the image patch are weighted according to the channel attention weight map, and spatial attention is applied to the weighted features to obtain a spatial attention weight map.

4. The method according to claim 3, characterized in that, The step of performing spatial attention processing on the weighted features to obtain a spatial attention weight map includes: The weighted features are then subjected to average pooling and max pooling respectively. The average pooling result and the max pooling result are concatenated; The splicing result is activated to obtain a spatial attention weight map.

5. The method according to claim 3, characterized in that, The step of performing channel attention processing on the original features of the image patch through the saliency feature extraction module to obtain a channel attention weight map includes: The original features of the image patch are subjected to average pooling and max pooling processing by the saliency feature extraction module. The average pooling and max pooling results are processed by multi-layer perception to obtain the channel attention weight map.

6. The method according to claim 1, characterized in that, The visual image classification model includes an encoder and a classification head; the encoding and classification processes based on the feature vectors and positional codes of multiple image patches to obtain the visual image classification result include: The encoder sums the feature vectors and position codes of multiple image blocks to obtain summation results for each image block. Attention processing is then applied to the summation results of the multiple image blocks to obtain attention weights for each image block. The feature vectors of the multiple image blocks are then weighted according to the attention weights. The weighted processing result is classified using the classification head to obtain the visual image classification result.

7. A visual image classification device, characterized in that, The device includes: The acquisition module is used to acquire the image to be processed and input the image to be processed into a pre-trained visual image classification model. The processing module is used to segment the image to be processed using the visual image classification model to obtain multiple image blocks and their respective position codes; for each image block, the module performs original feature extraction and salient feature extraction, concatenates the original features of the image block with the corresponding salient features, and vectorizes the concatenated features of the image block to obtain the feature vector of the image block; and performs encoding and classification processing based on the feature vectors and position codes of the multiple image blocks to obtain the visual image classification result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images